2023-11-05T08:17:36.979 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-05T08:17:36.980 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_benhanokh@teuthology --verbose --name benhanokh-2023-10-31_16:46:33-rados-DBG_maintain_prefix_itr_2-distro-default-smithi --description rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} --archive /home/teuthworker/archive/benhanokh-2023-10-31_16:46:33-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442398 -- /home/teuthworker/archive/benhanokh-2023-10-31_16:46:33-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442398/orig.config.yaml 2023-11-05T08:17:37.019 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-05T08:17:37.103 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/benhanokh-2023-10-31_16:46:33-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442398 branch: DBG_maintain_prefix_itr_2 description: rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} email: null first_in_suite: false job_id: '7442398' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: benhanokh-2023-10-31_16:46:33-rados-DBG_maintain_prefix_itr_2-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: DBG_maintain_prefix_itr_2 ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 3 ms inject socket failures: 5000 osd_async_recovery_min_cost: 1 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.1 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd heartbeat use min delay socket: true osd max backfills: 3 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - slow request - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from sha1: 420cd129397301b1521ceb74e1bdbd9b855d2189 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 420cd129397301b1521ceb74e1bdbd9b855d2189 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: DBG_maintain_prefix_itr_2 sha1: 420cd129397301b1521ceb74e1bdbd9b855d2189 owner: scheduled_benhanokh@teuthology priority: 100 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.4 - osd.8 - osd.12 - - mon.b - osd.1 - osd.5 - osd.9 - osd.13 - - mon.c - osd.2 - osd.6 - osd.10 - osd.14 - - mgr.x - osd.3 - osd.7 - osd.11 - osd.15 - client.0 seed: 9009 sha1: 420cd129397301b1521ceb74e1bdbd9b855d2189 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: DBG_maintain_prefix_itr_2 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 420cd129397301b1521ceb74e1bdbd9b855d2189 targets: smithi002.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFYnh2HerGKUWiNK+JZ1+sOjYIB226bYu1t+bYcFbrDbwRbToSpMKjOKohouju5rpo4JGem8VCkcd8WCrEfDO90= smithi053.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJhqMGwSC/AaSpescnzcOcLSXMVxyFf8TWoJBg6tZz9J7np1OvfHqj0Q6WgZ/zHx4+f+RgNMtOA5jvGv7ZeAjsU= smithi069.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDUJGpZU1aZPH20MVSpQexFWypB4vrahbDUJdquA7RrNdQ0snsbTNu0jtgDOtsAX775dxnSa84npB5kPsKt0MRM= smithi090.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNtyBiM3I8tIXFthmH5r8xkuDBDlJ50iU5mm6jCAXzGFO6LLWNfz+LKJeC7G8VVF6lJYYzPO5xDEdjKdlVey0UI= tasks: - install: null - ceph: conf: osd: debug monc: 20 - thrashosds: bluestore_new_sharding: random chance_bluestore_reshard: 1 chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 min_in: 8 timeout: 1200 - rados: clients: - client.0 ec_pool: true erasure_code_profile: c: 2 crush-failure-domain: osd k: 4 m: 3 name: shecprofile plugin: shec objects: 50 op_weights: append: 100 copy_from: 50 delete: 50 read: 100 rmattr: 25 rollback: 50 setattr: 25 snap_create: 50 snap_remove: 50 write: 0 ops: 400 write_append_excl: false teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-10-31_16:46:33 tube: smithi user: benhanokh verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3084822 2023-11-05T08:17:37.103 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa; will attempt to use it 2023-11-05T08:17:37.104 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks 2023-11-05T08:17:37.104 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-05T08:17:37.106 INFO:teuthology.task.internal:Checking packages... 2023-11-05T08:17:37.130 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '420cd129397301b1521ceb74e1bdbd9b855d2189' 2023-11-05T08:17:37.130 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-05T08:17:37.130 INFO:teuthology.packaging:ref: None 2023-11-05T08:17:37.131 INFO:teuthology.packaging:tag: None 2023-11-05T08:17:37.131 INFO:teuthology.packaging:branch: DBG_maintain_prefix_itr_2 2023-11-05T08:17:37.131 INFO:teuthology.packaging:sha1: 420cd129397301b1521ceb74e1bdbd9b855d2189 2023-11-05T08:17:37.132 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=DBG_maintain_prefix_itr_2 2023-11-05T08:17:37.277 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6776-g420cd129-1jammy 2023-11-05T08:17:37.278 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-05T08:17:37.286 INFO:teuthology.task.internal:no buildpackages task found 2023-11-05T08:17:37.286 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-05T08:17:37.306 INFO:teuthology.task.internal:Saving configuration 2023-11-05T08:17:37.320 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-05T08:17:37.327 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-05T08:17:37.354 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi002.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/benhanokh-2023-10-31_16:46:33-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442398', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-05 08:11:03.357295', 'locked_by': 'scheduled_benhanokh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOgpErus2jm6yMx5FA/mP0Y1J7Zubw5WtbRTOexDy9Xb'} 2023-11-05T08:17:37.378 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi053.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/benhanokh-2023-10-31_16:46:33-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442398', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-05 08:11:03.353634', 'locked_by': 'scheduled_benhanokh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGRExjgpCBh+raVS3mg72tkvU528AZTuMq67hSmou+4x'} 2023-11-05T08:17:37.405 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/benhanokh-2023-10-31_16:46:33-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442398', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-05 08:11:03.358959', 'locked_by': 'scheduled_benhanokh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAyz4K9T6UmXPJ+dEaAPU/PCNgpJIJvmSzpMe2vQJpu6'} 2023-11-05T08:17:37.426 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi090.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/benhanokh-2023-10-31_16:46:33-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442398', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-05 08:11:03.355607', 'locked_by': 'scheduled_benhanokh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOrL3xx0+ION8M9lB25OE7IrMtnteuMNTVTnWxCDZT9O'} 2023-11-05T08:17:37.426 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-05T08:17:37.434 INFO:teuthology.task.internal:roles: ubuntu@smithi002.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'] 2023-11-05T08:17:37.435 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'] 2023-11-05T08:17:37.435 INFO:teuthology.task.internal:roles: ubuntu@smithi069.front.sepia.ceph.com - ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'] 2023-11-05T08:17:37.435 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0'] 2023-11-05T08:17:37.435 INFO:teuthology.run_tasks:Running task console_log... 2023-11-05T08:17:37.571 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6fa2823af0>, signals=[15]) 2023-11-05T08:17:37.573 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-05T08:17:37.580 INFO:teuthology.task.internal:Opening connections... 2023-11-05T08:17:37.580 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi002.front.sepia.ceph.com 2023-11-05T08:17:37.583 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-05T08:17:37.659 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi053.front.sepia.ceph.com 2023-11-05T08:17:37.660 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-05T08:17:37.733 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi069.front.sepia.ceph.com 2023-11-05T08:17:37.733 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-05T08:17:37.809 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi090.front.sepia.ceph.com 2023-11-05T08:17:37.810 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-05T08:17:37.886 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-05T08:17:37.893 DEBUG:teuthology.orchestra.run.smithi002:> uname -m 2023-11-05T08:17:37.900 INFO:teuthology.orchestra.run.smithi002.stdout:x86_64 2023-11-05T08:17:37.900 DEBUG:teuthology.orchestra.run.smithi002:> cat /etc/os-release 2023-11-05T08:17:37.948 INFO:teuthology.orchestra.run.smithi002.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-11-05T08:17:37.949 INFO:teuthology.orchestra.run.smithi002.stdout:NAME="Ubuntu" 2023-11-05T08:17:37.949 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION_ID="22.04" 2023-11-05T08:17:37.949 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-11-05T08:17:37.949 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION_CODENAME=jammy 2023-11-05T08:17:37.950 INFO:teuthology.orchestra.run.smithi002.stdout:ID=ubuntu 2023-11-05T08:17:37.950 INFO:teuthology.orchestra.run.smithi002.stdout:ID_LIKE=debian 2023-11-05T08:17:37.950 INFO:teuthology.orchestra.run.smithi002.stdout:HOME_URL="https://www.ubuntu.com/" 2023-11-05T08:17:37.950 INFO:teuthology.orchestra.run.smithi002.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-11-05T08:17:37.950 INFO:teuthology.orchestra.run.smithi002.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-11-05T08:17:37.951 INFO:teuthology.orchestra.run.smithi002.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-11-05T08:17:37.951 INFO:teuthology.orchestra.run.smithi002.stdout:UBUNTU_CODENAME=jammy 2023-11-05T08:17:37.951 INFO:teuthology.lock.ops:Updating smithi002.front.sepia.ceph.com on lock server 2023-11-05T08:17:37.978 DEBUG:teuthology.orchestra.run.smithi053:> uname -m 2023-11-05T08:17:37.983 INFO:teuthology.orchestra.run.smithi053.stdout:x86_64 2023-11-05T08:17:37.983 DEBUG:teuthology.orchestra.run.smithi053:> cat /etc/os-release 2023-11-05T08:17:38.028 INFO:teuthology.orchestra.run.smithi053.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-11-05T08:17:38.029 INFO:teuthology.orchestra.run.smithi053.stdout:NAME="Ubuntu" 2023-11-05T08:17:38.029 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION_ID="22.04" 2023-11-05T08:17:38.029 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-11-05T08:17:38.029 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION_CODENAME=jammy 2023-11-05T08:17:38.029 INFO:teuthology.orchestra.run.smithi053.stdout:ID=ubuntu 2023-11-05T08:17:38.030 INFO:teuthology.orchestra.run.smithi053.stdout:ID_LIKE=debian 2023-11-05T08:17:38.030 INFO:teuthology.orchestra.run.smithi053.stdout:HOME_URL="https://www.ubuntu.com/" 2023-11-05T08:17:38.030 INFO:teuthology.orchestra.run.smithi053.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-11-05T08:17:38.030 INFO:teuthology.orchestra.run.smithi053.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-11-05T08:17:38.030 INFO:teuthology.orchestra.run.smithi053.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-11-05T08:17:38.031 INFO:teuthology.orchestra.run.smithi053.stdout:UBUNTU_CODENAME=jammy 2023-11-05T08:17:38.031 INFO:teuthology.lock.ops:Updating smithi053.front.sepia.ceph.com on lock server 2023-11-05T08:17:38.052 DEBUG:teuthology.orchestra.run.smithi069:> uname -m 2023-11-05T08:17:38.058 INFO:teuthology.orchestra.run.smithi069.stdout:x86_64 2023-11-05T08:17:38.059 DEBUG:teuthology.orchestra.run.smithi069:> cat /etc/os-release 2023-11-05T08:17:38.104 INFO:teuthology.orchestra.run.smithi069.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-11-05T08:17:38.104 INFO:teuthology.orchestra.run.smithi069.stdout:NAME="Ubuntu" 2023-11-05T08:17:38.104 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION_ID="22.04" 2023-11-05T08:17:38.104 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-11-05T08:17:38.105 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION_CODENAME=jammy 2023-11-05T08:17:38.105 INFO:teuthology.orchestra.run.smithi069.stdout:ID=ubuntu 2023-11-05T08:17:38.105 INFO:teuthology.orchestra.run.smithi069.stdout:ID_LIKE=debian 2023-11-05T08:17:38.105 INFO:teuthology.orchestra.run.smithi069.stdout:HOME_URL="https://www.ubuntu.com/" 2023-11-05T08:17:38.105 INFO:teuthology.orchestra.run.smithi069.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-11-05T08:17:38.106 INFO:teuthology.orchestra.run.smithi069.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-11-05T08:17:38.106 INFO:teuthology.orchestra.run.smithi069.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-11-05T08:17:38.106 INFO:teuthology.orchestra.run.smithi069.stdout:UBUNTU_CODENAME=jammy 2023-11-05T08:17:38.106 INFO:teuthology.lock.ops:Updating smithi069.front.sepia.ceph.com on lock server 2023-11-05T08:17:38.135 DEBUG:teuthology.orchestra.run.smithi090:> uname -m 2023-11-05T08:17:38.141 INFO:teuthology.orchestra.run.smithi090.stdout:x86_64 2023-11-05T08:17:38.142 DEBUG:teuthology.orchestra.run.smithi090:> cat /etc/os-release 2023-11-05T08:17:38.187 INFO:teuthology.orchestra.run.smithi090.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-11-05T08:17:38.188 INFO:teuthology.orchestra.run.smithi090.stdout:NAME="Ubuntu" 2023-11-05T08:17:38.188 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION_ID="22.04" 2023-11-05T08:17:38.188 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-11-05T08:17:38.188 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION_CODENAME=jammy 2023-11-05T08:17:38.189 INFO:teuthology.orchestra.run.smithi090.stdout:ID=ubuntu 2023-11-05T08:17:38.189 INFO:teuthology.orchestra.run.smithi090.stdout:ID_LIKE=debian 2023-11-05T08:17:38.189 INFO:teuthology.orchestra.run.smithi090.stdout:HOME_URL="https://www.ubuntu.com/" 2023-11-05T08:17:38.189 INFO:teuthology.orchestra.run.smithi090.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-11-05T08:17:38.189 INFO:teuthology.orchestra.run.smithi090.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-11-05T08:17:38.189 INFO:teuthology.orchestra.run.smithi090.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-11-05T08:17:38.190 INFO:teuthology.orchestra.run.smithi090.stdout:UBUNTU_CODENAME=jammy 2023-11-05T08:17:38.190 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server 2023-11-05T08:17:38.211 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-05T08:17:38.220 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-05T08:17:38.232 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-05T08:17:38.233 DEBUG:teuthology.orchestra.run.smithi002:> test '!' -e /home/ubuntu/cephtest 2023-11-05T08:17:38.236 DEBUG:teuthology.orchestra.run.smithi053:> test '!' -e /home/ubuntu/cephtest 2023-11-05T08:17:38.240 DEBUG:teuthology.orchestra.run.smithi069:> test '!' -e /home/ubuntu/cephtest 2023-11-05T08:17:38.243 DEBUG:teuthology.orchestra.run.smithi090:> test '!' -e /home/ubuntu/cephtest 2023-11-05T08:17:38.248 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-05T08:17:38.259 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-05T08:17:38.260 DEBUG:teuthology.orchestra.run.smithi002:> test -z $(ls -A /var/lib/ceph) 2023-11-05T08:17:38.286 DEBUG:teuthology.orchestra.run.smithi053:> test -z $(ls -A /var/lib/ceph) 2023-11-05T08:17:38.288 DEBUG:teuthology.orchestra.run.smithi069:> test -z $(ls -A /var/lib/ceph) 2023-11-05T08:17:38.291 DEBUG:teuthology.orchestra.run.smithi090:> test -z $(ls -A /var/lib/ceph) 2023-11-05T08:17:38.324 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-05T08:17:38.431 INFO:teuthology.run_tasks:Running task kernel... 2023-11-05T08:17:38.451 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-11-05T08:17:38.452 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-11-05T08:17:38.452 DEBUG:teuthology.orchestra.run.smithi002:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-05T08:17:38.453 DEBUG:teuthology.orchestra.run.smithi053:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-05T08:17:38.454 DEBUG:teuthology.orchestra.run.smithi069:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-05T08:17:38.454 DEBUG:teuthology.orchestra.run.smithi090:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-05T08:17:38.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-05T08:17:38.459 DEBUG:teuthology.orchestra.run.smithi069:> uname -r 2023-11-05T08:17:38.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-05T08:17:38.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-05T08:17:38.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-05T08:17:38.462 DEBUG:teuthology.orchestra.run.smithi090:> uname -r 2023-11-05T08:17:38.463 DEBUG:teuthology.orchestra.run.smithi002:> uname -r 2023-11-05T08:17:38.463 DEBUG:teuthology.orchestra.run.smithi053:> uname -r 2023-11-05T08:17:38.509 INFO:teuthology.orchestra.run.smithi069.stdout:5.15.0-56-generic 2023-11-05T08:17:38.509 INFO:teuthology.orchestra.run.smithi090.stdout:5.15.0-56-generic 2023-11-05T08:17:38.509 INFO:teuthology.task.kernel:Running kernel on smithi069: 5.15.0-56-generic 2023-11-05T08:17:38.509 DEBUG:teuthology.orchestra.run.smithi069:> sudo apt-get clean 2023-11-05T08:17:38.510 INFO:teuthology.task.kernel:Running kernel on smithi090: 5.15.0-56-generic 2023-11-05T08:17:38.510 DEBUG:teuthology.orchestra.run.smithi090:> sudo apt-get clean 2023-11-05T08:17:38.512 INFO:teuthology.orchestra.run.smithi053.stdout:5.15.0-56-generic 2023-11-05T08:17:38.512 INFO:teuthology.orchestra.run.smithi002.stdout:5.15.0-56-generic 2023-11-05T08:17:38.512 INFO:teuthology.task.kernel:Running kernel on smithi053: 5.15.0-56-generic 2023-11-05T08:17:38.513 DEBUG:teuthology.orchestra.run.smithi053:> sudo apt-get clean 2023-11-05T08:17:38.513 INFO:teuthology.task.kernel:Running kernel on smithi002: 5.15.0-56-generic 2023-11-05T08:17:38.514 DEBUG:teuthology.orchestra.run.smithi002:> sudo apt-get clean 2023-11-05T08:17:38.629 DEBUG:teuthology.orchestra.run.smithi053:> sudo apt-get update 2023-11-05T08:17:38.632 DEBUG:teuthology.orchestra.run.smithi090:> sudo apt-get update 2023-11-05T08:17:38.648 DEBUG:teuthology.orchestra.run.smithi002:> sudo apt-get update 2023-11-05T08:17:38.672 DEBUG:teuthology.orchestra.run.smithi069:> sudo apt-get update 2023-11-05T08:17:38.803 INFO:teuthology.orchestra.run.smithi090.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-11-05T08:17:38.814 INFO:teuthology.orchestra.run.smithi069.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-11-05T08:17:38.814 INFO:teuthology.orchestra.run.smithi069.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-11-05T08:17:38.815 INFO:teuthology.orchestra.run.smithi069.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-11-05T08:17:38.817 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-11-05T08:17:38.818 INFO:teuthology.orchestra.run.smithi002.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-11-05T08:17:38.818 INFO:teuthology.orchestra.run.smithi002.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-11-05T08:17:38.825 INFO:teuthology.orchestra.run.smithi090.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-11-05T08:17:38.886 INFO:teuthology.orchestra.run.smithi002.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-11-05T08:17:38.894 INFO:teuthology.orchestra.run.smithi090.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-11-05T08:17:38.899 INFO:teuthology.orchestra.run.smithi069.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-11-05T08:17:38.914 INFO:teuthology.orchestra.run.smithi090.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-11-05T08:17:38.931 INFO:teuthology.orchestra.run.smithi053.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-11-05T08:17:38.932 INFO:teuthology.orchestra.run.smithi053.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-11-05T08:17:38.993 INFO:teuthology.orchestra.run.smithi053.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-11-05T08:17:39.035 INFO:teuthology.orchestra.run.smithi069.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [940 kB] 2023-11-05T08:17:39.057 INFO:teuthology.orchestra.run.smithi090.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,149 kB] 2023-11-05T08:17:39.063 INFO:teuthology.orchestra.run.smithi002.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [360 kB] 2023-11-05T08:17:39.106 INFO:teuthology.orchestra.run.smithi069.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,149 kB] 2023-11-05T08:17:39.121 INFO:teuthology.orchestra.run.smithi069.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [360 kB] 2023-11-05T08:17:39.125 INFO:teuthology.orchestra.run.smithi002.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [940 kB] 2023-11-05T08:17:39.126 INFO:teuthology.orchestra.run.smithi069.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [186 kB] 2023-11-05T08:17:39.128 INFO:teuthology.orchestra.run.smithi069.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2023-11-05T08:17:39.129 INFO:teuthology.orchestra.run.smithi069.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [32.4 kB] 2023-11-05T08:17:39.129 INFO:teuthology.orchestra.run.smithi069.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,085 kB] 2023-11-05T08:17:39.139 INFO:teuthology.orchestra.run.smithi002.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,149 kB] 2023-11-05T08:17:39.156 INFO:teuthology.orchestra.run.smithi002.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [186 kB] 2023-11-05T08:17:39.157 INFO:teuthology.orchestra.run.smithi090.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [524 kB] 2023-11-05T08:17:39.158 INFO:teuthology.orchestra.run.smithi069.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [176 kB] 2023-11-05T08:17:39.159 INFO:teuthology.orchestra.run.smithi069.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2023-11-05T08:17:39.160 INFO:teuthology.orchestra.run.smithi069.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [797 kB] 2023-11-05T08:17:39.160 INFO:teuthology.orchestra.run.smithi002.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2023-11-05T08:17:39.160 INFO:teuthology.orchestra.run.smithi002.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,085 kB] 2023-11-05T08:17:39.163 INFO:teuthology.orchestra.run.smithi090.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [245 kB] 2023-11-05T08:17:39.166 INFO:teuthology.orchestra.run.smithi090.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2023-11-05T08:17:39.167 INFO:teuthology.orchestra.run.smithi090.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,105 kB] 2023-11-05T08:17:39.171 INFO:teuthology.orchestra.run.smithi069.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [563 kB] 2023-11-05T08:17:39.173 INFO:teuthology.orchestra.run.smithi069.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [146 kB] 2023-11-05T08:17:39.174 INFO:teuthology.orchestra.run.smithi002.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [32.4 kB] 2023-11-05T08:17:39.175 INFO:teuthology.orchestra.run.smithi002.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [176 kB] 2023-11-05T08:17:39.175 INFO:teuthology.orchestra.run.smithi069.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2023-11-05T08:17:39.176 INFO:teuthology.orchestra.run.smithi069.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-11-05T08:17:39.176 INFO:teuthology.orchestra.run.smithi069.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-11-05T08:17:39.176 INFO:teuthology.orchestra.run.smithi069.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-11-05T08:17:39.179 INFO:teuthology.orchestra.run.smithi069.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-11-05T08:17:39.184 INFO:teuthology.orchestra.run.smithi090.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [32.8 kB] 2023-11-05T08:17:39.185 INFO:teuthology.orchestra.run.smithi090.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [179 kB] 2023-11-05T08:17:39.187 INFO:teuthology.orchestra.run.smithi090.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2023-11-05T08:17:39.187 INFO:teuthology.orchestra.run.smithi090.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [999 kB] 2023-11-05T08:17:39.190 INFO:teuthology.orchestra.run.smithi002.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2023-11-05T08:17:39.190 INFO:teuthology.orchestra.run.smithi002.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [797 kB] 2023-11-05T08:17:39.192 INFO:teuthology.orchestra.run.smithi069.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [524 kB] 2023-11-05T08:17:39.196 INFO:teuthology.orchestra.run.smithi002.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [563 kB] 2023-11-05T08:17:39.200 INFO:teuthology.orchestra.run.smithi090.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [661 kB] 2023-11-05T08:17:39.200 INFO:teuthology.orchestra.run.smithi002.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [146 kB] 2023-11-05T08:17:39.202 INFO:teuthology.orchestra.run.smithi069.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [245 kB] 2023-11-05T08:17:39.205 INFO:teuthology.orchestra.run.smithi002.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2023-11-05T08:17:39.206 INFO:teuthology.orchestra.run.smithi002.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-11-05T08:17:39.206 INFO:teuthology.orchestra.run.smithi002.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-11-05T08:17:39.208 INFO:teuthology.orchestra.run.smithi069.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2023-11-05T08:17:39.209 INFO:teuthology.orchestra.run.smithi069.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,105 kB] 2023-11-05T08:17:39.209 INFO:teuthology.orchestra.run.smithi002.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-11-05T08:17:39.209 INFO:teuthology.orchestra.run.smithi002.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-11-05T08:17:39.210 INFO:teuthology.orchestra.run.smithi090.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [219 kB] 2023-11-05T08:17:39.213 INFO:teuthology.orchestra.run.smithi002.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [524 kB] 2023-11-05T08:17:39.218 INFO:teuthology.orchestra.run.smithi002.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [245 kB] 2023-11-05T08:17:39.221 INFO:teuthology.orchestra.run.smithi002.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2023-11-05T08:17:39.221 INFO:teuthology.orchestra.run.smithi002.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,105 kB] 2023-11-05T08:17:39.221 INFO:teuthology.orchestra.run.smithi069.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [32.8 kB] 2023-11-05T08:17:39.222 INFO:teuthology.orchestra.run.smithi069.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [179 kB] 2023-11-05T08:17:39.225 INFO:teuthology.orchestra.run.smithi069.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2023-11-05T08:17:39.226 INFO:teuthology.orchestra.run.smithi069.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [999 kB] 2023-11-05T08:17:39.227 INFO:teuthology.orchestra.run.smithi090.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2023-11-05T08:17:39.228 INFO:teuthology.orchestra.run.smithi090.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-11-05T08:17:39.229 INFO:teuthology.orchestra.run.smithi090.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-11-05T08:17:39.229 INFO:teuthology.orchestra.run.smithi090.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-11-05T08:17:39.233 INFO:teuthology.orchestra.run.smithi090.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2023-11-05T08:17:39.234 INFO:teuthology.orchestra.run.smithi090.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [56.3 kB] 2023-11-05T08:17:39.235 INFO:teuthology.orchestra.run.smithi090.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [64.2 kB] 2023-11-05T08:17:39.235 INFO:teuthology.orchestra.run.smithi090.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-11-05T08:17:39.236 INFO:teuthology.orchestra.run.smithi090.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-11-05T08:17:39.236 INFO:teuthology.orchestra.run.smithi069.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [661 kB] 2023-11-05T08:17:39.237 INFO:teuthology.orchestra.run.smithi002.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [32.8 kB] 2023-11-05T08:17:39.237 INFO:teuthology.orchestra.run.smithi002.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [179 kB] 2023-11-05T08:17:39.238 INFO:teuthology.orchestra.run.smithi002.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2023-11-05T08:17:39.238 INFO:teuthology.orchestra.run.smithi002.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [999 kB] 2023-11-05T08:17:39.242 INFO:teuthology.orchestra.run.smithi069.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [219 kB] 2023-11-05T08:17:39.244 INFO:teuthology.orchestra.run.smithi069.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2023-11-05T08:17:39.244 INFO:teuthology.orchestra.run.smithi069.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-11-05T08:17:39.245 INFO:teuthology.orchestra.run.smithi069.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-11-05T08:17:39.245 INFO:teuthology.orchestra.run.smithi069.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-11-05T08:17:39.246 INFO:teuthology.orchestra.run.smithi069.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2023-11-05T08:17:39.246 INFO:teuthology.orchestra.run.smithi002.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [661 kB] 2023-11-05T08:17:39.251 INFO:teuthology.orchestra.run.smithi002.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [219 kB] 2023-11-05T08:17:39.252 INFO:teuthology.orchestra.run.smithi002.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2023-11-05T08:17:39.254 INFO:teuthology.orchestra.run.smithi002.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-11-05T08:17:39.255 INFO:teuthology.orchestra.run.smithi002.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-11-05T08:17:39.255 INFO:teuthology.orchestra.run.smithi002.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-11-05T08:17:39.255 INFO:teuthology.orchestra.run.smithi002.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2023-11-05T08:17:39.257 INFO:teuthology.orchestra.run.smithi090.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.9 kB] 2023-11-05T08:17:39.257 INFO:teuthology.orchestra.run.smithi090.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.8 kB] 2023-11-05T08:17:39.257 INFO:teuthology.orchestra.run.smithi090.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.4 kB] 2023-11-05T08:17:39.257 INFO:teuthology.orchestra.run.smithi090.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2023-11-05T08:17:39.263 INFO:teuthology.orchestra.run.smithi069.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [64.2 kB] 2023-11-05T08:17:39.264 INFO:teuthology.orchestra.run.smithi069.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [56.3 kB] 2023-11-05T08:17:39.265 INFO:teuthology.orchestra.run.smithi069.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-11-05T08:17:39.265 INFO:teuthology.orchestra.run.smithi069.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-11-05T08:17:39.269 INFO:teuthology.orchestra.run.smithi002.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [64.2 kB] 2023-11-05T08:17:39.270 INFO:teuthology.orchestra.run.smithi002.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [56.3 kB] 2023-11-05T08:17:39.271 INFO:teuthology.orchestra.run.smithi002.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-11-05T08:17:39.271 INFO:teuthology.orchestra.run.smithi002.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-11-05T08:17:39.271 INFO:teuthology.orchestra.run.smithi002.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.9 kB] 2023-11-05T08:17:39.272 INFO:teuthology.orchestra.run.smithi002.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.8 kB] 2023-11-05T08:17:39.283 INFO:teuthology.orchestra.run.smithi069.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.9 kB] 2023-11-05T08:17:39.283 INFO:teuthology.orchestra.run.smithi069.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.8 kB] 2023-11-05T08:17:39.284 INFO:teuthology.orchestra.run.smithi069.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.4 kB] 2023-11-05T08:17:39.284 INFO:teuthology.orchestra.run.smithi069.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2023-11-05T08:17:39.288 INFO:teuthology.orchestra.run.smithi002.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.4 kB] 2023-11-05T08:17:39.288 INFO:teuthology.orchestra.run.smithi002.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2023-11-05T08:17:39.339 INFO:teuthology.orchestra.run.smithi090.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [360 kB] 2023-11-05T08:17:39.346 INFO:teuthology.orchestra.run.smithi053.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-11-05T08:17:39.356 INFO:teuthology.orchestra.run.smithi053.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [360 kB] 2023-11-05T08:17:39.485 INFO:teuthology.orchestra.run.smithi053.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,149 kB] 2023-11-05T08:17:39.602 INFO:teuthology.orchestra.run.smithi090.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [940 kB] 2023-11-05T08:17:39.626 INFO:teuthology.orchestra.run.smithi053.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [940 kB] 2023-11-05T08:17:39.751 INFO:teuthology.orchestra.run.smithi090.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [186 kB] 2023-11-05T08:17:39.762 INFO:teuthology.orchestra.run.smithi053.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [524 kB] 2023-11-05T08:17:39.771 INFO:teuthology.orchestra.run.smithi090.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2023-11-05T08:17:39.771 INFO:teuthology.orchestra.run.smithi090.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [32.4 kB] 2023-11-05T08:17:39.773 INFO:teuthology.orchestra.run.smithi090.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,085 kB] 2023-11-05T08:17:39.775 INFO:teuthology.orchestra.run.smithi053.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [186 kB] 2023-11-05T08:17:39.794 INFO:teuthology.orchestra.run.smithi053.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2023-11-05T08:17:39.795 INFO:teuthology.orchestra.run.smithi053.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [32.4 kB] 2023-11-05T08:17:39.797 INFO:teuthology.orchestra.run.smithi053.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,085 kB] 2023-11-05T08:17:39.800 INFO:teuthology.orchestra.run.smithi053.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [245 kB] 2023-11-05T08:17:39.818 INFO:teuthology.orchestra.run.smithi053.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2023-11-05T08:17:39.819 INFO:teuthology.orchestra.run.smithi053.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [32.8 kB] 2023-11-05T08:17:39.821 INFO:teuthology.orchestra.run.smithi053.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,105 kB] 2023-11-05T08:17:39.841 INFO:teuthology.orchestra.run.smithi090.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [176 kB] 2023-11-05T08:17:39.850 INFO:teuthology.orchestra.run.smithi090.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2023-11-05T08:17:39.850 INFO:teuthology.orchestra.run.smithi090.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [563 kB] 2023-11-05T08:17:39.865 INFO:teuthology.orchestra.run.smithi053.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [176 kB] 2023-11-05T08:17:39.872 INFO:teuthology.orchestra.run.smithi090.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [797 kB] 2023-11-05T08:17:39.907 INFO:teuthology.orchestra.run.smithi053.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [179 kB] 2023-11-05T08:17:39.907 INFO:teuthology.orchestra.run.smithi053.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2023-11-05T08:17:39.908 INFO:teuthology.orchestra.run.smithi053.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [797 kB] 2023-11-05T08:17:39.910 INFO:teuthology.orchestra.run.smithi053.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2023-11-05T08:17:39.921 INFO:teuthology.orchestra.run.smithi053.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [563 kB] 2023-11-05T08:17:40.062 INFO:teuthology.orchestra.run.smithi090.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [146 kB] 2023-11-05T08:17:40.074 INFO:teuthology.orchestra.run.smithi053.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [999 kB] 2023-11-05T08:17:40.095 INFO:teuthology.orchestra.run.smithi053.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [146 kB] 2023-11-05T08:17:40.447 INFO:teuthology.orchestra.run.smithi090.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2023-11-05T08:17:40.454 INFO:teuthology.orchestra.run.smithi090.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-11-05T08:17:40.470 INFO:teuthology.orchestra.run.smithi090.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-11-05T08:17:40.470 INFO:teuthology.orchestra.run.smithi090.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-11-05T08:17:40.473 INFO:teuthology.orchestra.run.smithi090.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-11-05T08:17:40.478 INFO:teuthology.orchestra.run.smithi053.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2023-11-05T08:17:40.486 INFO:teuthology.orchestra.run.smithi053.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-11-05T08:17:40.500 INFO:teuthology.orchestra.run.smithi053.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-11-05T08:17:40.501 INFO:teuthology.orchestra.run.smithi053.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-11-05T08:17:40.504 INFO:teuthology.orchestra.run.smithi053.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-11-05T08:17:40.693 INFO:teuthology.orchestra.run.smithi053.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [661 kB] 2023-11-05T08:17:40.731 INFO:teuthology.orchestra.run.smithi053.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [219 kB] 2023-11-05T08:17:40.743 INFO:teuthology.orchestra.run.smithi053.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2023-11-05T08:17:40.744 INFO:teuthology.orchestra.run.smithi053.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-11-05T08:17:40.744 INFO:teuthology.orchestra.run.smithi053.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-11-05T08:17:40.747 INFO:teuthology.orchestra.run.smithi053.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-11-05T08:17:40.747 INFO:teuthology.orchestra.run.smithi053.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2023-11-05T08:17:40.748 INFO:teuthology.orchestra.run.smithi053.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [56.3 kB] 2023-11-05T08:17:40.774 INFO:teuthology.orchestra.run.smithi053.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [64.2 kB] 2023-11-05T08:17:40.778 INFO:teuthology.orchestra.run.smithi053.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-11-05T08:17:40.943 INFO:teuthology.orchestra.run.smithi053.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-11-05T08:17:41.025 INFO:teuthology.orchestra.run.smithi053.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.9 kB] 2023-11-05T08:17:41.107 INFO:teuthology.orchestra.run.smithi053.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.8 kB] 2023-11-05T08:17:41.161 INFO:teuthology.orchestra.run.smithi053.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.4 kB] 2023-11-05T08:17:41.218 INFO:teuthology.orchestra.run.smithi053.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2023-11-05T08:17:44.459 INFO:teuthology.orchestra.run.smithi069.stdout:Fetched 10.1 MB in 1s (6,864 kB/s) 2023-11-05T08:17:44.797 INFO:teuthology.orchestra.run.smithi002.stdout:Fetched 10.1 MB in 2s (6,669 kB/s) 2023-11-05T08:17:44.875 INFO:teuthology.orchestra.run.smithi090.stdout:Fetched 10.1 MB in 2s (5,757 kB/s) 2023-11-05T08:17:45.730 INFO:teuthology.orchestra.run.smithi053.stdout:Fetched 10.1 MB in 3s (4,020 kB/s) 2023-11-05T08:17:45.768 INFO:teuthology.orchestra.run.smithi069.stdout:Reading package lists... 2023-11-05T08:17:45.787 DEBUG:teuthology.orchestra.run.smithi069:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-11-05T08:17:45.848 INFO:teuthology.orchestra.run.smithi069.stdout:Reading package lists... 2023-11-05T08:17:46.034 INFO:teuthology.orchestra.run.smithi069.stdout:Building dependency tree... 2023-11-05T08:17:46.035 INFO:teuthology.orchestra.run.smithi069.stdout:Reading state information... 2023-11-05T08:17:46.149 INFO:teuthology.orchestra.run.smithi002.stdout:Reading package lists... 2023-11-05T08:17:46.166 DEBUG:teuthology.orchestra.run.smithi002:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-11-05T08:17:46.189 INFO:teuthology.orchestra.run.smithi069.stdout:The following packages were automatically installed and are no longer required: 2023-11-05T08:17:46.189 INFO:teuthology.orchestra.run.smithi069.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-11-05T08:17:46.189 INFO:teuthology.orchestra.run.smithi069.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-11-05T08:17:46.190 INFO:teuthology.orchestra.run.smithi069.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-11-05T08:17:46.190 INFO:teuthology.orchestra.run.smithi069.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-11-05T08:17:46.190 INFO:teuthology.orchestra.run.smithi069.stdout: python2.7-minimal 2023-11-05T08:17:46.190 INFO:teuthology.orchestra.run.smithi069.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-05T08:17:46.191 INFO:teuthology.orchestra.run.smithi069.stdout:The following additional packages will be installed: 2023-11-05T08:17:46.191 INFO:teuthology.orchestra.run.smithi069.stdout: linux-generic linux-headers-5.15.0-88 linux-headers-5.15.0-88-generic 2023-11-05T08:17:46.191 INFO:teuthology.orchestra.run.smithi069.stdout: linux-headers-generic linux-image-5.15.0-88-generic 2023-11-05T08:17:46.192 INFO:teuthology.orchestra.run.smithi069.stdout: linux-modules-5.15.0-88-generic linux-modules-extra-5.15.0-88-generic 2023-11-05T08:17:46.192 INFO:teuthology.orchestra.run.smithi069.stdout:Suggested packages: 2023-11-05T08:17:46.192 INFO:teuthology.orchestra.run.smithi069.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-11-05T08:17:46.193 INFO:teuthology.orchestra.run.smithi069.stdout:Recommended packages: 2023-11-05T08:17:46.193 INFO:teuthology.orchestra.run.smithi069.stdout: thermald 2023-11-05T08:17:46.234 INFO:teuthology.orchestra.run.smithi002.stdout:Reading package lists... 2023-11-05T08:17:46.252 INFO:teuthology.orchestra.run.smithi069.stdout:The following NEW packages will be installed: 2023-11-05T08:17:46.252 INFO:teuthology.orchestra.run.smithi069.stdout: linux-headers-5.15.0-88 linux-headers-5.15.0-88-generic 2023-11-05T08:17:46.253 INFO:teuthology.orchestra.run.smithi069.stdout: linux-image-5.15.0-88-generic linux-modules-5.15.0-88-generic 2023-11-05T08:17:46.253 INFO:teuthology.orchestra.run.smithi069.stdout: linux-modules-extra-5.15.0-88-generic 2023-11-05T08:17:46.254 INFO:teuthology.orchestra.run.smithi069.stdout:The following packages will be upgraded: 2023-11-05T08:17:46.254 INFO:teuthology.orchestra.run.smithi069.stdout: linux-generic linux-headers-generic linux-image-generic 2023-11-05T08:17:46.283 INFO:teuthology.orchestra.run.smithi090.stdout:Reading package lists... 2023-11-05T08:17:46.301 DEBUG:teuthology.orchestra.run.smithi090:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-11-05T08:17:46.355 INFO:teuthology.orchestra.run.smithi090.stdout:Reading package lists... 2023-11-05T08:17:46.425 INFO:teuthology.orchestra.run.smithi002.stdout:Building dependency tree... 2023-11-05T08:17:46.425 INFO:teuthology.orchestra.run.smithi002.stdout:Reading state information... 2023-11-05T08:17:46.433 INFO:teuthology.orchestra.run.smithi069.stdout:3 upgraded, 5 newly installed, 0 to remove and 270 not upgraded. 2023-11-05T08:17:46.433 INFO:teuthology.orchestra.run.smithi069.stdout:Need to get 113 MB of archives. 2023-11-05T08:17:46.433 INFO:teuthology.orchestra.run.smithi069.stdout:After this operation, 582 MB of additional disk space will be used. 2023-11-05T08:17:46.433 INFO:teuthology.orchestra.run.smithi069.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-88-generic amd64 5.15.0-88.98 [22.4 MB] 2023-11-05T08:17:46.523 INFO:teuthology.orchestra.run.smithi090.stdout:Building dependency tree... 2023-11-05T08:17:46.524 INFO:teuthology.orchestra.run.smithi090.stdout:Reading state information... 2023-11-05T08:17:46.594 INFO:teuthology.orchestra.run.smithi002.stdout:The following packages were automatically installed and are no longer required: 2023-11-05T08:17:46.594 INFO:teuthology.orchestra.run.smithi002.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-11-05T08:17:46.595 INFO:teuthology.orchestra.run.smithi002.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-11-05T08:17:46.595 INFO:teuthology.orchestra.run.smithi002.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-11-05T08:17:46.595 INFO:teuthology.orchestra.run.smithi002.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-11-05T08:17:46.595 INFO:teuthology.orchestra.run.smithi002.stdout: python2.7-minimal 2023-11-05T08:17:46.595 INFO:teuthology.orchestra.run.smithi002.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-05T08:17:46.595 INFO:teuthology.orchestra.run.smithi002.stdout:The following additional packages will be installed: 2023-11-05T08:17:46.596 INFO:teuthology.orchestra.run.smithi002.stdout: linux-generic linux-headers-5.15.0-88 linux-headers-5.15.0-88-generic 2023-11-05T08:17:46.596 INFO:teuthology.orchestra.run.smithi002.stdout: linux-headers-generic linux-image-5.15.0-88-generic 2023-11-05T08:17:46.596 INFO:teuthology.orchestra.run.smithi002.stdout: linux-modules-5.15.0-88-generic linux-modules-extra-5.15.0-88-generic 2023-11-05T08:17:46.597 INFO:teuthology.orchestra.run.smithi002.stdout:Suggested packages: 2023-11-05T08:17:46.597 INFO:teuthology.orchestra.run.smithi002.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-11-05T08:17:46.597 INFO:teuthology.orchestra.run.smithi002.stdout:Recommended packages: 2023-11-05T08:17:46.597 INFO:teuthology.orchestra.run.smithi002.stdout: thermald 2023-11-05T08:17:46.647 INFO:teuthology.orchestra.run.smithi002.stdout:The following NEW packages will be installed: 2023-11-05T08:17:46.647 INFO:teuthology.orchestra.run.smithi002.stdout: linux-headers-5.15.0-88 linux-headers-5.15.0-88-generic 2023-11-05T08:17:46.647 INFO:teuthology.orchestra.run.smithi002.stdout: linux-image-5.15.0-88-generic linux-modules-5.15.0-88-generic 2023-11-05T08:17:46.648 INFO:teuthology.orchestra.run.smithi002.stdout: linux-modules-extra-5.15.0-88-generic 2023-11-05T08:17:46.649 INFO:teuthology.orchestra.run.smithi002.stdout:The following packages will be upgraded: 2023-11-05T08:17:46.649 INFO:teuthology.orchestra.run.smithi002.stdout: linux-generic linux-headers-generic linux-image-generic 2023-11-05T08:17:46.665 INFO:teuthology.orchestra.run.smithi090.stdout:The following packages were automatically installed and are no longer required: 2023-11-05T08:17:46.665 INFO:teuthology.orchestra.run.smithi090.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-11-05T08:17:46.666 INFO:teuthology.orchestra.run.smithi090.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-11-05T08:17:46.666 INFO:teuthology.orchestra.run.smithi090.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-11-05T08:17:46.666 INFO:teuthology.orchestra.run.smithi090.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-11-05T08:17:46.667 INFO:teuthology.orchestra.run.smithi090.stdout: python2.7-minimal 2023-11-05T08:17:46.667 INFO:teuthology.orchestra.run.smithi090.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-05T08:17:46.667 INFO:teuthology.orchestra.run.smithi090.stdout:The following additional packages will be installed: 2023-11-05T08:17:46.668 INFO:teuthology.orchestra.run.smithi090.stdout: linux-generic linux-headers-5.15.0-88 linux-headers-5.15.0-88-generic 2023-11-05T08:17:46.668 INFO:teuthology.orchestra.run.smithi090.stdout: linux-headers-generic linux-image-5.15.0-88-generic 2023-11-05T08:17:46.668 INFO:teuthology.orchestra.run.smithi090.stdout: linux-modules-5.15.0-88-generic linux-modules-extra-5.15.0-88-generic 2023-11-05T08:17:46.669 INFO:teuthology.orchestra.run.smithi090.stdout:Suggested packages: 2023-11-05T08:17:46.669 INFO:teuthology.orchestra.run.smithi090.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-11-05T08:17:46.669 INFO:teuthology.orchestra.run.smithi090.stdout:Recommended packages: 2023-11-05T08:17:46.669 INFO:teuthology.orchestra.run.smithi090.stdout: thermald 2023-11-05T08:17:46.699 INFO:teuthology.orchestra.run.smithi002.stdout:3 upgraded, 5 newly installed, 0 to remove and 270 not upgraded. 2023-11-05T08:17:46.700 INFO:teuthology.orchestra.run.smithi002.stdout:Need to get 113 MB of archives. 2023-11-05T08:17:46.700 INFO:teuthology.orchestra.run.smithi002.stdout:After this operation, 582 MB of additional disk space will be used. 2023-11-05T08:17:46.700 INFO:teuthology.orchestra.run.smithi002.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-88-generic amd64 5.15.0-88.98 [22.4 MB] 2023-11-05T08:17:46.712 INFO:teuthology.orchestra.run.smithi090.stdout:The following NEW packages will be installed: 2023-11-05T08:17:46.712 INFO:teuthology.orchestra.run.smithi090.stdout: linux-headers-5.15.0-88 linux-headers-5.15.0-88-generic 2023-11-05T08:17:46.712 INFO:teuthology.orchestra.run.smithi090.stdout: linux-image-5.15.0-88-generic linux-modules-5.15.0-88-generic 2023-11-05T08:17:46.713 INFO:teuthology.orchestra.run.smithi090.stdout: linux-modules-extra-5.15.0-88-generic 2023-11-05T08:17:46.713 INFO:teuthology.orchestra.run.smithi090.stdout:The following packages will be upgraded: 2023-11-05T08:17:46.714 INFO:teuthology.orchestra.run.smithi090.stdout: linux-generic linux-headers-generic linux-image-generic 2023-11-05T08:17:46.767 INFO:teuthology.orchestra.run.smithi090.stdout:3 upgraded, 5 newly installed, 0 to remove and 270 not upgraded. 2023-11-05T08:17:46.768 INFO:teuthology.orchestra.run.smithi090.stdout:Need to get 113 MB of archives. 2023-11-05T08:17:46.768 INFO:teuthology.orchestra.run.smithi090.stdout:After this operation, 582 MB of additional disk space will be used. 2023-11-05T08:17:46.768 INFO:teuthology.orchestra.run.smithi090.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-88-generic amd64 5.15.0-88.98 [22.4 MB] 2023-11-05T08:17:46.998 INFO:teuthology.orchestra.run.smithi053.stdout:Reading package lists... 2023-11-05T08:17:47.014 DEBUG:teuthology.orchestra.run.smithi053:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-11-05T08:17:47.076 INFO:teuthology.orchestra.run.smithi053.stdout:Reading package lists... 2023-11-05T08:17:47.102 INFO:teuthology.orchestra.run.smithi002.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-88-generic amd64 5.15.0-88.98 [11.5 MB] 2023-11-05T08:17:47.198 INFO:teuthology.orchestra.run.smithi002.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-88-generic amd64 5.15.0-88.98 [64.0 MB] 2023-11-05T08:17:47.224 INFO:teuthology.orchestra.run.smithi090.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-88-generic amd64 5.15.0-88.98 [11.5 MB] 2023-11-05T08:17:47.262 INFO:teuthology.orchestra.run.smithi053.stdout:Building dependency tree... 2023-11-05T08:17:47.263 INFO:teuthology.orchestra.run.smithi053.stdout:Reading state information... 2023-11-05T08:17:47.337 INFO:teuthology.orchestra.run.smithi090.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-88-generic amd64 5.15.0-88.98 [64.0 MB] 2023-11-05T08:17:47.429 INFO:teuthology.orchestra.run.smithi053.stdout:The following packages were automatically installed and are no longer required: 2023-11-05T08:17:47.429 INFO:teuthology.orchestra.run.smithi053.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-11-05T08:17:47.429 INFO:teuthology.orchestra.run.smithi053.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-11-05T08:17:47.430 INFO:teuthology.orchestra.run.smithi053.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-11-05T08:17:47.430 INFO:teuthology.orchestra.run.smithi053.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-11-05T08:17:47.430 INFO:teuthology.orchestra.run.smithi053.stdout: python2.7-minimal 2023-11-05T08:17:47.431 INFO:teuthology.orchestra.run.smithi053.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-05T08:17:47.431 INFO:teuthology.orchestra.run.smithi053.stdout:The following additional packages will be installed: 2023-11-05T08:17:47.432 INFO:teuthology.orchestra.run.smithi053.stdout: linux-generic linux-headers-5.15.0-88 linux-headers-5.15.0-88-generic 2023-11-05T08:17:47.432 INFO:teuthology.orchestra.run.smithi053.stdout: linux-headers-generic linux-image-5.15.0-88-generic 2023-11-05T08:17:47.432 INFO:teuthology.orchestra.run.smithi053.stdout: linux-modules-5.15.0-88-generic linux-modules-extra-5.15.0-88-generic 2023-11-05T08:17:47.433 INFO:teuthology.orchestra.run.smithi053.stdout:Suggested packages: 2023-11-05T08:17:47.433 INFO:teuthology.orchestra.run.smithi053.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-11-05T08:17:47.433 INFO:teuthology.orchestra.run.smithi053.stdout:Recommended packages: 2023-11-05T08:17:47.433 INFO:teuthology.orchestra.run.smithi053.stdout: thermald 2023-11-05T08:17:47.489 INFO:teuthology.orchestra.run.smithi053.stdout:The following NEW packages will be installed: 2023-11-05T08:17:47.490 INFO:teuthology.orchestra.run.smithi053.stdout: linux-headers-5.15.0-88 linux-headers-5.15.0-88-generic 2023-11-05T08:17:47.490 INFO:teuthology.orchestra.run.smithi053.stdout: linux-image-5.15.0-88-generic linux-modules-5.15.0-88-generic 2023-11-05T08:17:47.491 INFO:teuthology.orchestra.run.smithi053.stdout: linux-modules-extra-5.15.0-88-generic 2023-11-05T08:17:47.491 INFO:teuthology.orchestra.run.smithi053.stdout:The following packages will be upgraded: 2023-11-05T08:17:47.492 INFO:teuthology.orchestra.run.smithi053.stdout: linux-generic linux-headers-generic linux-image-generic 2023-11-05T08:17:47.672 INFO:teuthology.orchestra.run.smithi053.stdout:3 upgraded, 5 newly installed, 0 to remove and 270 not upgraded. 2023-11-05T08:17:47.672 INFO:teuthology.orchestra.run.smithi053.stdout:Need to get 113 MB of archives. 2023-11-05T08:17:47.673 INFO:teuthology.orchestra.run.smithi053.stdout:After this operation, 582 MB of additional disk space will be used. 2023-11-05T08:17:47.673 INFO:teuthology.orchestra.run.smithi053.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-88-generic amd64 5.15.0-88.98 [22.4 MB] 2023-11-05T08:17:47.929 INFO:teuthology.orchestra.run.smithi069.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-88-generic amd64 5.15.0-88.98 [11.5 MB] 2023-11-05T08:17:48.011 INFO:teuthology.orchestra.run.smithi002.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.88.85 [1,698 B] 2023-11-05T08:17:48.011 INFO:teuthology.orchestra.run.smithi002.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.88.85 [2,488 B] 2023-11-05T08:17:48.011 INFO:teuthology.orchestra.run.smithi002.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-88 all 5.15.0-88.98 [12.3 MB] 2023-11-05T08:17:48.142 INFO:teuthology.orchestra.run.smithi002.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-88-generic amd64 5.15.0-88.98 [2,878 kB] 2023-11-05T08:17:48.171 INFO:teuthology.orchestra.run.smithi002.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.88.85 [2,340 B] 2023-11-05T08:17:48.258 INFO:teuthology.orchestra.run.smithi090.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.88.85 [1,698 B] 2023-11-05T08:17:48.258 INFO:teuthology.orchestra.run.smithi090.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.88.85 [2,488 B] 2023-11-05T08:17:48.259 INFO:teuthology.orchestra.run.smithi090.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-88 all 5.15.0-88.98 [12.3 MB] 2023-11-05T08:17:48.274 INFO:teuthology.orchestra.run.smithi069.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-88-generic amd64 5.15.0-88.98 [64.0 MB] 2023-11-05T08:17:48.380 INFO:teuthology.orchestra.run.smithi090.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-88-generic amd64 5.15.0-88.98 [2,878 kB] 2023-11-05T08:17:48.402 INFO:teuthology.orchestra.run.smithi090.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.88.85 [2,340 B] 2023-11-05T08:17:48.664 INFO:teuthology.orchestra.run.smithi002.stdout:Fetched 113 MB in 2s (71.6 MB/s) 2023-11-05T08:17:48.775 INFO:teuthology.orchestra.run.smithi090.stdout:Fetched 113 MB in 2s (67.2 MB/s) 2023-11-05T08:17:48.841 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package linux-modules-5.15.0-88-generic. 2023-11-05T08:17:48.941 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package linux-modules-5.15.0-88-generic. 2023-11-05T08:17:49.084 INFO:teuthology.orchestra.run.smithi053.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-88-generic amd64 5.15.0-88.98 [11.5 MB] 2023-11-05T08:17:49.440 INFO:teuthology.orchestra.run.smithi053.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-88-generic amd64 5.15.0-88.98 [64.0 MB] 2023-11-05T08:17:50.411 INFO:teuthology.orchestra.run.smithi069.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.88.85 [1,698 B] 2023-11-05T08:17:50.411 INFO:teuthology.orchestra.run.smithi069.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.88.85 [2,488 B] 2023-11-05T08:17:50.412 INFO:teuthology.orchestra.run.smithi069.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-88 all 5.15.0-88.98 [12.3 MB] 2023-11-05T08:17:50.854 INFO:teuthology.orchestra.run.smithi002.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-11-05T08:17:50.858 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../0-linux-modules-5.15.0-88-generic_5.15.0-88.98_amd64.deb ... 2023-11-05T08:17:50.995 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking linux-modules-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-05T08:17:51.308 INFO:teuthology.orchestra.run.smithi069.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-88-generic amd64 5.15.0-88.98 [2,878 kB] 2023-11-05T08:17:51.445 INFO:teuthology.orchestra.run.smithi069.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.88.85 [2,340 B] 2023-11-05T08:17:51.488 INFO:teuthology.orchestra.run.smithi090.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-11-05T08:17:51.495 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../0-linux-modules-5.15.0-88-generic_5.15.0-88.98_amd64.deb ... 2023-11-05T08:17:51.552 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking linux-modules-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-05T08:17:51.634 INFO:teuthology.orchestra.run.smithi053.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.88.85 [1,698 B] 2023-11-05T08:17:51.635 INFO:teuthology.orchestra.run.smithi053.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.88.85 [2,488 B] 2023-11-05T08:17:51.702 INFO:teuthology.orchestra.run.smithi053.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-88 all 5.15.0-88.98 [12.3 MB] 2023-11-05T08:17:51.855 INFO:teuthology.orchestra.run.smithi069.stdout:Fetched 113 MB in 5s (21.8 MB/s) 2023-11-05T08:17:52.057 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package linux-modules-5.15.0-88-generic. 2023-11-05T08:17:52.369 INFO:teuthology.orchestra.run.smithi053.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-88-generic amd64 5.15.0-88.98 [2,878 kB] 2023-11-05T08:17:52.485 INFO:teuthology.orchestra.run.smithi053.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.88.85 [2,340 B] 2023-11-05T08:17:52.890 INFO:teuthology.orchestra.run.smithi053.stdout:Fetched 113 MB in 5s (22.7 MB/s) 2023-11-05T08:17:53.055 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package linux-modules-5.15.0-88-generic. 2023-11-05T08:17:53.212 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package linux-image-5.15.0-88-generic. 2023-11-05T08:17:53.234 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../1-linux-image-5.15.0-88-generic_5.15.0-88.98_amd64.deb ... 2023-11-05T08:17:53.349 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking linux-image-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-05T08:17:53.866 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-88-generic. 2023-11-05T08:17:53.886 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-88-generic_5.15.0-88.98_amd64.deb ... 2023-11-05T08:17:53.932 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking linux-modules-extra-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-05T08:17:54.067 INFO:teuthology.orchestra.run.smithi069.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-11-05T08:17:54.071 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../0-linux-modules-5.15.0-88-generic_5.15.0-88.98_amd64.deb ... 2023-11-05T08:17:54.129 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking linux-modules-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-05T08:17:54.485 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package linux-image-5.15.0-88-generic. 2023-11-05T08:17:54.506 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../1-linux-image-5.15.0-88-generic_5.15.0-88.98_amd64.deb ... 2023-11-05T08:17:54.602 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking linux-image-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-05T08:17:55.066 INFO:teuthology.orchestra.run.smithi053.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-11-05T08:17:55.071 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../0-linux-modules-5.15.0-88-generic_5.15.0-88.98_amd64.deb ... 2023-11-05T08:17:55.162 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking linux-modules-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-05T08:17:55.747 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-88-generic. 2023-11-05T08:17:55.767 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-88-generic_5.15.0-88.98_amd64.deb ... 2023-11-05T08:17:55.805 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking linux-modules-extra-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-05T08:17:56.204 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package linux-image-5.15.0-88-generic. 2023-11-05T08:17:56.226 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../1-linux-image-5.15.0-88-generic_5.15.0-88.98_amd64.deb ... 2023-11-05T08:17:56.328 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking linux-image-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-05T08:17:56.749 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-88-generic. 2023-11-05T08:17:56.769 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-88-generic_5.15.0-88.98_amd64.deb ... 2023-11-05T08:17:56.807 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking linux-modules-extra-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-05T08:17:57.680 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package linux-image-5.15.0-88-generic. 2023-11-05T08:17:57.702 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../1-linux-image-5.15.0-88-generic_5.15.0-88.98_amd64.deb ... 2023-11-05T08:17:57.798 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking linux-image-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-05T08:17:58.284 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-88-generic. 2023-11-05T08:17:58.305 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-88-generic_5.15.0-88.98_amd64.deb ... 2023-11-05T08:17:58.342 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking linux-modules-extra-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-05T08:18:00.986 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../3-linux-generic_5.15.0.88.85_amd64.deb ... 2023-11-05T08:18:01.135 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking linux-generic (5.15.0.88.85) over (5.15.0.56.54) ... 2023-11-05T08:18:01.486 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.88.85_amd64.deb ... 2023-11-05T08:18:01.588 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking linux-image-generic (5.15.0.88.85) over (5.15.0.56.54) ... 2023-11-05T08:18:01.900 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package linux-headers-5.15.0-88. 2023-11-05T08:18:01.922 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../5-linux-headers-5.15.0-88_5.15.0-88.98_all.deb ... 2023-11-05T08:18:02.000 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking linux-headers-5.15.0-88 (5.15.0-88.98) ... 2023-11-05T08:18:03.042 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../3-linux-generic_5.15.0.88.85_amd64.deb ... 2023-11-05T08:18:03.148 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking linux-generic (5.15.0.88.85) over (5.15.0.56.54) ... 2023-11-05T08:18:03.464 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.88.85_amd64.deb ... 2023-11-05T08:18:03.585 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking linux-image-generic (5.15.0.88.85) over (5.15.0.56.54) ... 2023-11-05T08:18:03.879 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package linux-headers-5.15.0-88. 2023-11-05T08:18:03.902 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../5-linux-headers-5.15.0-88_5.15.0-88.98_all.deb ... 2023-11-05T08:18:03.946 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking linux-headers-5.15.0-88 (5.15.0-88.98) ... 2023-11-05T08:18:04.790 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../3-linux-generic_5.15.0.88.85_amd64.deb ... 2023-11-05T08:18:04.842 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../3-linux-generic_5.15.0.88.85_amd64.deb ... 2023-11-05T08:18:04.906 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking linux-generic (5.15.0.88.85) over (5.15.0.56.54) ... 2023-11-05T08:18:04.963 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking linux-generic (5.15.0.88.85) over (5.15.0.56.54) ... 2023-11-05T08:18:05.239 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.88.85_amd64.deb ... 2023-11-05T08:18:05.304 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.88.85_amd64.deb ... 2023-11-05T08:18:05.418 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking linux-image-generic (5.15.0.88.85) over (5.15.0.56.54) ... 2023-11-05T08:18:05.456 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking linux-image-generic (5.15.0.88.85) over (5.15.0.56.54) ... 2023-11-05T08:18:05.738 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package linux-headers-5.15.0-88. 2023-11-05T08:18:05.752 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../5-linux-headers-5.15.0-88_5.15.0-88.98_all.deb ... 2023-11-05T08:18:05.822 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking linux-headers-5.15.0-88 (5.15.0-88.98) ... 2023-11-05T08:18:05.896 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package linux-headers-5.15.0-88. 2023-11-05T08:18:05.919 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../5-linux-headers-5.15.0-88_5.15.0-88.98_all.deb ... 2023-11-05T08:18:05.962 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking linux-headers-5.15.0-88 (5.15.0-88.98) ... 2023-11-05T08:18:08.264 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package linux-headers-5.15.0-88-generic. 2023-11-05T08:18:08.301 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../6-linux-headers-5.15.0-88-generic_5.15.0-88.98_amd64.deb ... 2023-11-05T08:18:08.338 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking linux-headers-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-05T08:18:09.695 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package linux-headers-5.15.0-88-generic. 2023-11-05T08:18:09.731 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../6-linux-headers-5.15.0-88-generic_5.15.0-88.98_amd64.deb ... 2023-11-05T08:18:09.770 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking linux-headers-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-05T08:18:11.177 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.88.85_amd64.deb ... 2023-11-05T08:18:11.318 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking linux-headers-generic (5.15.0.88.85) over (5.15.0.56.54) ... 2023-11-05T08:18:11.352 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package linux-headers-5.15.0-88-generic. 2023-11-05T08:18:11.389 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../6-linux-headers-5.15.0-88-generic_5.15.0-88.98_amd64.deb ... 2023-11-05T08:18:11.427 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking linux-headers-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-05T08:18:11.685 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up linux-headers-5.15.0-88 (5.15.0-88.98) ... 2023-11-05T08:18:11.814 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up linux-headers-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-05T08:18:11.957 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up linux-headers-generic (5.15.0.88.85) ... 2023-11-05T08:18:12.084 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up linux-image-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-05T08:18:12.299 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.88.85_amd64.deb ... 2023-11-05T08:18:12.422 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking linux-headers-generic (5.15.0.88.85) over (5.15.0.56.54) ... 2023-11-05T08:18:12.771 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up linux-headers-5.15.0-88 (5.15.0-88.98) ... 2023-11-05T08:18:12.893 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up linux-headers-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-05T08:18:13.019 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up linux-headers-generic (5.15.0.88.85) ... 2023-11-05T08:18:13.146 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up linux-image-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-05T08:18:13.214 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package linux-headers-5.15.0-88-generic. 2023-11-05T08:18:13.250 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../6-linux-headers-5.15.0-88-generic_5.15.0-88.98_amd64.deb ... 2023-11-05T08:18:13.289 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking linux-headers-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-05T08:18:13.428 INFO:teuthology.orchestra.run.smithi002.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-11-05T08:18:13.428 INFO:teuthology.orchestra.run.smithi002.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-11-05T08:18:13.429 INFO:teuthology.orchestra.run.smithi002.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-88-generic 2023-11-05T08:18:13.429 INFO:teuthology.orchestra.run.smithi002.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-88-generic 2023-11-05T08:18:13.597 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up linux-modules-extra-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-05T08:18:14.516 INFO:teuthology.orchestra.run.smithi069.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-11-05T08:18:14.517 INFO:teuthology.orchestra.run.smithi069.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-11-05T08:18:14.517 INFO:teuthology.orchestra.run.smithi069.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-88-generic 2023-11-05T08:18:14.517 INFO:teuthology.orchestra.run.smithi069.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-88-generic 2023-11-05T08:18:14.742 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up linux-modules-extra-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-05T08:18:15.080 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up linux-modules-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-05T08:18:15.311 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.88.85_amd64.deb ... 2023-11-05T08:18:15.436 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking linux-headers-generic (5.15.0.88.85) over (5.15.0.56.54) ... 2023-11-05T08:18:15.770 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up linux-headers-5.15.0-88 (5.15.0-88.98) ... 2023-11-05T08:18:15.891 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up linux-headers-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-05T08:18:16.017 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up linux-headers-generic (5.15.0.88.85) ... 2023-11-05T08:18:16.126 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up linux-modules-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-05T08:18:16.144 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up linux-image-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-05T08:18:16.198 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.88.85_amd64.deb ... 2023-11-05T08:18:16.321 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking linux-headers-generic (5.15.0.88.85) over (5.15.0.56.54) ... 2023-11-05T08:18:16.515 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up linux-image-generic (5.15.0.88.85) ... 2023-11-05T08:18:16.658 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up linux-generic (5.15.0.88.85) ... 2023-11-05T08:18:16.673 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up linux-headers-5.15.0-88 (5.15.0-88.98) ... 2023-11-05T08:18:16.801 INFO:teuthology.orchestra.run.smithi002.stdout:Processing triggers for linux-image-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-05T08:18:16.803 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up linux-headers-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-05T08:18:16.857 INFO:teuthology.orchestra.run.smithi002.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-11-05T08:18:16.858 INFO:teuthology.orchestra.run.smithi002.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-88-generic 2023-11-05T08:18:16.929 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up linux-headers-generic (5.15.0.88.85) ... 2023-11-05T08:18:17.040 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up linux-image-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-05T08:18:17.515 INFO:teuthology.orchestra.run.smithi053.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-11-05T08:18:17.516 INFO:teuthology.orchestra.run.smithi053.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-11-05T08:18:17.516 INFO:teuthology.orchestra.run.smithi053.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-88-generic 2023-11-05T08:18:17.516 INFO:teuthology.orchestra.run.smithi053.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-88-generic 2023-11-05T08:18:17.519 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up linux-image-generic (5.15.0.88.85) ... 2023-11-05T08:18:17.645 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up linux-generic (5.15.0.88.85) ... 2023-11-05T08:18:17.666 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up linux-modules-extra-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-05T08:18:17.788 INFO:teuthology.orchestra.run.smithi069.stdout:Processing triggers for linux-image-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-05T08:18:17.849 INFO:teuthology.orchestra.run.smithi069.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-11-05T08:18:17.850 INFO:teuthology.orchestra.run.smithi069.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-88-generic 2023-11-05T08:18:18.335 INFO:teuthology.orchestra.run.smithi090.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-11-05T08:18:18.335 INFO:teuthology.orchestra.run.smithi090.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-11-05T08:18:18.335 INFO:teuthology.orchestra.run.smithi090.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-88-generic 2023-11-05T08:18:18.335 INFO:teuthology.orchestra.run.smithi090.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-88-generic 2023-11-05T08:18:18.490 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up linux-modules-extra-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-05T08:18:19.133 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up linux-modules-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-05T08:18:19.995 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up linux-modules-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-05T08:18:20.951 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up linux-image-generic (5.15.0.88.85) ... 2023-11-05T08:18:21.085 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up linux-generic (5.15.0.88.85) ... 2023-11-05T08:18:21.220 INFO:teuthology.orchestra.run.smithi053.stdout:Processing triggers for linux-image-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-05T08:18:21.270 INFO:teuthology.orchestra.run.smithi053.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-11-05T08:18:21.271 INFO:teuthology.orchestra.run.smithi053.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-88-generic 2023-11-05T08:18:22.235 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up linux-image-generic (5.15.0.88.85) ... 2023-11-05T08:18:22.361 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up linux-generic (5.15.0.88.85) ... 2023-11-05T08:18:22.489 INFO:teuthology.orchestra.run.smithi090.stdout:Processing triggers for linux-image-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-05T08:18:22.551 INFO:teuthology.orchestra.run.smithi090.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-11-05T08:18:22.551 INFO:teuthology.orchestra.run.smithi090.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-88-generic 2023-11-05T08:18:31.400 INFO:teuthology.orchestra.run.smithi069.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-11-05T08:18:31.401 INFO:teuthology.orchestra.run.smithi069.stdout:Sourcing file `/etc/default/grub' 2023-11-05T08:18:31.411 INFO:teuthology.orchestra.run.smithi069.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-11-05T08:18:31.438 INFO:teuthology.orchestra.run.smithi069.stdout:Generating grub configuration file ... 2023-11-05T08:18:31.678 INFO:teuthology.orchestra.run.smithi069.stdout:Found linux image: /boot/vmlinuz-5.15.0-88-generic 2023-11-05T08:18:31.697 INFO:teuthology.orchestra.run.smithi069.stdout:Found initrd image: /boot/initrd.img-5.15.0-88-generic 2023-11-05T08:18:31.838 INFO:teuthology.orchestra.run.smithi002.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-11-05T08:18:31.838 INFO:teuthology.orchestra.run.smithi002.stdout:Sourcing file `/etc/default/grub' 2023-11-05T08:18:31.861 INFO:teuthology.orchestra.run.smithi002.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-11-05T08:18:31.884 INFO:teuthology.orchestra.run.smithi002.stdout:Generating grub configuration file ... 2023-11-05T08:18:31.990 INFO:teuthology.orchestra.run.smithi069.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-11-05T08:18:31.996 INFO:teuthology.orchestra.run.smithi069.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-11-05T08:18:32.070 INFO:teuthology.orchestra.run.smithi069.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-11-05T08:18:32.076 INFO:teuthology.orchestra.run.smithi069.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-11-05T08:18:32.187 INFO:teuthology.orchestra.run.smithi002.stdout:Found linux image: /boot/vmlinuz-5.15.0-88-generic 2023-11-05T08:18:32.210 INFO:teuthology.orchestra.run.smithi002.stdout:Found initrd image: /boot/initrd.img-5.15.0-88-generic 2023-11-05T08:18:32.247 INFO:teuthology.orchestra.run.smithi069.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-11-05T08:18:32.248 INFO:teuthology.orchestra.run.smithi069.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-11-05T08:18:32.248 INFO:teuthology.orchestra.run.smithi069.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-11-05T08:18:32.295 INFO:teuthology.orchestra.run.smithi069.stdout:done 2023-11-05T08:18:32.518 INFO:teuthology.orchestra.run.smithi002.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-11-05T08:18:32.524 INFO:teuthology.orchestra.run.smithi002.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-11-05T08:18:32.594 INFO:teuthology.orchestra.run.smithi002.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-11-05T08:18:32.600 INFO:teuthology.orchestra.run.smithi002.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-11-05T08:18:32.739 DEBUG:teuthology.orchestra.run.smithi069:> dpkg -s linux-image-generic 2023-11-05T08:18:32.756 INFO:teuthology.orchestra.run.smithi069.stdout:Package: linux-image-generic 2023-11-05T08:18:32.757 INFO:teuthology.orchestra.run.smithi069.stdout:Status: install ok installed 2023-11-05T08:18:32.757 INFO:teuthology.orchestra.run.smithi069.stdout:Priority: optional 2023-11-05T08:18:32.757 INFO:teuthology.orchestra.run.smithi069.stdout:Section: kernel 2023-11-05T08:18:32.757 INFO:teuthology.orchestra.run.smithi069.stdout:Installed-Size: 20 2023-11-05T08:18:32.757 INFO:teuthology.orchestra.run.smithi069.stdout:Maintainer: Ubuntu Kernel Team 2023-11-05T08:18:32.758 INFO:teuthology.orchestra.run.smithi069.stdout:Architecture: amd64 2023-11-05T08:18:32.758 INFO:teuthology.orchestra.run.smithi069.stdout:Source: linux-meta 2023-11-05T08:18:32.758 INFO:teuthology.orchestra.run.smithi069.stdout:Version: 5.15.0.88.85 2023-11-05T08:18:32.758 INFO:teuthology.orchestra.run.smithi069.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-88), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-11-05T08:18:32.759 INFO:teuthology.orchestra.run.smithi069.stdout:Depends: linux-image-5.15.0-88-generic, linux-modules-extra-5.15.0-88-generic, linux-firmware, intel-microcode, amd64-microcode 2023-11-05T08:18:32.759 INFO:teuthology.orchestra.run.smithi069.stdout:Recommends: thermald 2023-11-05T08:18:32.759 INFO:teuthology.orchestra.run.smithi069.stdout:Description: Generic Linux kernel image 2023-11-05T08:18:32.759 INFO:teuthology.orchestra.run.smithi069.stdout: This package will always depend on the latest generic kernel image 2023-11-05T08:18:32.759 INFO:teuthology.orchestra.run.smithi069.stdout: available. 2023-11-05T08:18:32.760 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-88-generic 2023-11-05T08:18:32.760 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-05T08:18:32.760 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-05T08:18:32.761 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2023-11-05T08:18:32.761 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi069.front.sepia.ceph.com, path=None, version=distro) 2023-11-05T08:18:32.761 DEBUG:teuthology.orchestra.run.smithi069:> sudo apt-get clean 2023-11-05T08:18:32.771 INFO:teuthology.orchestra.run.smithi002.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-11-05T08:18:32.771 INFO:teuthology.orchestra.run.smithi002.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-11-05T08:18:32.772 INFO:teuthology.orchestra.run.smithi002.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-11-05T08:18:32.814 INFO:teuthology.orchestra.run.smithi002.stdout:done 2023-11-05T08:18:32.901 DEBUG:teuthology.orchestra.run.smithi069:> sudo apt-get update 2023-11-05T08:18:33.017 INFO:teuthology.orchestra.run.smithi069.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-11-05T08:18:33.018 INFO:teuthology.orchestra.run.smithi069.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-11-05T08:18:33.025 INFO:teuthology.orchestra.run.smithi069.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-11-05T08:18:33.042 INFO:teuthology.orchestra.run.smithi069.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-11-05T08:18:33.317 DEBUG:teuthology.orchestra.run.smithi002:> dpkg -s linux-image-generic 2023-11-05T08:18:33.375 INFO:teuthology.orchestra.run.smithi002.stdout:Package: linux-image-generic 2023-11-05T08:18:33.375 INFO:teuthology.orchestra.run.smithi002.stdout:Status: install ok installed 2023-11-05T08:18:33.375 INFO:teuthology.orchestra.run.smithi002.stdout:Priority: optional 2023-11-05T08:18:33.376 INFO:teuthology.orchestra.run.smithi002.stdout:Section: kernel 2023-11-05T08:18:33.376 INFO:teuthology.orchestra.run.smithi002.stdout:Installed-Size: 20 2023-11-05T08:18:33.376 INFO:teuthology.orchestra.run.smithi002.stdout:Maintainer: Ubuntu Kernel Team 2023-11-05T08:18:33.377 INFO:teuthology.orchestra.run.smithi002.stdout:Architecture: amd64 2023-11-05T08:18:33.377 INFO:teuthology.orchestra.run.smithi002.stdout:Source: linux-meta 2023-11-05T08:18:33.377 INFO:teuthology.orchestra.run.smithi002.stdout:Version: 5.15.0.88.85 2023-11-05T08:18:33.377 INFO:teuthology.orchestra.run.smithi002.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-88), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-11-05T08:18:33.377 INFO:teuthology.orchestra.run.smithi002.stdout:Depends: linux-image-5.15.0-88-generic, linux-modules-extra-5.15.0-88-generic, linux-firmware, intel-microcode, amd64-microcode 2023-11-05T08:18:33.378 INFO:teuthology.orchestra.run.smithi002.stdout:Recommends: thermald 2023-11-05T08:18:33.378 INFO:teuthology.orchestra.run.smithi002.stdout:Description: Generic Linux kernel image 2023-11-05T08:18:33.378 INFO:teuthology.orchestra.run.smithi002.stdout: This package will always depend on the latest generic kernel image 2023-11-05T08:18:33.378 INFO:teuthology.orchestra.run.smithi002.stdout: available. 2023-11-05T08:18:33.379 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-88-generic 2023-11-05T08:18:33.379 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-05T08:18:33.379 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-05T08:18:33.379 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-11-05T08:18:33.379 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi002.front.sepia.ceph.com, path=None, version=distro) 2023-11-05T08:18:33.380 DEBUG:teuthology.orchestra.run.smithi002:> sudo apt-get clean 2023-11-05T08:18:33.524 DEBUG:teuthology.orchestra.run.smithi002:> sudo apt-get update 2023-11-05T08:18:33.766 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-11-05T08:18:33.767 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-11-05T08:18:33.838 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-11-05T08:18:33.919 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-11-05T08:18:34.333 INFO:teuthology.orchestra.run.smithi069.stdout:Reading package lists... 2023-11-05T08:18:34.350 DEBUG:teuthology.orchestra.run.smithi069:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-11-05T08:18:34.409 INFO:teuthology.orchestra.run.smithi069.stdout:Reading package lists... 2023-11-05T08:18:34.592 INFO:teuthology.orchestra.run.smithi069.stdout:Building dependency tree... 2023-11-05T08:18:34.593 INFO:teuthology.orchestra.run.smithi069.stdout:Reading state information... 2023-11-05T08:18:34.750 INFO:teuthology.orchestra.run.smithi069.stdout:linux-image-generic is already the newest version (5.15.0.88.85). 2023-11-05T08:18:34.750 INFO:teuthology.orchestra.run.smithi069.stdout:The following packages were automatically installed and are no longer required: 2023-11-05T08:18:34.751 INFO:teuthology.orchestra.run.smithi069.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-11-05T08:18:34.751 INFO:teuthology.orchestra.run.smithi069.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-11-05T08:18:34.751 INFO:teuthology.orchestra.run.smithi069.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-11-05T08:18:34.751 INFO:teuthology.orchestra.run.smithi069.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-11-05T08:18:34.752 INFO:teuthology.orchestra.run.smithi069.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-11-05T08:18:34.752 INFO:teuthology.orchestra.run.smithi069.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-05T08:18:34.788 INFO:teuthology.orchestra.run.smithi069.stdout:0 upgraded, 0 newly installed, 0 to remove and 270 not upgraded. 2023-11-05T08:18:34.790 DEBUG:teuthology.orchestra.run.smithi069:> dpkg -s linux-image-generic 2023-11-05T08:18:34.805 INFO:teuthology.orchestra.run.smithi069.stdout:Package: linux-image-generic 2023-11-05T08:18:34.806 INFO:teuthology.orchestra.run.smithi069.stdout:Status: install ok installed 2023-11-05T08:18:34.806 INFO:teuthology.orchestra.run.smithi069.stdout:Priority: optional 2023-11-05T08:18:34.806 INFO:teuthology.orchestra.run.smithi069.stdout:Section: kernel 2023-11-05T08:18:34.806 INFO:teuthology.orchestra.run.smithi069.stdout:Installed-Size: 20 2023-11-05T08:18:34.807 INFO:teuthology.orchestra.run.smithi069.stdout:Maintainer: Ubuntu Kernel Team 2023-11-05T08:18:34.807 INFO:teuthology.orchestra.run.smithi069.stdout:Architecture: amd64 2023-11-05T08:18:34.807 INFO:teuthology.orchestra.run.smithi069.stdout:Source: linux-meta 2023-11-05T08:18:34.807 INFO:teuthology.orchestra.run.smithi069.stdout:Version: 5.15.0.88.85 2023-11-05T08:18:34.808 INFO:teuthology.orchestra.run.smithi069.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-88), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-11-05T08:18:34.808 INFO:teuthology.orchestra.run.smithi069.stdout:Depends: linux-image-5.15.0-88-generic, linux-modules-extra-5.15.0-88-generic, linux-firmware, intel-microcode, amd64-microcode 2023-11-05T08:18:34.808 INFO:teuthology.orchestra.run.smithi069.stdout:Recommends: thermald 2023-11-05T08:18:34.808 INFO:teuthology.orchestra.run.smithi069.stdout:Description: Generic Linux kernel image 2023-11-05T08:18:34.808 INFO:teuthology.orchestra.run.smithi069.stdout: This package will always depend on the latest generic kernel image 2023-11-05T08:18:34.809 INFO:teuthology.orchestra.run.smithi069.stdout: available. 2023-11-05T08:18:34.809 DEBUG:teuthology.orchestra.run.smithi069:> mktemp 2023-11-05T08:18:34.850 INFO:teuthology.orchestra.run.smithi069.stdout:/tmp/tmp.NcQiNnxAJD 2023-11-05T08:18:34.851 DEBUG:teuthology.orchestra.run.smithi069:> sudo cp /boot/grub/grub.cfg /tmp/tmp.NcQiNnxAJD 2023-11-05T08:18:34.905 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0666 /tmp/tmp.NcQiNnxAJD 2023-11-05T08:18:35.030 DEBUG:teuthology.orchestra.remote:smithi069:/tmp/tmp.NcQiNnxAJD is 10KB 2023-11-05T08:18:35.042 DEBUG:teuthology.orchestra.run.smithi069:> rm -fr /tmp/tmp.NcQiNnxAJD 2023-11-05T08:18:35.049 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-11-05T08:18:35.101 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-05T08:18:35.101 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-11-05T08:18:35.102 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-11-05T08:18:35.125 INFO:teuthology.orchestra.run.smithi002.stdout:Reading package lists... 2023-11-05T08:18:35.142 DEBUG:teuthology.orchestra.run.smithi002:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-11-05T08:18:35.166 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-88-generic 2023-11-05T08:18:35.166 DEBUG:teuthology.orchestra.run.smithi069:> sudo update-grub 2023-11-05T08:18:35.206 INFO:teuthology.orchestra.run.smithi002.stdout:Reading package lists... 2023-11-05T08:18:35.397 INFO:teuthology.orchestra.run.smithi002.stdout:Building dependency tree... 2023-11-05T08:18:35.398 INFO:teuthology.orchestra.run.smithi002.stdout:Reading state information... 2023-11-05T08:18:35.551 INFO:teuthology.orchestra.run.smithi053.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-11-05T08:18:35.551 INFO:teuthology.orchestra.run.smithi053.stdout:Sourcing file `/etc/default/grub' 2023-11-05T08:18:35.560 INFO:teuthology.orchestra.run.smithi053.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-11-05T08:18:35.570 INFO:teuthology.orchestra.run.smithi002.stdout:linux-image-generic is already the newest version (5.15.0.88.85). 2023-11-05T08:18:35.570 INFO:teuthology.orchestra.run.smithi002.stdout:The following packages were automatically installed and are no longer required: 2023-11-05T08:18:35.571 INFO:teuthology.orchestra.run.smithi002.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-11-05T08:18:35.571 INFO:teuthology.orchestra.run.smithi002.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-11-05T08:18:35.571 INFO:teuthology.orchestra.run.smithi002.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-11-05T08:18:35.572 INFO:teuthology.orchestra.run.smithi002.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-11-05T08:18:35.572 INFO:teuthology.orchestra.run.smithi002.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-11-05T08:18:35.572 INFO:teuthology.orchestra.run.smithi002.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-05T08:18:35.574 INFO:teuthology.orchestra.run.smithi053.stdout:Generating grub configuration file ... 2023-11-05T08:18:35.614 INFO:teuthology.orchestra.run.smithi002.stdout:0 upgraded, 0 newly installed, 0 to remove and 270 not upgraded. 2023-11-05T08:18:35.616 DEBUG:teuthology.orchestra.run.smithi002:> dpkg -s linux-image-generic 2023-11-05T08:18:35.629 INFO:teuthology.orchestra.run.smithi002.stdout:Package: linux-image-generic 2023-11-05T08:18:35.629 INFO:teuthology.orchestra.run.smithi002.stdout:Status: install ok installed 2023-11-05T08:18:35.629 INFO:teuthology.orchestra.run.smithi002.stdout:Priority: optional 2023-11-05T08:18:35.629 INFO:teuthology.orchestra.run.smithi002.stdout:Section: kernel 2023-11-05T08:18:35.630 INFO:teuthology.orchestra.run.smithi002.stdout:Installed-Size: 20 2023-11-05T08:18:35.630 INFO:teuthology.orchestra.run.smithi002.stdout:Maintainer: Ubuntu Kernel Team 2023-11-05T08:18:35.630 INFO:teuthology.orchestra.run.smithi002.stdout:Architecture: amd64 2023-11-05T08:18:35.630 INFO:teuthology.orchestra.run.smithi002.stdout:Source: linux-meta 2023-11-05T08:18:35.630 INFO:teuthology.orchestra.run.smithi002.stdout:Version: 5.15.0.88.85 2023-11-05T08:18:35.631 INFO:teuthology.orchestra.run.smithi002.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-88), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-11-05T08:18:35.631 INFO:teuthology.orchestra.run.smithi002.stdout:Depends: linux-image-5.15.0-88-generic, linux-modules-extra-5.15.0-88-generic, linux-firmware, intel-microcode, amd64-microcode 2023-11-05T08:18:35.631 INFO:teuthology.orchestra.run.smithi002.stdout:Recommends: thermald 2023-11-05T08:18:35.631 INFO:teuthology.orchestra.run.smithi002.stdout:Description: Generic Linux kernel image 2023-11-05T08:18:35.631 INFO:teuthology.orchestra.run.smithi002.stdout: This package will always depend on the latest generic kernel image 2023-11-05T08:18:35.631 INFO:teuthology.orchestra.run.smithi002.stdout: available. 2023-11-05T08:18:35.632 DEBUG:teuthology.orchestra.run.smithi002:> mktemp 2023-11-05T08:18:35.675 INFO:teuthology.orchestra.run.smithi002.stdout:/tmp/tmp.5aKtoFP8pB 2023-11-05T08:18:35.676 DEBUG:teuthology.orchestra.run.smithi002:> sudo cp /boot/grub/grub.cfg /tmp/tmp.5aKtoFP8pB 2023-11-05T08:18:35.731 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0666 /tmp/tmp.5aKtoFP8pB 2023-11-05T08:18:35.845 INFO:teuthology.orchestra.run.smithi053.stdout:Found linux image: /boot/vmlinuz-5.15.0-88-generic 2023-11-05T08:18:35.855 DEBUG:teuthology.orchestra.remote:smithi002:/tmp/tmp.5aKtoFP8pB is 10KB 2023-11-05T08:18:35.865 INFO:teuthology.orchestra.run.smithi053.stdout:Found initrd image: /boot/initrd.img-5.15.0-88-generic 2023-11-05T08:18:35.870 DEBUG:teuthology.orchestra.run.smithi002:> rm -fr /tmp/tmp.5aKtoFP8pB 2023-11-05T08:18:35.876 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-11-05T08:18:35.932 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-05T08:18:35.932 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-11-05T08:18:35.933 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-11-05T08:18:36.013 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-88-generic 2023-11-05T08:18:36.013 DEBUG:teuthology.orchestra.run.smithi002:> sudo update-grub 2023-11-05T08:18:36.205 INFO:teuthology.orchestra.run.smithi053.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-11-05T08:18:36.211 INFO:teuthology.orchestra.run.smithi053.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-11-05T08:18:36.231 INFO:teuthology.orchestra.run.smithi069.stderr:Sourcing file `/etc/default/grub' 2023-11-05T08:18:36.232 INFO:teuthology.orchestra.run.smithi069.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-11-05T08:18:36.241 INFO:teuthology.orchestra.run.smithi069.stderr:Generating grub configuration file ... 2023-11-05T08:18:36.279 INFO:teuthology.orchestra.run.smithi053.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-11-05T08:18:36.285 INFO:teuthology.orchestra.run.smithi053.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-11-05T08:18:36.428 INFO:teuthology.orchestra.run.smithi053.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-11-05T08:18:36.429 INFO:teuthology.orchestra.run.smithi053.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-11-05T08:18:36.429 INFO:teuthology.orchestra.run.smithi053.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-11-05T08:18:36.436 INFO:teuthology.orchestra.run.smithi069.stderr:Found linux image: /boot/vmlinuz-5.15.0-88-generic 2023-11-05T08:18:36.442 INFO:teuthology.orchestra.run.smithi069.stderr:Found initrd image: /boot/initrd.img-5.15.0-88-generic 2023-11-05T08:18:36.472 INFO:teuthology.orchestra.run.smithi053.stdout:done 2023-11-05T08:18:36.688 INFO:teuthology.orchestra.run.smithi069.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-11-05T08:18:36.693 INFO:teuthology.orchestra.run.smithi069.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-11-05T08:18:36.728 INFO:teuthology.orchestra.run.smithi069.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-11-05T08:18:36.732 INFO:teuthology.orchestra.run.smithi069.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-11-05T08:18:36.849 INFO:teuthology.orchestra.run.smithi069.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-11-05T08:18:36.849 INFO:teuthology.orchestra.run.smithi069.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-11-05T08:18:36.849 INFO:teuthology.orchestra.run.smithi069.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-11-05T08:18:36.871 INFO:teuthology.orchestra.run.smithi069.stderr:done 2023-11-05T08:18:36.883 DEBUG:teuthology.orchestra.run.smithi069:> sudo shutdown -r now 2023-11-05T08:18:36.927 DEBUG:teuthology.orchestra.run.smithi053:> dpkg -s linux-image-generic 2023-11-05T08:18:36.953 INFO:teuthology.orchestra.run.smithi053.stdout:Package: linux-image-generic 2023-11-05T08:18:36.953 INFO:teuthology.orchestra.run.smithi053.stdout:Status: install ok installed 2023-11-05T08:18:36.953 INFO:teuthology.orchestra.run.smithi053.stdout:Priority: optional 2023-11-05T08:18:36.954 INFO:teuthology.orchestra.run.smithi053.stdout:Section: kernel 2023-11-05T08:18:36.954 INFO:teuthology.orchestra.run.smithi053.stdout:Installed-Size: 20 2023-11-05T08:18:36.954 INFO:teuthology.orchestra.run.smithi053.stdout:Maintainer: Ubuntu Kernel Team 2023-11-05T08:18:36.954 INFO:teuthology.orchestra.run.smithi053.stdout:Architecture: amd64 2023-11-05T08:18:36.954 INFO:teuthology.orchestra.run.smithi053.stdout:Source: linux-meta 2023-11-05T08:18:36.955 INFO:teuthology.orchestra.run.smithi053.stdout:Version: 5.15.0.88.85 2023-11-05T08:18:36.955 INFO:teuthology.orchestra.run.smithi053.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-88), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-11-05T08:18:36.955 INFO:teuthology.orchestra.run.smithi053.stdout:Depends: linux-image-5.15.0-88-generic, linux-modules-extra-5.15.0-88-generic, linux-firmware, intel-microcode, amd64-microcode 2023-11-05T08:18:36.955 INFO:teuthology.orchestra.run.smithi053.stdout:Recommends: thermald 2023-11-05T08:18:36.956 INFO:teuthology.orchestra.run.smithi053.stdout:Description: Generic Linux kernel image 2023-11-05T08:18:36.956 INFO:teuthology.orchestra.run.smithi053.stdout: This package will always depend on the latest generic kernel image 2023-11-05T08:18:36.956 INFO:teuthology.orchestra.run.smithi053.stdout: available. 2023-11-05T08:18:36.957 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-88-generic 2023-11-05T08:18:36.957 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-05T08:18:36.957 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-05T08:18:36.958 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-11-05T08:18:36.958 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi053.front.sepia.ceph.com, path=None, version=distro) 2023-11-05T08:18:36.958 DEBUG:teuthology.orchestra.run.smithi053:> sudo apt-get clean 2023-11-05T08:18:37.151 DEBUG:teuthology.orchestra.run.smithi053:> sudo apt-get update 2023-11-05T08:18:37.290 INFO:teuthology.orchestra.run.smithi053.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-11-05T08:18:37.311 INFO:teuthology.orchestra.run.smithi053.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-11-05T08:18:37.313 INFO:teuthology.orchestra.run.smithi002.stderr:Sourcing file `/etc/default/grub' 2023-11-05T08:18:37.314 INFO:teuthology.orchestra.run.smithi002.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-11-05T08:18:37.318 INFO:teuthology.orchestra.run.smithi002.stderr:Generating grub configuration file ... 2023-11-05T08:18:37.340 INFO:teuthology.orchestra.run.smithi053.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-11-05T08:18:37.512 INFO:teuthology.orchestra.run.smithi002.stderr:Found linux image: /boot/vmlinuz-5.15.0-88-generic 2023-11-05T08:18:37.518 INFO:teuthology.orchestra.run.smithi002.stderr:Found initrd image: /boot/initrd.img-5.15.0-88-generic 2023-11-05T08:18:37.764 INFO:teuthology.orchestra.run.smithi002.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-11-05T08:18:37.770 INFO:teuthology.orchestra.run.smithi002.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-11-05T08:18:37.813 INFO:teuthology.orchestra.run.smithi002.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-11-05T08:18:37.819 INFO:teuthology.orchestra.run.smithi002.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-11-05T08:18:37.937 INFO:teuthology.orchestra.run.smithi002.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-11-05T08:18:37.938 INFO:teuthology.orchestra.run.smithi002.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-11-05T08:18:37.938 INFO:teuthology.orchestra.run.smithi002.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-11-05T08:18:37.954 INFO:teuthology.orchestra.run.smithi090.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-11-05T08:18:37.954 INFO:teuthology.orchestra.run.smithi090.stdout:Sourcing file `/etc/default/grub' 2023-11-05T08:18:37.973 INFO:teuthology.orchestra.run.smithi002.stderr:done 2023-11-05T08:18:37.975 DEBUG:teuthology.orchestra.run.smithi002:> sudo shutdown -r now 2023-11-05T08:18:37.984 INFO:teuthology.orchestra.run.smithi090.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-11-05T08:18:38.000 INFO:teuthology.orchestra.run.smithi090.stdout:Generating grub configuration file ... 2023-11-05T08:18:38.349 INFO:teuthology.orchestra.run.smithi090.stdout:Found linux image: /boot/vmlinuz-5.15.0-88-generic 2023-11-05T08:18:38.385 INFO:teuthology.orchestra.run.smithi090.stdout:Found initrd image: /boot/initrd.img-5.15.0-88-generic 2023-11-05T08:18:38.762 INFO:teuthology.orchestra.run.smithi090.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-11-05T08:18:38.767 INFO:teuthology.orchestra.run.smithi090.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-11-05T08:18:38.838 INFO:teuthology.orchestra.run.smithi090.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-11-05T08:18:38.844 INFO:teuthology.orchestra.run.smithi090.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-11-05T08:18:39.089 INFO:teuthology.orchestra.run.smithi090.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-11-05T08:18:39.090 INFO:teuthology.orchestra.run.smithi090.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-11-05T08:18:39.090 INFO:teuthology.orchestra.run.smithi090.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-11-05T08:18:39.199 INFO:teuthology.orchestra.run.smithi090.stdout:done 2023-11-05T08:18:39.708 DEBUG:teuthology.orchestra.run.smithi090:> dpkg -s linux-image-generic 2023-11-05T08:18:39.734 INFO:teuthology.orchestra.run.smithi090.stdout:Package: linux-image-generic 2023-11-05T08:18:39.734 INFO:teuthology.orchestra.run.smithi090.stdout:Status: install ok installed 2023-11-05T08:18:39.735 INFO:teuthology.orchestra.run.smithi090.stdout:Priority: optional 2023-11-05T08:18:39.735 INFO:teuthology.orchestra.run.smithi090.stdout:Section: kernel 2023-11-05T08:18:39.735 INFO:teuthology.orchestra.run.smithi090.stdout:Installed-Size: 20 2023-11-05T08:18:39.735 INFO:teuthology.orchestra.run.smithi090.stdout:Maintainer: Ubuntu Kernel Team 2023-11-05T08:18:39.735 INFO:teuthology.orchestra.run.smithi090.stdout:Architecture: amd64 2023-11-05T08:18:39.736 INFO:teuthology.orchestra.run.smithi090.stdout:Source: linux-meta 2023-11-05T08:18:39.736 INFO:teuthology.orchestra.run.smithi090.stdout:Version: 5.15.0.88.85 2023-11-05T08:18:39.736 INFO:teuthology.orchestra.run.smithi090.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-88), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-11-05T08:18:39.736 INFO:teuthology.orchestra.run.smithi090.stdout:Depends: linux-image-5.15.0-88-generic, linux-modules-extra-5.15.0-88-generic, linux-firmware, intel-microcode, amd64-microcode 2023-11-05T08:18:39.736 INFO:teuthology.orchestra.run.smithi090.stdout:Recommends: thermald 2023-11-05T08:18:39.737 INFO:teuthology.orchestra.run.smithi090.stdout:Description: Generic Linux kernel image 2023-11-05T08:18:39.737 INFO:teuthology.orchestra.run.smithi090.stdout: This package will always depend on the latest generic kernel image 2023-11-05T08:18:39.737 INFO:teuthology.orchestra.run.smithi090.stdout: available. 2023-11-05T08:18:39.737 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-88-generic 2023-11-05T08:18:39.738 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-05T08:18:39.738 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-05T08:18:39.738 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2023-11-05T08:18:39.738 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi090.front.sepia.ceph.com, path=None, version=distro) 2023-11-05T08:18:39.739 DEBUG:teuthology.orchestra.run.smithi090:> sudo apt-get clean 2023-11-05T08:18:39.765 INFO:teuthology.orchestra.run.smithi053.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-11-05T08:18:39.903 DEBUG:teuthology.orchestra.run.smithi090:> sudo apt-get update 2023-11-05T08:18:40.043 INFO:teuthology.orchestra.run.smithi090.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-11-05T08:18:40.047 INFO:teuthology.orchestra.run.smithi090.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-11-05T08:18:40.064 INFO:teuthology.orchestra.run.smithi090.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-11-05T08:18:40.160 INFO:teuthology.orchestra.run.smithi090.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-11-05T08:18:40.846 INFO:teuthology.orchestra.run.smithi053.stdout:Reading package lists... 2023-11-05T08:18:40.863 DEBUG:teuthology.orchestra.run.smithi053:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-11-05T08:18:40.924 INFO:teuthology.orchestra.run.smithi053.stdout:Reading package lists... 2023-11-05T08:18:41.122 INFO:teuthology.orchestra.run.smithi053.stdout:Building dependency tree... 2023-11-05T08:18:41.123 INFO:teuthology.orchestra.run.smithi053.stdout:Reading state information... 2023-11-05T08:18:41.296 INFO:teuthology.orchestra.run.smithi053.stdout:linux-image-generic is already the newest version (5.15.0.88.85). 2023-11-05T08:18:41.296 INFO:teuthology.orchestra.run.smithi053.stdout:The following packages were automatically installed and are no longer required: 2023-11-05T08:18:41.296 INFO:teuthology.orchestra.run.smithi053.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-11-05T08:18:41.296 INFO:teuthology.orchestra.run.smithi053.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-11-05T08:18:41.297 INFO:teuthology.orchestra.run.smithi053.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-11-05T08:18:41.297 INFO:teuthology.orchestra.run.smithi053.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-11-05T08:18:41.297 INFO:teuthology.orchestra.run.smithi053.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-11-05T08:18:41.298 INFO:teuthology.orchestra.run.smithi053.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-05T08:18:41.327 INFO:teuthology.orchestra.run.smithi090.stdout:Reading package lists... 2023-11-05T08:18:41.338 INFO:teuthology.orchestra.run.smithi053.stdout:0 upgraded, 0 newly installed, 0 to remove and 270 not upgraded. 2023-11-05T08:18:41.340 DEBUG:teuthology.orchestra.run.smithi053:> dpkg -s linux-image-generic 2023-11-05T08:18:41.344 DEBUG:teuthology.orchestra.run.smithi090:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-11-05T08:18:41.355 INFO:teuthology.orchestra.run.smithi053.stdout:Package: linux-image-generic 2023-11-05T08:18:41.356 INFO:teuthology.orchestra.run.smithi053.stdout:Status: install ok installed 2023-11-05T08:18:41.356 INFO:teuthology.orchestra.run.smithi053.stdout:Priority: optional 2023-11-05T08:18:41.356 INFO:teuthology.orchestra.run.smithi053.stdout:Section: kernel 2023-11-05T08:18:41.356 INFO:teuthology.orchestra.run.smithi053.stdout:Installed-Size: 20 2023-11-05T08:18:41.357 INFO:teuthology.orchestra.run.smithi053.stdout:Maintainer: Ubuntu Kernel Team 2023-11-05T08:18:41.357 INFO:teuthology.orchestra.run.smithi053.stdout:Architecture: amd64 2023-11-05T08:18:41.357 INFO:teuthology.orchestra.run.smithi053.stdout:Source: linux-meta 2023-11-05T08:18:41.357 INFO:teuthology.orchestra.run.smithi053.stdout:Version: 5.15.0.88.85 2023-11-05T08:18:41.357 INFO:teuthology.orchestra.run.smithi053.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-88), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-11-05T08:18:41.358 INFO:teuthology.orchestra.run.smithi053.stdout:Depends: linux-image-5.15.0-88-generic, linux-modules-extra-5.15.0-88-generic, linux-firmware, intel-microcode, amd64-microcode 2023-11-05T08:18:41.358 INFO:teuthology.orchestra.run.smithi053.stdout:Recommends: thermald 2023-11-05T08:18:41.358 INFO:teuthology.orchestra.run.smithi053.stdout:Description: Generic Linux kernel image 2023-11-05T08:18:41.358 INFO:teuthology.orchestra.run.smithi053.stdout: This package will always depend on the latest generic kernel image 2023-11-05T08:18:41.359 INFO:teuthology.orchestra.run.smithi053.stdout: available. 2023-11-05T08:18:41.359 DEBUG:teuthology.orchestra.run.smithi053:> mktemp 2023-11-05T08:18:41.405 INFO:teuthology.orchestra.run.smithi090.stdout:Reading package lists... 2023-11-05T08:18:41.408 INFO:teuthology.orchestra.run.smithi053.stdout:/tmp/tmp.rEH1SJAqZs 2023-11-05T08:18:41.408 DEBUG:teuthology.orchestra.run.smithi053:> sudo cp /boot/grub/grub.cfg /tmp/tmp.rEH1SJAqZs 2023-11-05T08:18:41.465 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0666 /tmp/tmp.rEH1SJAqZs 2023-11-05T08:18:41.582 DEBUG:teuthology.orchestra.remote:smithi053:/tmp/tmp.rEH1SJAqZs is 10KB 2023-11-05T08:18:41.594 DEBUG:teuthology.orchestra.run.smithi053:> rm -fr /tmp/tmp.rEH1SJAqZs 2023-11-05T08:18:41.602 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-11-05T08:18:41.646 INFO:teuthology.orchestra.run.smithi090.stdout:Building dependency tree... 2023-11-05T08:18:41.647 INFO:teuthology.orchestra.run.smithi090.stdout:Reading state information... 2023-11-05T08:18:41.656 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-11-05T08:18:41.656 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-11-05T08:18:41.656 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-11-05T08:18:41.733 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-88-generic 2023-11-05T08:18:41.734 DEBUG:teuthology.orchestra.run.smithi053:> sudo update-grub 2023-11-05T08:18:41.810 INFO:teuthology.orchestra.run.smithi090.stdout:linux-image-generic is already the newest version (5.15.0.88.85). 2023-11-05T08:18:41.810 INFO:teuthology.orchestra.run.smithi090.stdout:The following packages were automatically installed and are no longer required: 2023-11-05T08:18:41.810 INFO:teuthology.orchestra.run.smithi090.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-11-05T08:18:41.810 INFO:teuthology.orchestra.run.smithi090.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-11-05T08:18:41.811 INFO:teuthology.orchestra.run.smithi090.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-11-05T08:18:41.811 INFO:teuthology.orchestra.run.smithi090.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-11-05T08:18:41.812 INFO:teuthology.orchestra.run.smithi090.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-11-05T08:18:41.812 INFO:teuthology.orchestra.run.smithi090.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-05T08:18:41.852 INFO:teuthology.orchestra.run.smithi090.stdout:0 upgraded, 0 newly installed, 0 to remove and 270 not upgraded. 2023-11-05T08:18:41.855 DEBUG:teuthology.orchestra.run.smithi090:> dpkg -s linux-image-generic 2023-11-05T08:18:41.867 INFO:teuthology.orchestra.run.smithi090.stdout:Package: linux-image-generic 2023-11-05T08:18:41.868 INFO:teuthology.orchestra.run.smithi090.stdout:Status: install ok installed 2023-11-05T08:18:41.868 INFO:teuthology.orchestra.run.smithi090.stdout:Priority: optional 2023-11-05T08:18:41.868 INFO:teuthology.orchestra.run.smithi090.stdout:Section: kernel 2023-11-05T08:18:41.868 INFO:teuthology.orchestra.run.smithi090.stdout:Installed-Size: 20 2023-11-05T08:18:41.869 INFO:teuthology.orchestra.run.smithi090.stdout:Maintainer: Ubuntu Kernel Team 2023-11-05T08:18:41.869 INFO:teuthology.orchestra.run.smithi090.stdout:Architecture: amd64 2023-11-05T08:18:41.869 INFO:teuthology.orchestra.run.smithi090.stdout:Source: linux-meta 2023-11-05T08:18:41.869 INFO:teuthology.orchestra.run.smithi090.stdout:Version: 5.15.0.88.85 2023-11-05T08:18:41.869 INFO:teuthology.orchestra.run.smithi090.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-88), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-11-05T08:18:41.869 INFO:teuthology.orchestra.run.smithi090.stdout:Depends: linux-image-5.15.0-88-generic, linux-modules-extra-5.15.0-88-generic, linux-firmware, intel-microcode, amd64-microcode 2023-11-05T08:18:41.870 INFO:teuthology.orchestra.run.smithi090.stdout:Recommends: thermald 2023-11-05T08:18:41.870 INFO:teuthology.orchestra.run.smithi090.stdout:Description: Generic Linux kernel image 2023-11-05T08:18:41.870 INFO:teuthology.orchestra.run.smithi090.stdout: This package will always depend on the latest generic kernel image 2023-11-05T08:18:41.870 INFO:teuthology.orchestra.run.smithi090.stdout: available. 2023-11-05T08:18:41.871 DEBUG:teuthology.orchestra.run.smithi090:> mktemp 2023-11-05T08:18:41.915 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/tmp.mbTnNCwJs6 2023-11-05T08:18:41.915 DEBUG:teuthology.orchestra.run.smithi090:> sudo cp /boot/grub/grub.cfg /tmp/tmp.mbTnNCwJs6 2023-11-05T08:18:41.974 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0666 /tmp/tmp.mbTnNCwJs6 2023-11-05T08:18:42.100 DEBUG:teuthology.orchestra.remote:smithi090:/tmp/tmp.mbTnNCwJs6 is 10KB 2023-11-05T08:18:42.114 DEBUG:teuthology.orchestra.run.smithi090:> rm -fr /tmp/tmp.mbTnNCwJs6 2023-11-05T08:18:42.119 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-11-05T08:18:42.175 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-11-05T08:18:42.175 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-11-05T08:18:42.176 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-11-05T08:18:42.270 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-88-generic 2023-11-05T08:18:42.271 DEBUG:teuthology.orchestra.run.smithi090:> sudo update-grub 2023-11-05T08:18:42.907 INFO:teuthology.orchestra.run.smithi053.stderr:Sourcing file `/etc/default/grub' 2023-11-05T08:18:42.908 INFO:teuthology.orchestra.run.smithi053.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-11-05T08:18:42.910 INFO:teuthology.orchestra.run.smithi053.stderr:Generating grub configuration file ... 2023-11-05T08:18:43.098 INFO:teuthology.orchestra.run.smithi053.stderr:Found linux image: /boot/vmlinuz-5.15.0-88-generic 2023-11-05T08:18:43.103 INFO:teuthology.orchestra.run.smithi053.stderr:Found initrd image: /boot/initrd.img-5.15.0-88-generic 2023-11-05T08:18:43.362 INFO:teuthology.orchestra.run.smithi053.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-11-05T08:18:43.368 INFO:teuthology.orchestra.run.smithi053.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-11-05T08:18:43.412 INFO:teuthology.orchestra.run.smithi053.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-11-05T08:18:43.417 INFO:teuthology.orchestra.run.smithi053.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-11-05T08:18:43.542 INFO:teuthology.orchestra.run.smithi053.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-11-05T08:18:43.542 INFO:teuthology.orchestra.run.smithi053.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-11-05T08:18:43.542 INFO:teuthology.orchestra.run.smithi053.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-11-05T08:18:43.582 INFO:teuthology.orchestra.run.smithi053.stderr:done 2023-11-05T08:18:43.600 DEBUG:teuthology.orchestra.run.smithi053:> sudo shutdown -r now 2023-11-05T08:18:43.955 INFO:teuthology.orchestra.run.smithi090.stderr:Sourcing file `/etc/default/grub' 2023-11-05T08:18:43.975 INFO:teuthology.orchestra.run.smithi090.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-11-05T08:18:43.983 INFO:teuthology.orchestra.run.smithi090.stderr:Generating grub configuration file ... 2023-11-05T08:18:44.240 INFO:teuthology.orchestra.run.smithi090.stderr:Found linux image: /boot/vmlinuz-5.15.0-88-generic 2023-11-05T08:18:44.248 INFO:teuthology.orchestra.run.smithi090.stderr:Found initrd image: /boot/initrd.img-5.15.0-88-generic 2023-11-05T08:18:44.676 INFO:teuthology.orchestra.run.smithi090.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-11-05T08:18:44.682 INFO:teuthology.orchestra.run.smithi090.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-11-05T08:18:44.725 INFO:teuthology.orchestra.run.smithi090.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-11-05T08:18:44.730 INFO:teuthology.orchestra.run.smithi090.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-11-05T08:18:44.858 INFO:teuthology.orchestra.run.smithi090.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-11-05T08:18:44.858 INFO:teuthology.orchestra.run.smithi090.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-11-05T08:18:44.858 INFO:teuthology.orchestra.run.smithi090.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-11-05T08:18:44.874 INFO:teuthology.orchestra.run.smithi090.stderr:done 2023-11-05T08:18:44.876 DEBUG:teuthology.orchestra.run.smithi090:> sudo shutdown -r now 2023-11-05T08:19:06.950 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2023-11-05T08:19:06.951 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2023-11-05T08:19:06.952 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-05T08:19:07.978 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-11-05T08:19:07.978 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2023-11-05T08:19:07.979 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-05T08:19:13.606 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-11-05T08:19:13.606 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2023-11-05T08:19:13.607 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-05T08:19:14.880 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2023-11-05T08:19:14.881 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-11-05T08:19:14.881 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-05T08:19:25.434 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.69 2023-11-05T08:19:26.458 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.2 2023-11-05T08:19:32.094 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.53 2023-11-05T08:19:33.374 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2023-11-05T08:19:34.437 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2023-11-05T08:19:34.438 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-05T08:19:35.460 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2023-11-05T08:19:35.461 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-05T08:19:37.498 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.69 2023-11-05T08:19:41.098 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2023-11-05T08:19:41.099 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-05T08:19:42.376 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-11-05T08:19:42.376 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-05T08:19:45.434 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2023-11-05T08:19:49.502 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2023-11-05T08:19:49.503 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-05T08:19:57.442 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-11-05T08:19:57.443 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-05T08:19:59.486 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.53 2023-11-05T08:20:00.510 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2023-11-05T08:20:11.498 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2023-11-05T08:20:11.499 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-05T08:20:13.403 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.53 2023-11-05T08:20:15.513 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-11-05T08:20:15.514 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-05T08:20:21.192 DEBUG:teuthology.orchestra.run.smithi069:> true 2023-11-05T08:20:21.729 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi069.front.sepia.ceph.com' 2023-11-05T08:20:21.730 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.15.0-88-generic"... 2023-11-05T08:20:21.730 DEBUG:teuthology.orchestra.run.smithi069:> uname -r 2023-11-05T08:20:21.775 INFO:teuthology.orchestra.run.smithi069.stdout:5.15.0-88-generic 2023-11-05T08:20:21.775 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-88-generic vs 5.15.0-88-generic 2023-11-05T08:20:21.776 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-05T08:20:21.776 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-11-05T08:20:22.777 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2023-11-05T08:20:22.778 DEBUG:teuthology.orchestra.run.smithi069:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-05T08:20:23.147 INFO:teuthology.orchestra.run.smithi069.stdout:ttyS1 2023-11-05T08:20:23.169 DEBUG:teuthology.parallel:result is None 2023-11-05T08:20:28.405 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2023-11-05T08:20:28.406 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-05T08:20:28.687 DEBUG:teuthology.orchestra.run.smithi053:> true 2023-11-05T08:20:29.378 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi053.front.sepia.ceph.com' 2023-11-05T08:20:29.379 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-88-generic"... 2023-11-05T08:20:29.379 DEBUG:teuthology.orchestra.run.smithi053:> uname -r 2023-11-05T08:20:29.426 INFO:teuthology.orchestra.run.smithi053.stdout:5.15.0-88-generic 2023-11-05T08:20:29.426 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-88-generic vs 5.15.0-88-generic 2023-11-05T08:20:29.426 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-05T08:20:29.427 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-11-05T08:20:30.427 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-11-05T08:20:30.428 DEBUG:teuthology.orchestra.run.smithi053:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-05T08:20:30.510 INFO:teuthology.orchestra.run.smithi053.stdout:ttyS1 2023-11-05T08:20:30.547 DEBUG:teuthology.parallel:result is None 2023-11-05T08:20:34.042 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2023-11-05T08:20:35.464 DEBUG:teuthology.orchestra.remote:timed out 2023-11-05T08:20:47.466 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2023-11-05T08:20:47.468 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-05T08:20:47.681 DEBUG:teuthology.orchestra.run.smithi002:> true 2023-11-05T08:20:48.227 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi002.front.sepia.ceph.com' 2023-11-05T08:20:48.228 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-88-generic"... 2023-11-05T08:20:48.228 DEBUG:teuthology.orchestra.run.smithi002:> uname -r 2023-11-05T08:20:48.274 INFO:teuthology.orchestra.run.smithi002.stdout:5.15.0-88-generic 2023-11-05T08:20:48.274 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-88-generic vs 5.15.0-88-generic 2023-11-05T08:20:48.274 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-05T08:20:48.275 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-11-05T08:20:49.276 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-11-05T08:20:49.277 DEBUG:teuthology.orchestra.run.smithi002:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-05T08:20:49.356 INFO:teuthology.orchestra.run.smithi002.stdout:ttyS1 2023-11-05T08:20:49.372 DEBUG:teuthology.parallel:result is None 2023-11-05T08:20:52.044 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-11-05T08:20:52.045 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-05T08:20:52.257 DEBUG:teuthology.orchestra.run.smithi090:> true 2023-11-05T08:20:52.854 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-11-05T08:20:52.855 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.15.0-88-generic"... 2023-11-05T08:20:52.855 DEBUG:teuthology.orchestra.run.smithi090:> uname -r 2023-11-05T08:20:52.902 INFO:teuthology.orchestra.run.smithi090.stdout:5.15.0-88-generic 2023-11-05T08:20:52.902 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-88-generic vs 5.15.0-88-generic 2023-11-05T08:20:52.902 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-05T08:20:52.903 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-11-05T08:20:53.904 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2023-11-05T08:20:53.904 DEBUG:teuthology.orchestra.run.smithi090:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-05T08:20:54.009 INFO:teuthology.orchestra.run.smithi090.stdout:ttyS1 2023-11-05T08:20:54.028 DEBUG:teuthology.parallel:result is None 2023-11-05T08:20:54.028 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-05T08:20:54.035 INFO:teuthology.task.internal:Creating test directory... 2023-11-05T08:20:54.036 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-05T08:20:54.038 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-05T08:20:54.041 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-05T08:20:54.047 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-05T08:20:54.052 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-05T08:20:54.059 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-05T08:20:54.070 INFO:teuthology.task.internal:Creating archive directory... 2023-11-05T08:20:54.070 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-05T08:20:54.091 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-05T08:20:54.093 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-05T08:20:54.096 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-05T08:20:54.304 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-05T08:20:54.311 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-05T08:20:54.311 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-05T08:20:54.314 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-05T08:20:54.316 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-05T08:20:54.319 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-05T08:20:54.330 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-05T08:20:54.335 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-05T08:20:54.337 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-05T08:20:54.338 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-05T08:20:54.340 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-05T08:20:54.343 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-05T08:20:54.346 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-05T08:20:54.347 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-05T08:20:54.349 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-05T08:20:54.361 INFO:teuthology.task.internal:Configuring sudo... 2023-11-05T08:20:54.362 DEBUG:teuthology.orchestra.run.smithi002:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-05T08:20:54.383 DEBUG:teuthology.orchestra.run.smithi053:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-05T08:20:54.390 DEBUG:teuthology.orchestra.run.smithi069:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-05T08:20:54.393 DEBUG:teuthology.orchestra.run.smithi090:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-05T08:20:54.408 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-05T08:20:54.418 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-05T08:20:54.418 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-05T08:20:54.439 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-05T08:20:54.451 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-05T08:20:54.454 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-05T08:20:54.460 DEBUG:teuthology.orchestra.run.smithi002:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-05T08:20:54.492 DEBUG:teuthology.orchestra.run.smithi002:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-05T08:20:54.538 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-05T08:20:54.539 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-05T08:20:54.600 DEBUG:teuthology.orchestra.run.smithi053:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-05T08:20:54.606 DEBUG:teuthology.orchestra.run.smithi053:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-05T08:20:54.654 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-11-05T08:20:54.654 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-05T08:20:54.715 DEBUG:teuthology.orchestra.run.smithi069:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-05T08:20:54.721 DEBUG:teuthology.orchestra.run.smithi069:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-05T08:20:54.767 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-05T08:20:54.767 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-05T08:20:54.832 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-05T08:20:54.838 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-05T08:20:54.886 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-11-05T08:20:54.887 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-05T08:20:54.956 DEBUG:teuthology.orchestra.run.smithi002:> sudo service rsyslog restart 2023-11-05T08:20:54.958 DEBUG:teuthology.orchestra.run.smithi053:> sudo service rsyslog restart 2023-11-05T08:20:54.961 DEBUG:teuthology.orchestra.run.smithi069:> sudo service rsyslog restart 2023-11-05T08:20:54.963 DEBUG:teuthology.orchestra.run.smithi090:> sudo service rsyslog restart 2023-11-05T08:20:55.065 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-05T08:20:55.072 INFO:teuthology.task.internal:Starting timer... 2023-11-05T08:20:55.073 INFO:teuthology.run_tasks:Running task pcp... 2023-11-05T08:20:55.082 INFO:teuthology.run_tasks:Running task selinux... 2023-11-05T08:20:55.089 DEBUG:teuthology.task.selinux:Excluding smithi002: OS 'ubuntu' does not support SELinux 2023-11-05T08:20:55.090 DEBUG:teuthology.task.selinux:Excluding smithi053: OS 'ubuntu' does not support SELinux 2023-11-05T08:20:55.090 DEBUG:teuthology.task.selinux:Excluding smithi069: OS 'ubuntu' does not support SELinux 2023-11-05T08:20:55.090 DEBUG:teuthology.task.selinux:Excluding smithi090: OS 'ubuntu' does not support SELinux 2023-11-05T08:20:55.090 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-05T08:20:55.091 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-11-05T08:20:55.091 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-05T08:20:55.091 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-05T08:20:55.099 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-11-05T08:20:55.105 INFO:teuthology.repo_utils:Fetching main from origin 2023-11-05T08:20:55.268 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-05T08:20:55.284 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-05T08:20:55.285 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi002.front.sepia.ceph.com,smithi053.front.sepia.ceph.com,smithi069.front.sepia.ceph.com,smithi090.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-05T08:28:49.773 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi002.front.sepia.ceph.com'), Remote(name='ubuntu@smithi053.front.sepia.ceph.com'), Remote(name='ubuntu@smithi069.front.sepia.ceph.com'), Remote(name='ubuntu@smithi090.front.sepia.ceph.com')] 2023-11-05T08:28:49.775 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2023-11-05T08:28:49.776 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-05T08:28:49.855 DEBUG:teuthology.orchestra.run.smithi002:> true 2023-11-05T08:28:49.962 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi002.front.sepia.ceph.com' 2023-11-05T08:28:49.963 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2023-11-05T08:28:49.963 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-05T08:28:50.041 DEBUG:teuthology.orchestra.run.smithi053:> true 2023-11-05T08:28:50.138 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi053.front.sepia.ceph.com' 2023-11-05T08:28:50.138 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2023-11-05T08:28:50.139 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-05T08:28:50.215 DEBUG:teuthology.orchestra.run.smithi069:> true 2023-11-05T08:28:50.318 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi069.front.sepia.ceph.com' 2023-11-05T08:28:50.318 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-11-05T08:28:50.319 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-05T08:28:50.396 DEBUG:teuthology.orchestra.run.smithi090:> true 2023-11-05T08:28:50.494 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-11-05T08:28:50.496 INFO:teuthology.run_tasks:Running task clock... 2023-11-05T08:28:50.506 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-05T08:28:50.506 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-05T08:28:50.507 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-05T08:28:50.509 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-05T08:28:50.510 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-05T08:28:50.512 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-05T08:28:50.513 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-05T08:28:50.515 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-05T08:28:50.515 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-05T08:28:50.537 INFO:teuthology.orchestra.run.smithi002.stdout: 5 Nov 08:28:50 ntpd[27882]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-11-05T08:28:50.537 INFO:teuthology.orchestra.run.smithi002.stdout: 5 Nov 08:28:50 ntpd[27882]: Command line: ntpd -gq 2023-11-05T08:28:50.537 INFO:teuthology.orchestra.run.smithi002.stdout: 5 Nov 08:28:50 ntpd[27882]: ---------------------------------------------------- 2023-11-05T08:28:50.537 INFO:teuthology.orchestra.run.smithi002.stdout: 5 Nov 08:28:50 ntpd[27882]: ntp-4 is maintained by Network Time Foundation, 2023-11-05T08:28:50.538 INFO:teuthology.orchestra.run.smithi002.stdout: 5 Nov 08:28:50 ntpd[27882]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-11-05T08:28:50.538 INFO:teuthology.orchestra.run.smithi002.stdout: 5 Nov 08:28:50 ntpd[27882]: corporation. Support and training for ntp-4 are 2023-11-05T08:28:50.538 INFO:teuthology.orchestra.run.smithi002.stdout: 5 Nov 08:28:50 ntpd[27882]: available at https://www.nwtime.org/support 2023-11-05T08:28:50.538 INFO:teuthology.orchestra.run.smithi002.stdout: 5 Nov 08:28:50 ntpd[27882]: ---------------------------------------------------- 2023-11-05T08:28:50.538 INFO:teuthology.orchestra.run.smithi002.stdout: 5 Nov 08:28:50 ntpd[27882]: proto: precision = 0.046 usec (-24) 2023-11-05T08:28:50.539 INFO:teuthology.orchestra.run.smithi002.stdout: 5 Nov 08:28:50 ntpd[27882]: basedate set to 2022-02-04 2023-11-05T08:28:50.539 INFO:teuthology.orchestra.run.smithi002.stdout: 5 Nov 08:28:50 ntpd[27882]: gps base set to 2022-02-06 (week 2196) 2023-11-05T08:28:50.540 INFO:teuthology.orchestra.run.smithi002.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-05T08:28:50.540 INFO:teuthology.orchestra.run.smithi002.stdout: 5 Nov 08:28:50 ntpd[27882]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-05T08:28:50.540 INFO:teuthology.orchestra.run.smithi002.stdout: 5 Nov 08:28:50 ntpd[27882]: restrict ::: KOD does nothing without LIMITED. 2023-11-05T08:28:50.541 INFO:teuthology.orchestra.run.smithi002.stderr:restrict ::: KOD does nothing without LIMITED. 2023-11-05T08:28:50.542 INFO:teuthology.orchestra.run.smithi002.stdout: 5 Nov 08:28:50 ntpd[27882]: Listen and drop on 0 v6wildcard [::]:123 2023-11-05T08:28:50.542 INFO:teuthology.orchestra.run.smithi002.stdout: 5 Nov 08:28:50 ntpd[27882]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-11-05T08:28:50.542 INFO:teuthology.orchestra.run.smithi002.stdout: 5 Nov 08:28:50 ntpd[27882]: Listen normally on 2 lo 127.0.0.1:123 2023-11-05T08:28:50.542 INFO:teuthology.orchestra.run.smithi002.stdout: 5 Nov 08:28:50 ntpd[27882]: Listen normally on 3 ens1f0 172.21.15.2:123 2023-11-05T08:28:50.542 INFO:teuthology.orchestra.run.smithi002.stdout: 5 Nov 08:28:50 ntpd[27882]: Listen normally on 4 lo [::1]:123 2023-11-05T08:28:50.543 INFO:teuthology.orchestra.run.smithi002.stdout: 5 Nov 08:28:50 ntpd[27882]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1920%4]:123 2023-11-05T08:28:50.543 INFO:teuthology.orchestra.run.smithi002.stdout: 5 Nov 08:28:50 ntpd[27882]: Listening on routing socket on fd #22 for interface updates 2023-11-05T08:28:50.545 INFO:teuthology.orchestra.run.smithi069.stdout: 5 Nov 08:28:50 ntpd[28050]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-11-05T08:28:50.545 INFO:teuthology.orchestra.run.smithi069.stdout: 5 Nov 08:28:50 ntpd[28050]: Command line: ntpd -gq 2023-11-05T08:28:50.545 INFO:teuthology.orchestra.run.smithi069.stdout: 5 Nov 08:28:50 ntpd[28050]: ---------------------------------------------------- 2023-11-05T08:28:50.545 INFO:teuthology.orchestra.run.smithi069.stdout: 5 Nov 08:28:50 ntpd[28050]: ntp-4 is maintained by Network Time Foundation, 2023-11-05T08:28:50.546 INFO:teuthology.orchestra.run.smithi069.stdout: 5 Nov 08:28:50 ntpd[28050]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-11-05T08:28:50.546 INFO:teuthology.orchestra.run.smithi069.stdout: 5 Nov 08:28:50 ntpd[28050]: corporation. Support and training for ntp-4 are 2023-11-05T08:28:50.546 INFO:teuthology.orchestra.run.smithi069.stdout: 5 Nov 08:28:50 ntpd[28050]: available at https://www.nwtime.org/support 2023-11-05T08:28:50.546 INFO:teuthology.orchestra.run.smithi069.stdout: 5 Nov 08:28:50 ntpd[28050]: ---------------------------------------------------- 2023-11-05T08:28:50.547 INFO:teuthology.orchestra.run.smithi069.stdout: 5 Nov 08:28:50 ntpd[28050]: proto: precision = 0.045 usec (-24) 2023-11-05T08:28:50.547 INFO:teuthology.orchestra.run.smithi069.stdout: 5 Nov 08:28:50 ntpd[28050]: basedate set to 2022-02-04 2023-11-05T08:28:50.547 INFO:teuthology.orchestra.run.smithi069.stdout: 5 Nov 08:28:50 ntpd[28050]: gps base set to 2022-02-06 (week 2196) 2023-11-05T08:28:50.547 INFO:teuthology.orchestra.run.smithi069.stdout: 5 Nov 08:28:50 ntpd[28050]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-05T08:28:50.547 INFO:teuthology.orchestra.run.smithi069.stdout: 5 Nov 08:28:50 ntpd[28050]: restrict ::: KOD does nothing without LIMITED. 2023-11-05T08:28:50.548 INFO:teuthology.orchestra.run.smithi053.stdout: 5 Nov 08:28:50 ntpd[27919]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-11-05T08:28:50.548 INFO:teuthology.orchestra.run.smithi053.stdout: 5 Nov 08:28:50 ntpd[27919]: Command line: ntpd -gq 2023-11-05T08:28:50.548 INFO:teuthology.orchestra.run.smithi053.stdout: 5 Nov 08:28:50 ntpd[27919]: ---------------------------------------------------- 2023-11-05T08:28:50.548 INFO:teuthology.orchestra.run.smithi053.stdout: 5 Nov 08:28:50 ntpd[27919]: ntp-4 is maintained by Network Time Foundation, 2023-11-05T08:28:50.548 INFO:teuthology.orchestra.run.smithi053.stdout: 5 Nov 08:28:50 ntpd[27919]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-11-05T08:28:50.548 INFO:teuthology.orchestra.run.smithi053.stdout: 5 Nov 08:28:50 ntpd[27919]: corporation. Support and training for ntp-4 are 2023-11-05T08:28:50.549 INFO:teuthology.orchestra.run.smithi053.stdout: 5 Nov 08:28:50 ntpd[27919]: available at https://www.nwtime.org/support 2023-11-05T08:28:50.549 INFO:teuthology.orchestra.run.smithi053.stdout: 5 Nov 08:28:50 ntpd[27919]: ---------------------------------------------------- 2023-11-05T08:28:50.549 INFO:teuthology.orchestra.run.smithi053.stdout: 5 Nov 08:28:50 ntpd[27919]: proto: precision = 0.045 usec (-24) 2023-11-05T08:28:50.549 INFO:teuthology.orchestra.run.smithi053.stdout: 5 Nov 08:28:50 ntpd[27919]: basedate set to 2022-02-04 2023-11-05T08:28:50.549 INFO:teuthology.orchestra.run.smithi053.stdout: 5 Nov 08:28:50 ntpd[27919]: gps base set to 2022-02-06 (week 2196) 2023-11-05T08:28:50.550 INFO:teuthology.orchestra.run.smithi053.stdout: 5 Nov 08:28:50 ntpd[27919]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-05T08:28:50.550 INFO:teuthology.orchestra.run.smithi069.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-05T08:28:50.550 INFO:teuthology.orchestra.run.smithi069.stderr:restrict ::: KOD does nothing without LIMITED. 2023-11-05T08:28:50.550 INFO:teuthology.orchestra.run.smithi053.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-05T08:28:50.551 INFO:teuthology.orchestra.run.smithi053.stderr:restrict ::: KOD does nothing without LIMITED. 2023-11-05T08:28:50.552 INFO:teuthology.orchestra.run.smithi069.stdout: 5 Nov 08:28:50 ntpd[28050]: Listen and drop on 0 v6wildcard [::]:123 2023-11-05T08:28:50.552 INFO:teuthology.orchestra.run.smithi069.stdout: 5 Nov 08:28:50 ntpd[28050]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-11-05T08:28:50.552 INFO:teuthology.orchestra.run.smithi069.stdout: 5 Nov 08:28:50 ntpd[28050]: Listen normally on 2 lo 127.0.0.1:123 2023-11-05T08:28:50.552 INFO:teuthology.orchestra.run.smithi069.stdout: 5 Nov 08:28:50 ntpd[28050]: Listen normally on 3 enp3s0f1 172.21.15.69:123 2023-11-05T08:28:50.553 INFO:teuthology.orchestra.run.smithi069.stdout: 5 Nov 08:28:50 ntpd[28050]: Listen normally on 4 lo [::1]:123 2023-11-05T08:28:50.553 INFO:teuthology.orchestra.run.smithi069.stdout: 5 Nov 08:28:50 ntpd[28050]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d737%5]:123 2023-11-05T08:28:50.553 INFO:teuthology.orchestra.run.smithi069.stdout: 5 Nov 08:28:50 ntpd[28050]: Listening on routing socket on fd #22 for interface updates 2023-11-05T08:28:50.553 INFO:teuthology.orchestra.run.smithi053.stdout: 5 Nov 08:28:50 ntpd[27919]: restrict ::: KOD does nothing without LIMITED. 2023-11-05T08:28:50.554 INFO:teuthology.orchestra.run.smithi053.stdout: 5 Nov 08:28:50 ntpd[27919]: Listen and drop on 0 v6wildcard [::]:123 2023-11-05T08:28:50.554 INFO:teuthology.orchestra.run.smithi053.stdout: 5 Nov 08:28:50 ntpd[27919]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-11-05T08:28:50.554 INFO:teuthology.orchestra.run.smithi053.stdout: 5 Nov 08:28:50 ntpd[27919]: Listen normally on 2 lo 127.0.0.1:123 2023-11-05T08:28:50.554 INFO:teuthology.orchestra.run.smithi053.stdout: 5 Nov 08:28:50 ntpd[27919]: Listen normally on 3 ens1f0 172.21.15.53:123 2023-11-05T08:28:50.554 INFO:teuthology.orchestra.run.smithi053.stdout: 5 Nov 08:28:50 ntpd[27919]: Listen normally on 4 lo [::1]:123 2023-11-05T08:28:50.555 INFO:teuthology.orchestra.run.smithi053.stdout: 5 Nov 08:28:50 ntpd[27919]: Listen normally on 5 ens1f0 [fe80::ae1f:6bff:fef8:3718%4]:123 2023-11-05T08:28:50.555 INFO:teuthology.orchestra.run.smithi053.stdout: 5 Nov 08:28:50 ntpd[27919]: Listening on routing socket on fd #22 for interface updates 2023-11-05T08:28:50.562 INFO:teuthology.orchestra.run.smithi090.stdout: 5 Nov 08:28:50 ntpd[28074]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-11-05T08:28:50.563 INFO:teuthology.orchestra.run.smithi090.stdout: 5 Nov 08:28:50 ntpd[28074]: Command line: ntpd -gq 2023-11-05T08:28:50.563 INFO:teuthology.orchestra.run.smithi090.stdout: 5 Nov 08:28:50 ntpd[28074]: ---------------------------------------------------- 2023-11-05T08:28:50.563 INFO:teuthology.orchestra.run.smithi090.stdout: 5 Nov 08:28:50 ntpd[28074]: ntp-4 is maintained by Network Time Foundation, 2023-11-05T08:28:50.563 INFO:teuthology.orchestra.run.smithi090.stdout: 5 Nov 08:28:50 ntpd[28074]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-11-05T08:28:50.563 INFO:teuthology.orchestra.run.smithi090.stdout: 5 Nov 08:28:50 ntpd[28074]: corporation. Support and training for ntp-4 are 2023-11-05T08:28:50.564 INFO:teuthology.orchestra.run.smithi090.stdout: 5 Nov 08:28:50 ntpd[28074]: available at https://www.nwtime.org/support 2023-11-05T08:28:50.564 INFO:teuthology.orchestra.run.smithi090.stdout: 5 Nov 08:28:50 ntpd[28074]: ---------------------------------------------------- 2023-11-05T08:28:50.565 INFO:teuthology.orchestra.run.smithi090.stdout: 5 Nov 08:28:50 ntpd[28074]: proto: precision = 0.046 usec (-24) 2023-11-05T08:28:50.565 INFO:teuthology.orchestra.run.smithi090.stdout: 5 Nov 08:28:50 ntpd[28074]: basedate set to 2022-02-04 2023-11-05T08:28:50.565 INFO:teuthology.orchestra.run.smithi090.stdout: 5 Nov 08:28:50 ntpd[28074]: gps base set to 2022-02-06 (week 2196) 2023-11-05T08:28:50.565 INFO:teuthology.orchestra.run.smithi090.stdout: 5 Nov 08:28:50 ntpd[28074]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-05T08:28:50.566 INFO:teuthology.orchestra.run.smithi090.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-05T08:28:50.566 INFO:teuthology.orchestra.run.smithi090.stderr:restrict ::: KOD does nothing without LIMITED. 2023-11-05T08:28:50.567 INFO:teuthology.orchestra.run.smithi090.stdout: 5 Nov 08:28:50 ntpd[28074]: restrict ::: KOD does nothing without LIMITED. 2023-11-05T08:28:50.567 INFO:teuthology.orchestra.run.smithi090.stdout: 5 Nov 08:28:50 ntpd[28074]: Listen and drop on 0 v6wildcard [::]:123 2023-11-05T08:28:50.567 INFO:teuthology.orchestra.run.smithi090.stdout: 5 Nov 08:28:50 ntpd[28074]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-11-05T08:28:50.567 INFO:teuthology.orchestra.run.smithi090.stdout: 5 Nov 08:28:50 ntpd[28074]: Listen normally on 2 lo 127.0.0.1:123 2023-11-05T08:28:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: 5 Nov 08:28:50 ntpd[28074]: Listen normally on 3 enp3s0f1 172.21.15.90:123 2023-11-05T08:28:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: 5 Nov 08:28:50 ntpd[28074]: Listen normally on 4 lo [::1]:123 2023-11-05T08:28:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: 5 Nov 08:28:50 ntpd[28074]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d767%5]:123 2023-11-05T08:28:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: 5 Nov 08:28:50 ntpd[28074]: Listening on routing socket on fd #22 for interface updates 2023-11-05T08:28:51.538 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:51 ntpd[27882]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:51.539 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:51 ntpd[27882]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:51.539 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:51 ntpd[27882]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:51.539 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:51 ntpd[27882]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:51.539 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:51 ntpd[27882]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:51.539 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:51 ntpd[27882]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:51.539 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:51 ntpd[27882]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:51.540 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:51 ntpd[27882]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:51.540 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:51 ntpd[27882]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-05T08:28:51.540 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:51 ntpd[27882]: can't open /var/log/ntpstats/peerstats.20231105: Permission denied 2023-11-05T08:28:51.540 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:51 ntpd[27882]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:51.540 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:51 ntpd[27882]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:51.540 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:51 ntpd[27882]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-05T08:28:51.541 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:51 ntpd[27882]: can't open /var/log/ntpstats/peerstats.20231105: Permission denied 2023-11-05T08:28:51.541 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:51 ntpd[27882]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:51.541 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:51 ntpd[27882]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:51.541 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:51 ntpd[27882]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-05T08:28:51.541 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:51 ntpd[27882]: can't open /var/log/ntpstats/peerstats.20231105: Permission denied 2023-11-05T08:28:51.542 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:51 ntpd[27919]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:51.542 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:51 ntpd[27919]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:51.543 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:51 ntpd[27919]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:51.543 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:51 ntpd[27919]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:51.543 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:51 ntpd[27919]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:51.543 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:51 ntpd[27919]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:51.543 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:51 ntpd[27919]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-05T08:28:51.544 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:51 ntpd[27919]: can't open /var/log/ntpstats/peerstats.20231105: Permission denied 2023-11-05T08:28:51.563 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:51 ntpd[28074]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:51.563 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:51 ntpd[28074]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:51.564 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:51 ntpd[28074]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:51.564 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:51 ntpd[28074]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:51.564 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:51 ntpd[28074]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:51.564 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:51 ntpd[28074]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:51.564 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:51 ntpd[28074]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-05T08:28:51.565 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:51 ntpd[28074]: can't open /var/log/ntpstats/peerstats.20231105: Permission denied 2023-11-05T08:28:51.565 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:51 ntpd[28074]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:51.565 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:51 ntpd[28074]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:51.565 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:51 ntpd[28074]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-05T08:28:51.565 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:51 ntpd[28074]: can't open /var/log/ntpstats/peerstats.20231105: Permission denied 2023-11-05T08:28:52.537 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:52 ntpd[27882]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:52.537 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:52 ntpd[27882]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:52.540 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:52 ntpd[28050]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:52.541 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:52 ntpd[28050]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:52.541 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:52 ntpd[27919]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:52.542 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:52 ntpd[27919]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:52.542 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:52 ntpd[27919]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:52.542 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:52 ntpd[27919]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:52.542 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:52 ntpd[27919]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-05T08:28:52.542 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:52 ntpd[27919]: can't open /var/log/ntpstats/peerstats.20231105: Permission denied 2023-11-05T08:28:52.563 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:52 ntpd[28074]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:52.563 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:52 ntpd[28074]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:53.538 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:53 ntpd[27882]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:53.539 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:53 ntpd[27882]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:53.539 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:53 ntpd[27882]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:53.539 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:53 ntpd[27882]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:53.539 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:53 ntpd[27882]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:53.540 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:53 ntpd[27882]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:53.540 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:53 ntpd[27882]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:53.540 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:53 ntpd[27882]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:53.540 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:53 ntpd[27882]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-05T08:28:53.540 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:53 ntpd[27882]: can't open /var/log/ntpstats/peerstats.20231105: Permission denied 2023-11-05T08:28:53.541 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:53 ntpd[27882]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:53.541 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:53 ntpd[27882]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:53.541 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:53 ntpd[27882]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-05T08:28:53.541 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:53 ntpd[27882]: can't open /var/log/ntpstats/peerstats.20231105: Permission denied 2023-11-05T08:28:53.541 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:53 ntpd[27882]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:53.542 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:53 ntpd[27882]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:53.542 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:53 ntpd[27882]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-05T08:28:53.542 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:53 ntpd[27882]: can't open /var/log/ntpstats/peerstats.20231105: Permission denied 2023-11-05T08:28:53.543 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:53 ntpd[27919]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:53.543 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:53 ntpd[27919]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:53.543 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:53 ntpd[27919]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:53.543 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:53 ntpd[27919]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:53.544 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:53 ntpd[27919]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-05T08:28:53.544 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:53 ntpd[27919]: can't open /var/log/ntpstats/peerstats.20231105: Permission denied 2023-11-05T08:28:53.564 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:53 ntpd[28074]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:53.564 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:53 ntpd[28074]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:53.564 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:53 ntpd[28074]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:53.564 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:53 ntpd[28074]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:53.565 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:53 ntpd[28074]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:53.565 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:53 ntpd[28074]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:53.565 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:53 ntpd[28074]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:53.565 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:53 ntpd[28074]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:53.565 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:53 ntpd[28074]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-05T08:28:53.566 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:53 ntpd[28074]: can't open /var/log/ntpstats/peerstats.20231105: Permission denied 2023-11-05T08:28:53.566 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:53 ntpd[28074]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:53.566 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:53 ntpd[28074]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:53.566 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:53 ntpd[28074]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-05T08:28:53.566 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:53 ntpd[28074]: can't open /var/log/ntpstats/peerstats.20231105: Permission denied 2023-11-05T08:28:53.567 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:53 ntpd[28074]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:53.567 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:53 ntpd[28074]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:53.567 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:53 ntpd[28074]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-05T08:28:53.567 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:53 ntpd[28074]: can't open /var/log/ntpstats/peerstats.20231105: Permission denied 2023-11-05T08:28:54.541 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:54 ntpd[28050]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:54.542 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:54 ntpd[28050]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:54.542 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:54 ntpd[28050]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:54.542 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:54 ntpd[28050]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:54.542 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:54 ntpd[28050]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:54.542 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:54 ntpd[28050]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:54.543 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:54 ntpd[28050]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:54.543 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:54 ntpd[28050]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:54.543 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:54 ntpd[28050]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-05T08:28:54.543 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:54 ntpd[28050]: can't open /var/log/ntpstats/peerstats.20231105: Permission denied 2023-11-05T08:28:54.543 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:54 ntpd[28050]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:54.544 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:54 ntpd[28050]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:54.544 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:54 ntpd[28050]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-05T08:28:54.544 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:54 ntpd[28050]: can't open /var/log/ntpstats/peerstats.20231105: Permission denied 2023-11-05T08:28:54.544 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:54 ntpd[28050]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:54.544 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:54 ntpd[28050]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:54.545 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:54 ntpd[28050]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-05T08:28:54.545 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:54 ntpd[28050]: can't open /var/log/ntpstats/peerstats.20231105: Permission denied 2023-11-05T08:28:54.545 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:54 ntpd[27919]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:54.546 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:54 ntpd[27919]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:54.546 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:54 ntpd[27919]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:54.546 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:54 ntpd[27919]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:54.546 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:54 ntpd[27919]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-05T08:28:54.547 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:54 ntpd[27919]: can't open /var/log/ntpstats/peerstats.20231105: Permission denied 2023-11-05T08:28:55.539 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:55 ntpd[27882]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:55.539 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:55 ntpd[27882]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:55.539 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:55 ntpd[27882]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:55.539 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:55 ntpd[27882]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:55.539 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:55 ntpd[27882]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:55.540 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:55 ntpd[27882]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:55.540 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:55 ntpd[27882]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:55.540 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:55 ntpd[27882]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:55.540 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:55 ntpd[27882]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-05T08:28:55.540 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:55 ntpd[27882]: can't open /var/log/ntpstats/peerstats.20231105: Permission denied 2023-11-05T08:28:55.540 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:55 ntpd[27882]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:55.541 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:55 ntpd[27882]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:55.541 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:55 ntpd[27882]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-05T08:28:55.541 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:55 ntpd[27882]: can't open /var/log/ntpstats/peerstats.20231105: Permission denied 2023-11-05T08:28:55.541 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:55 ntpd[27882]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:55.541 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:55 ntpd[27882]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:55.541 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:55 ntpd[27882]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-05T08:28:55.542 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:55 ntpd[27882]: can't open /var/log/ntpstats/peerstats.20231105: Permission denied 2023-11-05T08:28:55.543 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:55 ntpd[27919]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:55.543 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:55 ntpd[27919]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:55.543 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:55 ntpd[27919]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:55.543 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:55 ntpd[27919]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:55.543 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:55 ntpd[27919]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:55.544 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:55 ntpd[27919]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:55.544 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:55 ntpd[27919]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-05T08:28:55.544 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:55 ntpd[27919]: can't open /var/log/ntpstats/peerstats.20231105: Permission denied 2023-11-05T08:28:55.544 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:55 ntpd[27919]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:55.544 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:55 ntpd[27919]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:55.544 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:55 ntpd[27919]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-05T08:28:55.545 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:55 ntpd[27919]: can't open /var/log/ntpstats/peerstats.20231105: Permission denied 2023-11-05T08:28:55.564 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:55 ntpd[28074]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:55.564 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:55 ntpd[28074]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:55.564 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:55 ntpd[28074]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:55.564 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:55 ntpd[28074]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:55.565 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:55 ntpd[28074]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:55.565 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:55 ntpd[28074]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:55.565 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:55 ntpd[28074]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:55.565 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:55 ntpd[28074]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:55.565 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:55 ntpd[28074]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-05T08:28:55.566 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:55 ntpd[28074]: can't open /var/log/ntpstats/peerstats.20231105: Permission denied 2023-11-05T08:28:55.566 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:55 ntpd[28074]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:55.566 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:55 ntpd[28074]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:55.566 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:55 ntpd[28074]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-05T08:28:55.566 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:55 ntpd[28074]: can't open /var/log/ntpstats/peerstats.20231105: Permission denied 2023-11-05T08:28:55.567 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:55 ntpd[28074]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:55.567 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:55 ntpd[28074]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:55.567 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:55 ntpd[28074]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-05T08:28:55.567 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:55 ntpd[28074]: can't open /var/log/ntpstats/peerstats.20231105: Permission denied 2023-11-05T08:28:56.541 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:56 ntpd[28050]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:56.541 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:56 ntpd[28050]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:56.542 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:56 ntpd[28050]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:56.542 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:56 ntpd[28050]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:56.542 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:56 ntpd[28050]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:56.542 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:56 ntpd[28050]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:56.542 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:56 ntpd[28050]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:56.543 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:56 ntpd[28050]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:56.543 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:56 ntpd[28050]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-05T08:28:56.543 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:56 ntpd[28050]: can't open /var/log/ntpstats/peerstats.20231105: Permission denied 2023-11-05T08:28:56.543 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:56 ntpd[28050]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:56.543 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:56 ntpd[28050]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:56.544 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:56 ntpd[28050]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-05T08:28:56.544 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:56 ntpd[28050]: can't open /var/log/ntpstats/peerstats.20231105: Permission denied 2023-11-05T08:28:56.544 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:56 ntpd[28050]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:56.544 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:56 ntpd[28050]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:56.544 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:56 ntpd[28050]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-05T08:28:56.545 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:56 ntpd[28050]: can't open /var/log/ntpstats/peerstats.20231105: Permission denied 2023-11-05T08:28:56.545 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:56 ntpd[27919]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:56.546 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:56 ntpd[27919]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:56.546 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:56 ntpd[27919]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:56.546 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:56 ntpd[27919]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:56.546 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:56 ntpd[27919]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-05T08:28:56.546 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:56 ntpd[27919]: can't open /var/log/ntpstats/peerstats.20231105: Permission denied 2023-11-05T08:28:57.538 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:57 ntpd[27882]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:57.538 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:57 ntpd[27882]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:57.538 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:57 ntpd[27882]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:57.538 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:57 ntpd[27882]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:57.538 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:57 ntpd[27882]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:57.539 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:57 ntpd[27882]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:57.539 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:57 ntpd[27882]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:57.539 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:57 ntpd[27882]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:57.539 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:57 ntpd[27882]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-05T08:28:57.539 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:57 ntpd[27882]: can't open /var/log/ntpstats/peerstats.20231105: Permission denied 2023-11-05T08:28:57.540 INFO:teuthology.orchestra.run.smithi002.stdout: 5 Nov 08:28:57 ntpd[27882]: ntpd: time slew +0.000437 s 2023-11-05T08:28:57.540 INFO:teuthology.orchestra.run.smithi002.stdout:ntpd: time slew +0.000437s 2023-11-05T08:28:57.540 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:57 ntpd[27882]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-11-05T08:28:57.541 INFO:teuthology.orchestra.run.smithi002.stderr: 5 Nov 08:28:57 ntpd[27882]: can't open /var/log/ntpstats/loopstats.20231105: Permission denied 2023-11-05T08:28:57.542 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:57 ntpd[27919]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:57.542 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:57 ntpd[27919]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:57.542 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:57 ntpd[27919]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:57.542 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:57 ntpd[27919]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:57.543 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:57 ntpd[27919]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:57.543 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:57 ntpd[27919]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:57.543 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:57 ntpd[27919]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-05T08:28:57.543 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:57 ntpd[27919]: can't open /var/log/ntpstats/peerstats.20231105: Permission denied 2023-11-05T08:28:57.543 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:57 ntpd[27919]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:57.544 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:57 ntpd[27919]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:57.544 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:57 ntpd[27919]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-05T08:28:57.544 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:57 ntpd[27919]: can't open /var/log/ntpstats/peerstats.20231105: Permission denied 2023-11-05T08:28:57.544 INFO:teuthology.orchestra.run.smithi053.stdout: 5 Nov 08:28:57 ntpd[27919]: ntpd: time slew -0.003362 s 2023-11-05T08:28:57.545 INFO:teuthology.orchestra.run.smithi053.stdout:ntpd: time slew -0.003362s 2023-11-05T08:28:57.545 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:57 ntpd[27919]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-11-05T08:28:57.545 INFO:teuthology.orchestra.run.smithi053.stderr: 5 Nov 08:28:57 ntpd[27919]: can't open /var/log/ntpstats/loopstats.20231105: Permission denied 2023-11-05T08:28:57.564 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:57 ntpd[28074]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:57.564 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:57 ntpd[28074]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:57.564 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:57 ntpd[28074]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:57.564 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:57 ntpd[28074]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:57.564 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:57 ntpd[28074]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:57.565 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:57 ntpd[28074]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:57.565 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:57 ntpd[28074]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:57.565 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:57 ntpd[28074]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:57.565 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:57 ntpd[28074]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-05T08:28:57.565 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:57 ntpd[28074]: can't open /var/log/ntpstats/peerstats.20231105: Permission denied 2023-11-05T08:28:57.566 INFO:teuthology.orchestra.run.smithi090.stdout: 5 Nov 08:28:57 ntpd[28074]: ntpd: time slew -0.000792 s 2023-11-05T08:28:57.566 INFO:teuthology.orchestra.run.smithi090.stdout:ntpd: time slew -0.000792s 2023-11-05T08:28:57.566 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:57 ntpd[28074]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-11-05T08:28:57.567 INFO:teuthology.orchestra.run.smithi090.stderr: 5 Nov 08:28:57 ntpd[28074]: can't open /var/log/ntpstats/loopstats.20231105: Permission denied 2023-11-05T08:28:57.613 INFO:teuthology.orchestra.run.smithi002.stdout: remote refid st t when poll reach delay offset jitter 2023-11-05T08:28:57.613 INFO:teuthology.orchestra.run.smithi002.stdout:============================================================================== 2023-11-05T08:28:57.614 INFO:teuthology.orchestra.run.smithi002.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-05T08:28:57.614 INFO:teuthology.orchestra.run.smithi002.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-05T08:28:57.614 INFO:teuthology.orchestra.run.smithi002.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-05T08:28:57.614 INFO:teuthology.orchestra.run.smithi002.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-05T08:28:57.626 INFO:teuthology.orchestra.run.smithi053.stdout: remote refid st t when poll reach delay offset jitter 2023-11-05T08:28:57.627 INFO:teuthology.orchestra.run.smithi053.stdout:============================================================================== 2023-11-05T08:28:57.627 INFO:teuthology.orchestra.run.smithi053.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-05T08:28:57.627 INFO:teuthology.orchestra.run.smithi053.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-05T08:28:57.627 INFO:teuthology.orchestra.run.smithi053.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-05T08:28:57.628 INFO:teuthology.orchestra.run.smithi053.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-05T08:28:57.634 INFO:teuthology.orchestra.run.smithi090.stdout: remote refid st t when poll reach delay offset jitter 2023-11-05T08:28:57.635 INFO:teuthology.orchestra.run.smithi090.stdout:============================================================================== 2023-11-05T08:28:57.635 INFO:teuthology.orchestra.run.smithi090.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-05T08:28:57.635 INFO:teuthology.orchestra.run.smithi090.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-05T08:28:57.635 INFO:teuthology.orchestra.run.smithi090.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-05T08:28:57.635 INFO:teuthology.orchestra.run.smithi090.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-05T08:28:58.541 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:58 ntpd[28050]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:58.541 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:58 ntpd[28050]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:58.542 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:58 ntpd[28050]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:58.542 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:58 ntpd[28050]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:58.542 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:58 ntpd[28050]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:58.542 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:58 ntpd[28050]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:58.542 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:58 ntpd[28050]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:58.543 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:58 ntpd[28050]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:58.543 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:58 ntpd[28050]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-05T08:28:58.543 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:58 ntpd[28050]: can't open /var/log/ntpstats/peerstats.20231105: Permission denied 2023-11-05T08:28:58.543 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:58 ntpd[28050]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:58.543 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:58 ntpd[28050]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:58.543 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:58 ntpd[28050]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-05T08:28:58.544 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:58 ntpd[28050]: can't open /var/log/ntpstats/peerstats.20231105: Permission denied 2023-11-05T08:28:58.544 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:58 ntpd[28050]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:28:58.544 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:58 ntpd[28050]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:28:58.544 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:58 ntpd[28050]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-05T08:28:58.544 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:28:58 ntpd[28050]: can't open /var/log/ntpstats/peerstats.20231105: Permission denied 2023-11-05T08:29:00.541 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:29:00 ntpd[28050]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:29:00.541 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:29:00 ntpd[28050]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:29:00.541 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:29:00 ntpd[28050]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:29:00.542 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:29:00 ntpd[28050]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:29:00.542 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:29:00 ntpd[28050]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:29:00.542 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:29:00 ntpd[28050]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:29:00.542 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:29:00 ntpd[28050]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-05T08:29:00.542 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:29:00 ntpd[28050]: can't open /var/log/ntpstats/rawstats.20231105: Permission denied 2023-11-05T08:29:00.542 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:29:00 ntpd[28050]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-05T08:29:00.543 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:29:00 ntpd[28050]: can't open /var/log/ntpstats/peerstats.20231105: Permission denied 2023-11-05T08:29:00.543 INFO:teuthology.orchestra.run.smithi069.stdout: 5 Nov 08:29:00 ntpd[28050]: ntpd: time slew -0.001309 s 2023-11-05T08:29:00.543 INFO:teuthology.orchestra.run.smithi069.stdout:ntpd: time slew -0.001309s 2023-11-05T08:29:00.544 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:29:00 ntpd[28050]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-11-05T08:29:00.544 INFO:teuthology.orchestra.run.smithi069.stderr: 5 Nov 08:29:00 ntpd[28050]: can't open /var/log/ntpstats/loopstats.20231105: Permission denied 2023-11-05T08:29:00.617 INFO:teuthology.orchestra.run.smithi069.stdout: remote refid st t when poll reach delay offset jitter 2023-11-05T08:29:00.618 INFO:teuthology.orchestra.run.smithi069.stdout:============================================================================== 2023-11-05T08:29:00.618 INFO:teuthology.orchestra.run.smithi069.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-05T08:29:00.618 INFO:teuthology.orchestra.run.smithi069.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-05T08:29:00.618 INFO:teuthology.orchestra.run.smithi069.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-05T08:29:00.618 INFO:teuthology.orchestra.run.smithi069.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-05T08:29:00.619 INFO:teuthology.run_tasks:Running task install... 2023-11-05T08:29:00.630 DEBUG:teuthology.task.install:project ceph 2023-11-05T08:29:00.630 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '420cd129397301b1521ceb74e1bdbd9b855d2189'}} 2023-11-05T08:29:00.631 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '420cd129397301b1521ceb74e1bdbd9b855d2189'} 2023-11-05T08:29:00.631 INFO:teuthology.task.install:Using flavor: default 2023-11-05T08:29:00.637 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-05T08:29:00.637 INFO:teuthology.task.install:extra packages: [] 2023-11-05T08:29:00.638 DEBUG:teuthology.orchestra.run.smithi002:> sudo apt-key list | grep Ceph 2023-11-05T08:29:00.638 DEBUG:teuthology.orchestra.run.smithi053:> sudo apt-key list | grep Ceph 2023-11-05T08:29:00.639 DEBUG:teuthology.orchestra.run.smithi069:> sudo apt-key list | grep Ceph 2023-11-05T08:29:00.639 DEBUG:teuthology.orchestra.run.smithi090:> sudo apt-key list | grep Ceph 2023-11-05T08:29:00.705 INFO:teuthology.orchestra.run.smithi002.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-11-05T08:29:00.705 INFO:teuthology.orchestra.run.smithi090.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-11-05T08:29:00.706 INFO:teuthology.orchestra.run.smithi053.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-11-05T08:29:00.718 INFO:teuthology.orchestra.run.smithi069.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-11-05T08:29:00.734 INFO:teuthology.orchestra.run.smithi002.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-11-05T08:29:00.734 INFO:teuthology.orchestra.run.smithi002.stdout:uid [ unknown] Ceph.com (release key) 2023-11-05T08:29:00.735 INFO:teuthology.orchestra.run.smithi090.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-11-05T08:29:00.735 INFO:teuthology.orchestra.run.smithi090.stdout:uid [ unknown] Ceph.com (release key) 2023-11-05T08:29:00.736 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-11-05T08:29:00.736 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=420cd129397301b1521ceb74e1bdbd9b855d2189 2023-11-05T08:29:00.739 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-11-05T08:29:00.739 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=420cd129397301b1521ceb74e1bdbd9b855d2189 2023-11-05T08:29:00.744 INFO:teuthology.orchestra.run.smithi053.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-11-05T08:29:00.744 INFO:teuthology.orchestra.run.smithi053.stdout:uid [ unknown] Ceph.com (release key) 2023-11-05T08:29:00.744 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-11-05T08:29:00.745 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=420cd129397301b1521ceb74e1bdbd9b855d2189 2023-11-05T08:29:00.748 INFO:teuthology.orchestra.run.smithi069.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-11-05T08:29:00.748 INFO:teuthology.orchestra.run.smithi069.stdout:uid [ unknown] Ceph.com (release key) 2023-11-05T08:29:00.748 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-11-05T08:29:00.748 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=420cd129397301b1521ceb74e1bdbd9b855d2189 2023-11-05T08:29:00.882 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default/ 2023-11-05T08:29:00.882 INFO:teuthology.task.install.deb:Package version is 18.0.0-6776-g420cd129-1jammy 2023-11-05T08:29:00.896 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default/ 2023-11-05T08:29:00.896 INFO:teuthology.task.install.deb:Package version is 18.0.0-6776-g420cd129-1jammy 2023-11-05T08:29:00.953 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default/ 2023-11-05T08:29:00.954 INFO:teuthology.task.install.deb:Package version is 18.0.0-6776-g420cd129-1jammy 2023-11-05T08:29:00.959 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default/ 2023-11-05T08:29:00.959 INFO:teuthology.task.install.deb:Package version is 18.0.0-6776-g420cd129-1jammy 2023-11-05T08:29:01.018 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-05T08:29:01.018 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-11-05T08:29:01.033 DEBUG:teuthology.orchestra.run.smithi069:> sudo apt-get update 2023-11-05T08:29:01.055 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-05T08:29:01.055 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-11-05T08:29:01.065 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-11-05T08:29:01.065 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-11-05T08:29:01.069 DEBUG:teuthology.orchestra.run.smithi002:> sudo apt-get update 2023-11-05T08:29:01.079 DEBUG:teuthology.orchestra.run.smithi053:> sudo apt-get update 2023-11-05T08:29:01.118 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-11-05T08:29:01.118 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-11-05T08:29:01.132 DEBUG:teuthology.orchestra.run.smithi090:> sudo apt-get update 2023-11-05T08:29:01.196 INFO:teuthology.orchestra.run.smithi069.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-11-05T08:29:01.206 INFO:teuthology.orchestra.run.smithi069.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-11-05T08:29:01.223 INFO:teuthology.orchestra.run.smithi069.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-11-05T08:29:01.238 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-11-05T08:29:01.239 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-11-05T08:29:01.244 INFO:teuthology.orchestra.run.smithi053.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-11-05T08:29:01.248 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-11-05T08:29:01.251 INFO:teuthology.orchestra.run.smithi069.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy InRelease 2023-11-05T08:29:01.254 INFO:teuthology.orchestra.run.smithi053.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-11-05T08:29:01.265 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-11-05T08:29:01.271 INFO:teuthology.orchestra.run.smithi053.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-11-05T08:29:01.275 INFO:teuthology.orchestra.run.smithi069.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-11-05T08:29:01.297 INFO:teuthology.orchestra.run.smithi069.stdout:Ign:6 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy Release.gpg 2023-11-05T08:29:01.317 INFO:teuthology.orchestra.run.smithi069.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,409 B] 2023-11-05T08:29:01.341 INFO:teuthology.orchestra.run.smithi069.stdout:Hit:8 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-11-05T08:29:01.342 INFO:teuthology.orchestra.run.smithi069.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2023-11-05T08:29:01.354 INFO:teuthology.orchestra.run.smithi002.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy InRelease 2023-11-05T08:29:01.357 INFO:teuthology.orchestra.run.smithi053.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy InRelease 2023-11-05T08:29:01.368 INFO:teuthology.orchestra.run.smithi053.stdout:Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-11-05T08:29:01.391 INFO:teuthology.orchestra.run.smithi002.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-11-05T08:29:01.393 INFO:teuthology.orchestra.run.smithi053.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-11-05T08:29:01.413 INFO:teuthology.orchestra.run.smithi090.stdout:Ign:1 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy InRelease 2023-11-05T08:29:01.428 INFO:teuthology.orchestra.run.smithi002.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy Release.gpg 2023-11-05T08:29:01.429 INFO:teuthology.orchestra.run.smithi090.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-11-05T08:29:01.430 INFO:teuthology.orchestra.run.smithi053.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy Release.gpg 2023-11-05T08:29:01.449 INFO:teuthology.orchestra.run.smithi090.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-11-05T08:29:01.465 INFO:teuthology.orchestra.run.smithi002.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,409 B] 2023-11-05T08:29:01.465 INFO:teuthology.orchestra.run.smithi053.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,409 B] 2023-11-05T08:29:01.487 INFO:teuthology.orchestra.run.smithi090.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy Release.gpg 2023-11-05T08:29:01.500 INFO:teuthology.orchestra.run.smithi090.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-11-05T08:29:01.507 INFO:teuthology.orchestra.run.smithi053.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2023-11-05T08:29:01.509 INFO:teuthology.orchestra.run.smithi002.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2023-11-05T08:29:01.523 INFO:teuthology.orchestra.run.smithi090.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2023-11-05T08:29:01.547 INFO:teuthology.orchestra.run.smithi069.stdout:Fetched 28.5 kB in 0s (84.1 kB/s) 2023-11-05T08:29:01.566 INFO:teuthology.orchestra.run.smithi090.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,409 B] 2023-11-05T08:29:01.583 INFO:teuthology.orchestra.run.smithi090.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-11-05T08:29:01.602 INFO:teuthology.orchestra.run.smithi002.stdout:Fetched 28.5 kB in 0s (81.1 kB/s) 2023-11-05T08:29:01.612 INFO:teuthology.orchestra.run.smithi053.stdout:Fetched 28.5 kB in 0s (80.7 kB/s) 2023-11-05T08:29:02.441 INFO:teuthology.orchestra.run.smithi069.stdout:Reading package lists... 2023-11-05T08:29:02.455 DEBUG:teuthology.orchestra.run.smithi069:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6776-g420cd129-1jammy cephadm=18.0.0-6776-g420cd129-1jammy ceph-mds=18.0.0-6776-g420cd129-1jammy ceph-mgr=18.0.0-6776-g420cd129-1jammy ceph-common=18.0.0-6776-g420cd129-1jammy ceph-fuse=18.0.0-6776-g420cd129-1jammy ceph-test=18.0.0-6776-g420cd129-1jammy ceph-volume=18.0.0-6776-g420cd129-1jammy radosgw=18.0.0-6776-g420cd129-1jammy python3-rados=18.0.0-6776-g420cd129-1jammy python3-rgw=18.0.0-6776-g420cd129-1jammy python3-cephfs=18.0.0-6776-g420cd129-1jammy python3-rbd=18.0.0-6776-g420cd129-1jammy libcephfs2=18.0.0-6776-g420cd129-1jammy libcephfs-dev=18.0.0-6776-g420cd129-1jammy librados2=18.0.0-6776-g420cd129-1jammy librbd1=18.0.0-6776-g420cd129-1jammy rbd-fuse=18.0.0-6776-g420cd129-1jammy 2023-11-05T08:29:02.469 INFO:teuthology.orchestra.run.smithi002.stdout:Reading package lists... 2023-11-05T08:29:02.485 DEBUG:teuthology.orchestra.run.smithi002:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6776-g420cd129-1jammy cephadm=18.0.0-6776-g420cd129-1jammy ceph-mds=18.0.0-6776-g420cd129-1jammy ceph-mgr=18.0.0-6776-g420cd129-1jammy ceph-common=18.0.0-6776-g420cd129-1jammy ceph-fuse=18.0.0-6776-g420cd129-1jammy ceph-test=18.0.0-6776-g420cd129-1jammy ceph-volume=18.0.0-6776-g420cd129-1jammy radosgw=18.0.0-6776-g420cd129-1jammy python3-rados=18.0.0-6776-g420cd129-1jammy python3-rgw=18.0.0-6776-g420cd129-1jammy python3-cephfs=18.0.0-6776-g420cd129-1jammy python3-rbd=18.0.0-6776-g420cd129-1jammy libcephfs2=18.0.0-6776-g420cd129-1jammy libcephfs-dev=18.0.0-6776-g420cd129-1jammy librados2=18.0.0-6776-g420cd129-1jammy librbd1=18.0.0-6776-g420cd129-1jammy rbd-fuse=18.0.0-6776-g420cd129-1jammy 2023-11-05T08:29:02.513 INFO:teuthology.orchestra.run.smithi053.stdout:Reading package lists... 2023-11-05T08:29:02.515 INFO:teuthology.orchestra.run.smithi069.stdout:Reading package lists... 2023-11-05T08:29:02.528 DEBUG:teuthology.orchestra.run.smithi053:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6776-g420cd129-1jammy cephadm=18.0.0-6776-g420cd129-1jammy ceph-mds=18.0.0-6776-g420cd129-1jammy ceph-mgr=18.0.0-6776-g420cd129-1jammy ceph-common=18.0.0-6776-g420cd129-1jammy ceph-fuse=18.0.0-6776-g420cd129-1jammy ceph-test=18.0.0-6776-g420cd129-1jammy ceph-volume=18.0.0-6776-g420cd129-1jammy radosgw=18.0.0-6776-g420cd129-1jammy python3-rados=18.0.0-6776-g420cd129-1jammy python3-rgw=18.0.0-6776-g420cd129-1jammy python3-cephfs=18.0.0-6776-g420cd129-1jammy python3-rbd=18.0.0-6776-g420cd129-1jammy libcephfs2=18.0.0-6776-g420cd129-1jammy libcephfs-dev=18.0.0-6776-g420cd129-1jammy librados2=18.0.0-6776-g420cd129-1jammy librbd1=18.0.0-6776-g420cd129-1jammy rbd-fuse=18.0.0-6776-g420cd129-1jammy 2023-11-05T08:29:02.549 INFO:teuthology.orchestra.run.smithi002.stdout:Reading package lists... 2023-11-05T08:29:02.580 INFO:teuthology.orchestra.run.smithi053.stdout:Reading package lists... 2023-11-05T08:29:02.702 INFO:teuthology.orchestra.run.smithi069.stdout:Building dependency tree... 2023-11-05T08:29:02.703 INFO:teuthology.orchestra.run.smithi069.stdout:Reading state information... 2023-11-05T08:29:02.741 INFO:teuthology.orchestra.run.smithi002.stdout:Building dependency tree... 2023-11-05T08:29:02.742 INFO:teuthology.orchestra.run.smithi002.stdout:Reading state information... 2023-11-05T08:29:02.753 INFO:teuthology.orchestra.run.smithi053.stdout:Building dependency tree... 2023-11-05T08:29:02.754 INFO:teuthology.orchestra.run.smithi053.stdout:Reading state information... 2023-11-05T08:29:02.865 INFO:teuthology.orchestra.run.smithi069.stdout:The following packages were automatically installed and are no longer required: 2023-11-05T08:29:02.865 INFO:teuthology.orchestra.run.smithi069.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-11-05T08:29:02.866 INFO:teuthology.orchestra.run.smithi069.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-11-05T08:29:02.866 INFO:teuthology.orchestra.run.smithi069.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-11-05T08:29:02.866 INFO:teuthology.orchestra.run.smithi069.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-11-05T08:29:02.867 INFO:teuthology.orchestra.run.smithi069.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-11-05T08:29:02.868 INFO:teuthology.orchestra.run.smithi069.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-05T08:29:02.868 INFO:teuthology.orchestra.run.smithi069.stdout:The following additional packages will be installed: 2023-11-05T08:29:02.868 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-11-05T08:29:02.868 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-11-05T08:29:02.869 INFO:teuthology.orchestra.run.smithi069.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-11-05T08:29:02.869 INFO:teuthology.orchestra.run.smithi069.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-11-05T08:29:02.869 INFO:teuthology.orchestra.run.smithi069.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-11-05T08:29:02.869 INFO:teuthology.orchestra.run.smithi069.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-11-05T08:29:02.869 INFO:teuthology.orchestra.run.smithi069.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-11-05T08:29:02.870 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-11-05T08:29:02.870 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2023-11-05T08:29:02.870 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-11-05T08:29:02.870 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-11-05T08:29:02.870 INFO:teuthology.orchestra.run.smithi069.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-11-05T08:29:02.871 INFO:teuthology.orchestra.run.smithi069.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-11-05T08:29:02.871 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-11-05T08:29:02.871 INFO:teuthology.orchestra.run.smithi069.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-11-05T08:29:02.871 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2023-11-05T08:29:02.871 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2023-11-05T08:29:02.872 INFO:teuthology.orchestra.run.smithi069.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-11-05T08:29:02.872 INFO:teuthology.orchestra.run.smithi069.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-11-05T08:29:02.872 INFO:teuthology.orchestra.run.smithi069.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-11-05T08:29:02.872 INFO:teuthology.orchestra.run.smithi069.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-11-05T08:29:02.872 INFO:teuthology.orchestra.run.smithi069.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2023-11-05T08:29:02.873 INFO:teuthology.orchestra.run.smithi069.stdout:Suggested packages: 2023-11-05T08:29:02.873 INFO:teuthology.orchestra.run.smithi069.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-11-05T08:29:02.873 INFO:teuthology.orchestra.run.smithi069.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-11-05T08:29:02.874 INFO:teuthology.orchestra.run.smithi069.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-11-05T08:29:02.874 INFO:teuthology.orchestra.run.smithi069.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-11-05T08:29:02.874 INFO:teuthology.orchestra.run.smithi069.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-11-05T08:29:02.874 INFO:teuthology.orchestra.run.smithi069.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-11-05T08:29:02.874 INFO:teuthology.orchestra.run.smithi069.stdout: python3-watchdog 2023-11-05T08:29:02.875 INFO:teuthology.orchestra.run.smithi069.stdout:Recommended packages: 2023-11-05T08:29:02.875 INFO:teuthology.orchestra.run.smithi069.stdout: btrfs-tools 2023-11-05T08:29:02.910 INFO:teuthology.orchestra.run.smithi053.stdout:The following packages were automatically installed and are no longer required: 2023-11-05T08:29:02.910 INFO:teuthology.orchestra.run.smithi053.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-11-05T08:29:02.911 INFO:teuthology.orchestra.run.smithi053.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-11-05T08:29:02.911 INFO:teuthology.orchestra.run.smithi053.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-11-05T08:29:02.912 INFO:teuthology.orchestra.run.smithi053.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-11-05T08:29:02.913 INFO:teuthology.orchestra.run.smithi053.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-11-05T08:29:02.913 INFO:teuthology.orchestra.run.smithi053.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-05T08:29:02.913 INFO:teuthology.orchestra.run.smithi053.stdout:The following additional packages will be installed: 2023-11-05T08:29:02.913 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-11-05T08:29:02.914 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-11-05T08:29:02.914 INFO:teuthology.orchestra.run.smithi053.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-11-05T08:29:02.914 INFO:teuthology.orchestra.run.smithi053.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-11-05T08:29:02.914 INFO:teuthology.orchestra.run.smithi053.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-11-05T08:29:02.914 INFO:teuthology.orchestra.run.smithi053.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-11-05T08:29:02.915 INFO:teuthology.orchestra.run.smithi053.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-11-05T08:29:02.915 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-11-05T08:29:02.915 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2023-11-05T08:29:02.915 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-11-05T08:29:02.915 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-11-05T08:29:02.916 INFO:teuthology.orchestra.run.smithi053.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-11-05T08:29:02.916 INFO:teuthology.orchestra.run.smithi053.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-11-05T08:29:02.916 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-11-05T08:29:02.916 INFO:teuthology.orchestra.run.smithi053.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-11-05T08:29:02.916 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2023-11-05T08:29:02.917 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2023-11-05T08:29:02.917 INFO:teuthology.orchestra.run.smithi053.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-11-05T08:29:02.917 INFO:teuthology.orchestra.run.smithi053.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-11-05T08:29:02.917 INFO:teuthology.orchestra.run.smithi053.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-11-05T08:29:02.917 INFO:teuthology.orchestra.run.smithi053.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-11-05T08:29:02.918 INFO:teuthology.orchestra.run.smithi053.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2023-11-05T08:29:02.919 INFO:teuthology.orchestra.run.smithi053.stdout:Suggested packages: 2023-11-05T08:29:02.919 INFO:teuthology.orchestra.run.smithi053.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-11-05T08:29:02.919 INFO:teuthology.orchestra.run.smithi053.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-11-05T08:29:02.919 INFO:teuthology.orchestra.run.smithi053.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-11-05T08:29:02.919 INFO:teuthology.orchestra.run.smithi053.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-11-05T08:29:02.920 INFO:teuthology.orchestra.run.smithi053.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-11-05T08:29:02.920 INFO:teuthology.orchestra.run.smithi053.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-11-05T08:29:02.920 INFO:teuthology.orchestra.run.smithi053.stdout: python3-watchdog 2023-11-05T08:29:02.920 INFO:teuthology.orchestra.run.smithi053.stdout:Recommended packages: 2023-11-05T08:29:02.920 INFO:teuthology.orchestra.run.smithi053.stdout: btrfs-tools 2023-11-05T08:29:02.922 INFO:teuthology.orchestra.run.smithi002.stdout:The following packages were automatically installed and are no longer required: 2023-11-05T08:29:02.922 INFO:teuthology.orchestra.run.smithi002.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-11-05T08:29:02.922 INFO:teuthology.orchestra.run.smithi002.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-11-05T08:29:02.923 INFO:teuthology.orchestra.run.smithi002.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-11-05T08:29:02.923 INFO:teuthology.orchestra.run.smithi002.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-11-05T08:29:02.923 INFO:teuthology.orchestra.run.smithi002.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-11-05T08:29:02.923 INFO:teuthology.orchestra.run.smithi002.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-05T08:29:02.924 INFO:teuthology.orchestra.run.smithi002.stdout:The following additional packages will be installed: 2023-11-05T08:29:02.924 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-11-05T08:29:02.924 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-11-05T08:29:02.924 INFO:teuthology.orchestra.run.smithi002.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-11-05T08:29:02.924 INFO:teuthology.orchestra.run.smithi002.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-11-05T08:29:02.925 INFO:teuthology.orchestra.run.smithi002.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-11-05T08:29:02.925 INFO:teuthology.orchestra.run.smithi002.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-11-05T08:29:02.925 INFO:teuthology.orchestra.run.smithi002.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-11-05T08:29:02.925 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-11-05T08:29:02.925 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2023-11-05T08:29:02.926 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-11-05T08:29:02.926 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-11-05T08:29:02.926 INFO:teuthology.orchestra.run.smithi002.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-11-05T08:29:02.926 INFO:teuthology.orchestra.run.smithi002.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-11-05T08:29:02.927 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-11-05T08:29:02.927 INFO:teuthology.orchestra.run.smithi002.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-11-05T08:29:02.927 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2023-11-05T08:29:02.927 INFO:teuthology.orchestra.run.smithi002.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2023-11-05T08:29:02.927 INFO:teuthology.orchestra.run.smithi002.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-11-05T08:29:02.928 INFO:teuthology.orchestra.run.smithi002.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-11-05T08:29:02.928 INFO:teuthology.orchestra.run.smithi002.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-11-05T08:29:02.928 INFO:teuthology.orchestra.run.smithi002.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-11-05T08:29:02.928 INFO:teuthology.orchestra.run.smithi002.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2023-11-05T08:29:02.928 INFO:teuthology.orchestra.run.smithi002.stdout:Suggested packages: 2023-11-05T08:29:02.929 INFO:teuthology.orchestra.run.smithi002.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-11-05T08:29:02.929 INFO:teuthology.orchestra.run.smithi002.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-11-05T08:29:02.929 INFO:teuthology.orchestra.run.smithi002.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-11-05T08:29:02.929 INFO:teuthology.orchestra.run.smithi002.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-11-05T08:29:02.929 INFO:teuthology.orchestra.run.smithi002.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-11-05T08:29:02.930 INFO:teuthology.orchestra.run.smithi002.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-11-05T08:29:02.930 INFO:teuthology.orchestra.run.smithi002.stdout: python3-watchdog 2023-11-05T08:29:02.930 INFO:teuthology.orchestra.run.smithi002.stdout:Recommended packages: 2023-11-05T08:29:02.930 INFO:teuthology.orchestra.run.smithi002.stdout: btrfs-tools 2023-11-05T08:29:02.962 INFO:teuthology.orchestra.run.smithi069.stdout:The following NEW packages will be installed: 2023-11-05T08:29:02.962 INFO:teuthology.orchestra.run.smithi069.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-11-05T08:29:02.962 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-11-05T08:29:02.963 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2023-11-05T08:29:02.964 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2023-11-05T08:29:02.964 INFO:teuthology.orchestra.run.smithi069.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2023-11-05T08:29:02.964 INFO:teuthology.orchestra.run.smithi069.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-11-05T08:29:02.965 INFO:teuthology.orchestra.run.smithi069.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-11-05T08:29:02.965 INFO:teuthology.orchestra.run.smithi069.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-11-05T08:29:02.965 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-11-05T08:29:02.965 INFO:teuthology.orchestra.run.smithi069.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2023-11-05T08:29:02.965 INFO:teuthology.orchestra.run.smithi069.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-11-05T08:29:02.966 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-11-05T08:29:02.966 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-11-05T08:29:02.966 INFO:teuthology.orchestra.run.smithi069.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-11-05T08:29:02.966 INFO:teuthology.orchestra.run.smithi069.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-11-05T08:29:02.967 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-11-05T08:29:02.967 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-11-05T08:29:02.967 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2023-11-05T08:29:02.967 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-11-05T08:29:02.967 INFO:teuthology.orchestra.run.smithi069.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-11-05T08:29:02.968 INFO:teuthology.orchestra.run.smithi069.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-11-05T08:29:02.968 INFO:teuthology.orchestra.run.smithi069.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-11-05T08:29:02.968 INFO:teuthology.orchestra.run.smithi069.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-11-05T08:29:02.968 INFO:teuthology.orchestra.run.smithi069.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-11-05T08:29:02.969 INFO:teuthology.orchestra.run.smithi069.stdout: socat xmlstarlet 2023-11-05T08:29:03.013 INFO:teuthology.orchestra.run.smithi053.stdout:The following NEW packages will be installed: 2023-11-05T08:29:03.013 INFO:teuthology.orchestra.run.smithi053.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-11-05T08:29:03.014 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-11-05T08:29:03.014 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2023-11-05T08:29:03.014 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2023-11-05T08:29:03.014 INFO:teuthology.orchestra.run.smithi053.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2023-11-05T08:29:03.014 INFO:teuthology.orchestra.run.smithi053.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-11-05T08:29:03.016 INFO:teuthology.orchestra.run.smithi053.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-11-05T08:29:03.016 INFO:teuthology.orchestra.run.smithi053.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-11-05T08:29:03.016 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-11-05T08:29:03.016 INFO:teuthology.orchestra.run.smithi053.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2023-11-05T08:29:03.017 INFO:teuthology.orchestra.run.smithi053.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-11-05T08:29:03.017 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-11-05T08:29:03.017 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-11-05T08:29:03.017 INFO:teuthology.orchestra.run.smithi053.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-11-05T08:29:03.018 INFO:teuthology.orchestra.run.smithi053.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-11-05T08:29:03.018 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-11-05T08:29:03.018 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-11-05T08:29:03.018 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2023-11-05T08:29:03.019 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-11-05T08:29:03.019 INFO:teuthology.orchestra.run.smithi053.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-11-05T08:29:03.019 INFO:teuthology.orchestra.run.smithi053.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-11-05T08:29:03.019 INFO:teuthology.orchestra.run.smithi053.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-11-05T08:29:03.019 INFO:teuthology.orchestra.run.smithi053.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-11-05T08:29:03.020 INFO:teuthology.orchestra.run.smithi053.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-11-05T08:29:03.020 INFO:teuthology.orchestra.run.smithi053.stdout: socat xmlstarlet 2023-11-05T08:29:03.021 INFO:teuthology.orchestra.run.smithi002.stdout:The following NEW packages will be installed: 2023-11-05T08:29:03.022 INFO:teuthology.orchestra.run.smithi002.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-11-05T08:29:03.022 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-11-05T08:29:03.022 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2023-11-05T08:29:03.024 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2023-11-05T08:29:03.024 INFO:teuthology.orchestra.run.smithi002.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2023-11-05T08:29:03.024 INFO:teuthology.orchestra.run.smithi002.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-11-05T08:29:03.024 INFO:teuthology.orchestra.run.smithi002.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-11-05T08:29:03.024 INFO:teuthology.orchestra.run.smithi002.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-11-05T08:29:03.025 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-11-05T08:29:03.025 INFO:teuthology.orchestra.run.smithi002.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2023-11-05T08:29:03.025 INFO:teuthology.orchestra.run.smithi002.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-11-05T08:29:03.025 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-11-05T08:29:03.026 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-11-05T08:29:03.026 INFO:teuthology.orchestra.run.smithi002.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-11-05T08:29:03.026 INFO:teuthology.orchestra.run.smithi002.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-11-05T08:29:03.026 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-11-05T08:29:03.026 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-11-05T08:29:03.027 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2023-11-05T08:29:03.027 INFO:teuthology.orchestra.run.smithi002.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-11-05T08:29:03.027 INFO:teuthology.orchestra.run.smithi002.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-11-05T08:29:03.027 INFO:teuthology.orchestra.run.smithi002.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-11-05T08:29:03.028 INFO:teuthology.orchestra.run.smithi002.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-11-05T08:29:03.028 INFO:teuthology.orchestra.run.smithi002.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-11-05T08:29:03.028 INFO:teuthology.orchestra.run.smithi002.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-11-05T08:29:03.028 INFO:teuthology.orchestra.run.smithi002.stdout: socat xmlstarlet 2023-11-05T08:29:03.161 INFO:teuthology.orchestra.run.smithi053.stdout:0 upgraded, 109 newly installed, 0 to remove and 258 not upgraded. 2023-11-05T08:29:03.162 INFO:teuthology.orchestra.run.smithi053.stdout:Need to get 184 MB of archives. 2023-11-05T08:29:03.162 INFO:teuthology.orchestra.run.smithi053.stdout:After this operation, 825 MB of additional disk space will be used. 2023-11-05T08:29:03.162 INFO:teuthology.orchestra.run.smithi053.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6776-g420cd129-1jammy [3,449 kB] 2023-11-05T08:29:03.166 INFO:teuthology.orchestra.run.smithi002.stdout:0 upgraded, 109 newly installed, 0 to remove and 258 not upgraded. 2023-11-05T08:29:03.166 INFO:teuthology.orchestra.run.smithi002.stdout:Need to get 184 MB of archives. 2023-11-05T08:29:03.166 INFO:teuthology.orchestra.run.smithi002.stdout:After this operation, 825 MB of additional disk space will be used. 2023-11-05T08:29:03.167 INFO:teuthology.orchestra.run.smithi002.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6776-g420cd129-1jammy [3,449 kB] 2023-11-05T08:29:03.183 INFO:teuthology.orchestra.run.smithi069.stdout:0 upgraded, 109 newly installed, 0 to remove and 258 not upgraded. 2023-11-05T08:29:03.183 INFO:teuthology.orchestra.run.smithi069.stdout:Need to get 184 MB of archives. 2023-11-05T08:29:03.184 INFO:teuthology.orchestra.run.smithi069.stdout:After this operation, 825 MB of additional disk space will be used. 2023-11-05T08:29:03.184 INFO:teuthology.orchestra.run.smithi069.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6776-g420cd129-1jammy [3,449 kB] 2023-11-05T08:29:03.211 INFO:teuthology.orchestra.run.smithi002.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-11-05T08:29:03.486 INFO:teuthology.orchestra.run.smithi002.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-11-05T08:29:03.510 INFO:teuthology.orchestra.run.smithi002.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6776-g420cd129-1jammy [3,130 kB] 2023-11-05T08:29:03.522 INFO:teuthology.orchestra.run.smithi053.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6776-g420cd129-1jammy [3,130 kB] 2023-11-05T08:29:03.657 INFO:teuthology.orchestra.run.smithi002.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-11-05T08:29:03.671 INFO:teuthology.orchestra.run.smithi002.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-11-05T08:29:03.737 INFO:teuthology.orchestra.run.smithi002.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6776-g420cd129-1jammy [688 kB] 2023-11-05T08:29:03.743 INFO:teuthology.orchestra.run.smithi002.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-11-05T08:29:03.780 INFO:teuthology.orchestra.run.smithi069.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6776-g420cd129-1jammy [3,130 kB] 2023-11-05T08:29:03.794 INFO:teuthology.orchestra.run.smithi002.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6776-g420cd129-1jammy [356 kB] 2023-11-05T08:29:03.799 INFO:teuthology.orchestra.run.smithi053.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6776-g420cd129-1jammy [688 kB] 2023-11-05T08:29:03.828 INFO:teuthology.orchestra.run.smithi002.stdout:Get:10 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6776-g420cd129-1jammy [32.8 kB] 2023-11-05T08:29:03.830 INFO:teuthology.orchestra.run.smithi002.stdout:Get:11 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6776-g420cd129-1jammy [180 kB] 2023-11-05T08:29:03.842 INFO:teuthology.orchestra.run.smithi002.stdout:Get:12 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6776-g420cd129-1jammy [62.4 kB] 2023-11-05T08:29:03.846 INFO:teuthology.orchestra.run.smithi002.stdout:Get:13 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6776-g420cd129-1jammy [330 kB] 2023-11-05T08:29:03.861 INFO:teuthology.orchestra.run.smithi053.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6776-g420cd129-1jammy [356 kB] 2023-11-05T08:29:03.864 INFO:teuthology.orchestra.run.smithi069.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-11-05T08:29:03.873 INFO:teuthology.orchestra.run.smithi053.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6776-g420cd129-1jammy [32.8 kB] 2023-11-05T08:29:03.873 INFO:teuthology.orchestra.run.smithi053.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6776-g420cd129-1jammy [180 kB] 2023-11-05T08:29:03.884 INFO:teuthology.orchestra.run.smithi002.stdout:Get:14 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6776-g420cd129-1jammy [6,220 kB] 2023-11-05T08:29:03.913 INFO:teuthology.orchestra.run.smithi053.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6776-g420cd129-1jammy [62.4 kB] 2023-11-05T08:29:03.916 INFO:teuthology.orchestra.run.smithi053.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6776-g420cd129-1jammy [330 kB] 2023-11-05T08:29:03.921 INFO:teuthology.orchestra.run.smithi069.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-11-05T08:29:03.931 INFO:teuthology.orchestra.run.smithi053.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-11-05T08:29:03.932 INFO:teuthology.orchestra.run.smithi053.stdout:Get:10 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6776-g420cd129-1jammy [6,220 kB] 2023-11-05T08:29:03.937 INFO:teuthology.orchestra.run.smithi002.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-11-05T08:29:03.947 INFO:teuthology.orchestra.run.smithi002.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-11-05T08:29:03.956 INFO:teuthology.orchestra.run.smithi069.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-11-05T08:29:03.959 INFO:teuthology.orchestra.run.smithi069.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-11-05T08:29:03.973 INFO:teuthology.orchestra.run.smithi069.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-11-05T08:29:03.975 INFO:teuthology.orchestra.run.smithi002.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-11-05T08:29:03.984 INFO:teuthology.orchestra.run.smithi002.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-11-05T08:29:03.985 INFO:teuthology.orchestra.run.smithi002.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-11-05T08:29:03.986 INFO:teuthology.orchestra.run.smithi002.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-11-05T08:29:03.988 INFO:teuthology.orchestra.run.smithi053.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-11-05T08:29:04.024 INFO:teuthology.orchestra.run.smithi053.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-11-05T08:29:04.024 INFO:teuthology.orchestra.run.smithi069.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-11-05T08:29:04.027 INFO:teuthology.orchestra.run.smithi053.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-11-05T08:29:04.029 INFO:teuthology.orchestra.run.smithi069.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-11-05T08:29:04.040 INFO:teuthology.orchestra.run.smithi069.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-11-05T08:29:04.042 INFO:teuthology.orchestra.run.smithi053.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-11-05T08:29:04.044 INFO:teuthology.orchestra.run.smithi069.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-11-05T08:29:04.045 INFO:teuthology.orchestra.run.smithi069.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-11-05T08:29:04.046 INFO:teuthology.orchestra.run.smithi069.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-11-05T08:29:04.054 INFO:teuthology.orchestra.run.smithi069.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-11-05T08:29:04.054 INFO:teuthology.orchestra.run.smithi069.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-11-05T08:29:04.055 INFO:teuthology.orchestra.run.smithi069.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-11-05T08:29:04.056 INFO:teuthology.orchestra.run.smithi069.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-11-05T08:29:04.062 INFO:teuthology.orchestra.run.smithi069.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-11-05T08:29:04.062 INFO:teuthology.orchestra.run.smithi069.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-11-05T08:29:04.080 INFO:teuthology.orchestra.run.smithi069.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-11-05T08:29:04.081 INFO:teuthology.orchestra.run.smithi069.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-11-05T08:29:04.081 INFO:teuthology.orchestra.run.smithi069.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-11-05T08:29:04.082 INFO:teuthology.orchestra.run.smithi069.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-11-05T08:29:04.084 INFO:teuthology.orchestra.run.smithi069.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-11-05T08:29:04.085 INFO:teuthology.orchestra.run.smithi069.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-11-05T08:29:04.085 INFO:teuthology.orchestra.run.smithi069.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-11-05T08:29:04.086 INFO:teuthology.orchestra.run.smithi069.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-11-05T08:29:04.093 INFO:teuthology.orchestra.run.smithi053.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-11-05T08:29:04.097 INFO:teuthology.orchestra.run.smithi053.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-11-05T08:29:04.098 INFO:teuthology.orchestra.run.smithi069.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-11-05T08:29:04.099 INFO:teuthology.orchestra.run.smithi069.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-11-05T08:29:04.109 INFO:teuthology.orchestra.run.smithi053.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-11-05T08:29:04.114 INFO:teuthology.orchestra.run.smithi053.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-11-05T08:29:04.114 INFO:teuthology.orchestra.run.smithi053.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-11-05T08:29:04.115 INFO:teuthology.orchestra.run.smithi069.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-11-05T08:29:04.115 INFO:teuthology.orchestra.run.smithi053.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-11-05T08:29:04.116 INFO:teuthology.orchestra.run.smithi069.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-11-05T08:29:04.117 INFO:teuthology.orchestra.run.smithi069.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-11-05T08:29:04.117 INFO:teuthology.orchestra.run.smithi069.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-11-05T08:29:04.123 INFO:teuthology.orchestra.run.smithi069.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-11-05T08:29:04.123 INFO:teuthology.orchestra.run.smithi069.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-11-05T08:29:04.123 INFO:teuthology.orchestra.run.smithi069.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-11-05T08:29:04.124 INFO:teuthology.orchestra.run.smithi069.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-11-05T08:29:04.124 INFO:teuthology.orchestra.run.smithi053.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-11-05T08:29:04.125 INFO:teuthology.orchestra.run.smithi053.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-11-05T08:29:04.125 INFO:teuthology.orchestra.run.smithi053.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-11-05T08:29:04.126 INFO:teuthology.orchestra.run.smithi053.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-11-05T08:29:04.130 INFO:teuthology.orchestra.run.smithi053.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-11-05T08:29:04.131 INFO:teuthology.orchestra.run.smithi053.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-11-05T08:29:04.131 INFO:teuthology.orchestra.run.smithi069.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-11-05T08:29:04.133 INFO:teuthology.orchestra.run.smithi069.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-11-05T08:29:04.142 INFO:teuthology.orchestra.run.smithi069.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-11-05T08:29:04.143 INFO:teuthology.orchestra.run.smithi069.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-11-05T08:29:04.144 INFO:teuthology.orchestra.run.smithi069.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-11-05T08:29:04.146 INFO:teuthology.orchestra.run.smithi069.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-11-05T08:29:04.147 INFO:teuthology.orchestra.run.smithi069.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-11-05T08:29:04.148 INFO:teuthology.orchestra.run.smithi053.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-11-05T08:29:04.148 INFO:teuthology.orchestra.run.smithi053.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-11-05T08:29:04.149 INFO:teuthology.orchestra.run.smithi053.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-11-05T08:29:04.149 INFO:teuthology.orchestra.run.smithi053.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-11-05T08:29:04.149 INFO:teuthology.orchestra.run.smithi069.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-11-05T08:29:04.151 INFO:teuthology.orchestra.run.smithi053.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-11-05T08:29:04.152 INFO:teuthology.orchestra.run.smithi053.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-11-05T08:29:04.152 INFO:teuthology.orchestra.run.smithi053.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-11-05T08:29:04.153 INFO:teuthology.orchestra.run.smithi053.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-11-05T08:29:04.164 INFO:teuthology.orchestra.run.smithi053.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-11-05T08:29:04.165 INFO:teuthology.orchestra.run.smithi053.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-11-05T08:29:04.168 INFO:teuthology.orchestra.run.smithi002.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-11-05T08:29:04.176 INFO:teuthology.orchestra.run.smithi069.stdout:Get:46 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6776-g420cd129-1jammy [688 kB] 2023-11-05T08:29:04.181 INFO:teuthology.orchestra.run.smithi053.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-11-05T08:29:04.182 INFO:teuthology.orchestra.run.smithi053.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-11-05T08:29:04.183 INFO:teuthology.orchestra.run.smithi053.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-11-05T08:29:04.184 INFO:teuthology.orchestra.run.smithi053.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-11-05T08:29:04.189 INFO:teuthology.orchestra.run.smithi053.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-11-05T08:29:04.190 INFO:teuthology.orchestra.run.smithi053.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-11-05T08:29:04.190 INFO:teuthology.orchestra.run.smithi053.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-11-05T08:29:04.191 INFO:teuthology.orchestra.run.smithi053.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-11-05T08:29:04.198 INFO:teuthology.orchestra.run.smithi053.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-11-05T08:29:04.200 INFO:teuthology.orchestra.run.smithi053.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-11-05T08:29:04.201 INFO:teuthology.orchestra.run.smithi069.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-11-05T08:29:04.202 INFO:teuthology.orchestra.run.smithi069.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-11-05T08:29:04.204 INFO:teuthology.orchestra.run.smithi069.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-11-05T08:29:04.204 INFO:teuthology.orchestra.run.smithi069.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-11-05T08:29:04.210 INFO:teuthology.orchestra.run.smithi053.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-11-05T08:29:04.211 INFO:teuthology.orchestra.run.smithi053.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-11-05T08:29:04.212 INFO:teuthology.orchestra.run.smithi053.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-11-05T08:29:04.213 INFO:teuthology.orchestra.run.smithi069.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-11-05T08:29:04.214 INFO:teuthology.orchestra.run.smithi069.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-11-05T08:29:04.215 INFO:teuthology.orchestra.run.smithi053.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-11-05T08:29:04.215 INFO:teuthology.orchestra.run.smithi053.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-11-05T08:29:04.218 INFO:teuthology.orchestra.run.smithi053.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-11-05T08:29:04.231 INFO:teuthology.orchestra.run.smithi069.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-11-05T08:29:04.234 INFO:teuthology.orchestra.run.smithi069.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-11-05T08:29:04.234 INFO:teuthology.orchestra.run.smithi069.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-11-05T08:29:04.249 INFO:teuthology.orchestra.run.smithi069.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-11-05T08:29:04.249 INFO:teuthology.orchestra.run.smithi069.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-11-05T08:29:04.250 INFO:teuthology.orchestra.run.smithi069.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-11-05T08:29:04.250 INFO:teuthology.orchestra.run.smithi069.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-11-05T08:29:04.251 INFO:teuthology.orchestra.run.smithi069.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-11-05T08:29:04.251 INFO:teuthology.orchestra.run.smithi069.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-11-05T08:29:04.253 INFO:teuthology.orchestra.run.smithi069.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-11-05T08:29:04.253 INFO:teuthology.orchestra.run.smithi069.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-11-05T08:29:04.256 INFO:teuthology.orchestra.run.smithi069.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-11-05T08:29:04.270 INFO:teuthology.orchestra.run.smithi069.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-11-05T08:29:04.270 INFO:teuthology.orchestra.run.smithi069.stdout:Get:66 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6776-g420cd129-1jammy [356 kB] 2023-11-05T08:29:04.271 INFO:teuthology.orchestra.run.smithi069.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-11-05T08:29:04.271 INFO:teuthology.orchestra.run.smithi069.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-11-05T08:29:04.272 INFO:teuthology.orchestra.run.smithi053.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-11-05T08:29:04.273 INFO:teuthology.orchestra.run.smithi053.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-11-05T08:29:04.274 INFO:teuthology.orchestra.run.smithi053.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-11-05T08:29:04.274 INFO:teuthology.orchestra.run.smithi053.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-11-05T08:29:04.275 INFO:teuthology.orchestra.run.smithi053.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-11-05T08:29:04.275 INFO:teuthology.orchestra.run.smithi069.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-11-05T08:29:04.276 INFO:teuthology.orchestra.run.smithi053.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-11-05T08:29:04.277 INFO:teuthology.orchestra.run.smithi069.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-11-05T08:29:04.281 INFO:teuthology.orchestra.run.smithi069.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-11-05T08:29:04.283 INFO:teuthology.orchestra.run.smithi069.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-11-05T08:29:04.286 INFO:teuthology.orchestra.run.smithi069.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-11-05T08:29:04.287 INFO:teuthology.orchestra.run.smithi069.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-11-05T08:29:04.288 INFO:teuthology.orchestra.run.smithi069.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-11-05T08:29:04.293 INFO:teuthology.orchestra.run.smithi053.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-11-05T08:29:04.295 INFO:teuthology.orchestra.run.smithi053.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-11-05T08:29:04.295 INFO:teuthology.orchestra.run.smithi053.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-11-05T08:29:04.303 INFO:teuthology.orchestra.run.smithi069.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-11-05T08:29:04.304 INFO:teuthology.orchestra.run.smithi069.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-11-05T08:29:04.310 INFO:teuthology.orchestra.run.smithi053.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-11-05T08:29:04.310 INFO:teuthology.orchestra.run.smithi053.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-11-05T08:29:04.310 INFO:teuthology.orchestra.run.smithi053.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-11-05T08:29:04.311 INFO:teuthology.orchestra.run.smithi053.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-11-05T08:29:04.312 INFO:teuthology.orchestra.run.smithi053.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-11-05T08:29:04.312 INFO:teuthology.orchestra.run.smithi069.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-11-05T08:29:04.312 INFO:teuthology.orchestra.run.smithi069.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-11-05T08:29:04.312 INFO:teuthology.orchestra.run.smithi069.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-11-05T08:29:04.313 INFO:teuthology.orchestra.run.smithi053.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-11-05T08:29:04.313 INFO:teuthology.orchestra.run.smithi069.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-11-05T08:29:04.314 INFO:teuthology.orchestra.run.smithi069.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-11-05T08:29:04.324 INFO:teuthology.orchestra.run.smithi053.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-11-05T08:29:04.324 INFO:teuthology.orchestra.run.smithi053.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-11-05T08:29:04.327 INFO:teuthology.orchestra.run.smithi053.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-11-05T08:29:04.328 INFO:teuthology.orchestra.run.smithi069.stdout:Get:83 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6776-g420cd129-1jammy [32.8 kB] 2023-11-05T08:29:04.332 INFO:teuthology.orchestra.run.smithi069.stdout:Get:84 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6776-g420cd129-1jammy [180 kB] 2023-11-05T08:29:04.341 INFO:teuthology.orchestra.run.smithi053.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-11-05T08:29:04.342 INFO:teuthology.orchestra.run.smithi053.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-11-05T08:29:04.343 INFO:teuthology.orchestra.run.smithi053.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-11-05T08:29:04.346 INFO:teuthology.orchestra.run.smithi053.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-11-05T08:29:04.348 INFO:teuthology.orchestra.run.smithi053.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-11-05T08:29:04.350 INFO:teuthology.orchestra.run.smithi069.stdout:Get:85 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6776-g420cd129-1jammy [62.4 kB] 2023-11-05T08:29:04.352 INFO:teuthology.orchestra.run.smithi053.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-11-05T08:29:04.355 INFO:teuthology.orchestra.run.smithi053.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-11-05T08:29:04.358 INFO:teuthology.orchestra.run.smithi053.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-11-05T08:29:04.359 INFO:teuthology.orchestra.run.smithi053.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-11-05T08:29:04.359 INFO:teuthology.orchestra.run.smithi053.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-11-05T08:29:04.361 INFO:teuthology.orchestra.run.smithi069.stdout:Get:86 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6776-g420cd129-1jammy [330 kB] 2023-11-05T08:29:04.375 INFO:teuthology.orchestra.run.smithi053.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-11-05T08:29:04.376 INFO:teuthology.orchestra.run.smithi053.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-11-05T08:29:04.382 INFO:teuthology.orchestra.run.smithi053.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-11-05T08:29:04.382 INFO:teuthology.orchestra.run.smithi053.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-11-05T08:29:04.383 INFO:teuthology.orchestra.run.smithi053.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-11-05T08:29:04.383 INFO:teuthology.orchestra.run.smithi002.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-11-05T08:29:04.384 INFO:teuthology.orchestra.run.smithi053.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-11-05T08:29:04.385 INFO:teuthology.orchestra.run.smithi053.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-11-05T08:29:04.397 INFO:teuthology.orchestra.run.smithi002.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-11-05T08:29:04.459 INFO:teuthology.orchestra.run.smithi002.stdout:Get:24 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6776-g420cd129-1jammy [112 kB] 2023-11-05T08:29:04.462 INFO:teuthology.orchestra.run.smithi002.stdout:Get:25 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6776-g420cd129-1jammy [439 kB] 2023-11-05T08:29:04.469 INFO:teuthology.orchestra.run.smithi069.stdout:Get:87 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6776-g420cd129-1jammy [6,220 kB] 2023-11-05T08:29:04.470 INFO:teuthology.orchestra.run.smithi002.stdout:Get:26 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6776-g420cd129-1jammy [22.6 MB] 2023-11-05T08:29:04.505 INFO:teuthology.orchestra.run.smithi053.stdout:Get:88 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6776-g420cd129-1jammy [112 kB] 2023-11-05T08:29:04.507 INFO:teuthology.orchestra.run.smithi053.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6776-g420cd129-1jammy [439 kB] 2023-11-05T08:29:04.517 INFO:teuthology.orchestra.run.smithi002.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-11-05T08:29:04.526 INFO:teuthology.orchestra.run.smithi002.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-11-05T08:29:04.537 INFO:teuthology.orchestra.run.smithi002.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-11-05T08:29:04.546 INFO:teuthology.orchestra.run.smithi053.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6776-g420cd129-1jammy [22.6 MB] 2023-11-05T08:29:04.553 INFO:teuthology.orchestra.run.smithi002.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-11-05T08:29:04.569 INFO:teuthology.orchestra.run.smithi002.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-11-05T08:29:04.571 INFO:teuthology.orchestra.run.smithi002.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-11-05T08:29:04.576 INFO:teuthology.orchestra.run.smithi002.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-11-05T08:29:04.679 INFO:teuthology.orchestra.run.smithi002.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-11-05T08:29:04.854 INFO:teuthology.orchestra.run.smithi002.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-11-05T08:29:05.020 INFO:teuthology.orchestra.run.smithi002.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-11-05T08:29:05.042 INFO:teuthology.orchestra.run.smithi002.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-11-05T08:29:05.176 INFO:teuthology.orchestra.run.smithi002.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-11-05T08:29:05.193 INFO:teuthology.orchestra.run.smithi002.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-11-05T08:29:05.200 INFO:teuthology.orchestra.run.smithi002.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-11-05T08:29:05.273 INFO:teuthology.orchestra.run.smithi002.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-11-05T08:29:05.294 INFO:teuthology.orchestra.run.smithi069.stdout:Get:88 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6776-g420cd129-1jammy [112 kB] 2023-11-05T08:29:05.300 INFO:teuthology.orchestra.run.smithi002.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-11-05T08:29:05.307 INFO:teuthology.orchestra.run.smithi069.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6776-g420cd129-1jammy [439 kB] 2023-11-05T08:29:05.308 INFO:teuthology.orchestra.run.smithi002.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-11-05T08:29:05.372 INFO:teuthology.orchestra.run.smithi069.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6776-g420cd129-1jammy [22.6 MB] 2023-11-05T08:29:05.431 INFO:teuthology.orchestra.run.smithi002.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-11-05T08:29:05.431 INFO:teuthology.orchestra.run.smithi002.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-11-05T08:29:05.599 INFO:teuthology.orchestra.run.smithi002.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-11-05T08:29:05.788 INFO:teuthology.orchestra.run.smithi002.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-11-05T08:29:05.943 INFO:teuthology.orchestra.run.smithi002.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-11-05T08:29:06.048 INFO:teuthology.orchestra.run.smithi002.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-11-05T08:29:06.070 INFO:teuthology.orchestra.run.smithi002.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-11-05T08:29:06.101 INFO:teuthology.orchestra.run.smithi002.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-11-05T08:29:06.114 INFO:teuthology.orchestra.run.smithi002.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-11-05T08:29:06.127 INFO:teuthology.orchestra.run.smithi090.stdout:Hit:9 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-11-05T08:29:06.155 INFO:teuthology.orchestra.run.smithi002.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-11-05T08:29:06.163 INFO:teuthology.orchestra.run.smithi002.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-11-05T08:29:06.193 INFO:teuthology.orchestra.run.smithi002.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-11-05T08:29:06.288 INFO:teuthology.orchestra.run.smithi090.stdout:Fetched 28.5 kB in 5s (5,737 B/s) 2023-11-05T08:29:06.332 INFO:teuthology.orchestra.run.smithi002.stdout:Get:56 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6776-g420cd129-1jammy [6,853 kB] 2023-11-05T08:29:06.421 INFO:teuthology.orchestra.run.smithi002.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-11-05T08:29:06.587 INFO:teuthology.orchestra.run.smithi002.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-11-05T08:29:06.664 INFO:teuthology.orchestra.run.smithi053.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6776-g420cd129-1jammy [6,853 kB] 2023-11-05T08:29:06.844 INFO:teuthology.orchestra.run.smithi002.stdout:Get:59 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6776-g420cd129-1jammy [238 kB] 2023-11-05T08:29:06.859 INFO:teuthology.orchestra.run.smithi002.stdout:Get:60 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6776-g420cd129-1jammy [128 kB] 2023-11-05T08:29:06.864 INFO:teuthology.orchestra.run.smithi002.stdout:Get:61 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6776-g420cd129-1jammy [1,496 kB] 2023-11-05T08:29:06.936 INFO:teuthology.orchestra.run.smithi002.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-11-05T08:29:06.954 INFO:teuthology.orchestra.run.smithi002.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-11-05T08:29:06.962 INFO:teuthology.orchestra.run.smithi002.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-11-05T08:29:06.995 INFO:teuthology.orchestra.run.smithi002.stdout:Get:65 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6776-g420cd129-1jammy [9,863 kB] 2023-11-05T08:29:07.025 INFO:teuthology.orchestra.run.smithi002.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-11-05T08:29:07.189 INFO:teuthology.orchestra.run.smithi053.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6776-g420cd129-1jammy [238 kB] 2023-11-05T08:29:07.195 INFO:teuthology.orchestra.run.smithi090.stdout:Reading package lists... 2023-11-05T08:29:07.196 INFO:teuthology.orchestra.run.smithi053.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6776-g420cd129-1jammy [128 kB] 2023-11-05T08:29:07.211 DEBUG:teuthology.orchestra.run.smithi090:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6776-g420cd129-1jammy cephadm=18.0.0-6776-g420cd129-1jammy ceph-mds=18.0.0-6776-g420cd129-1jammy ceph-mgr=18.0.0-6776-g420cd129-1jammy ceph-common=18.0.0-6776-g420cd129-1jammy ceph-fuse=18.0.0-6776-g420cd129-1jammy ceph-test=18.0.0-6776-g420cd129-1jammy ceph-volume=18.0.0-6776-g420cd129-1jammy radosgw=18.0.0-6776-g420cd129-1jammy python3-rados=18.0.0-6776-g420cd129-1jammy python3-rgw=18.0.0-6776-g420cd129-1jammy python3-cephfs=18.0.0-6776-g420cd129-1jammy python3-rbd=18.0.0-6776-g420cd129-1jammy libcephfs2=18.0.0-6776-g420cd129-1jammy libcephfs-dev=18.0.0-6776-g420cd129-1jammy librados2=18.0.0-6776-g420cd129-1jammy librbd1=18.0.0-6776-g420cd129-1jammy rbd-fuse=18.0.0-6776-g420cd129-1jammy 2023-11-05T08:29:07.232 INFO:teuthology.orchestra.run.smithi053.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6776-g420cd129-1jammy [1,496 kB] 2023-11-05T08:29:07.262 INFO:teuthology.orchestra.run.smithi090.stdout:Reading package lists... 2023-11-05T08:29:07.348 INFO:teuthology.orchestra.run.smithi053.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6776-g420cd129-1jammy [9,863 kB] 2023-11-05T08:29:07.376 INFO:teuthology.orchestra.run.smithi002.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-11-05T08:29:07.393 INFO:teuthology.orchestra.run.smithi002.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-11-05T08:29:07.395 INFO:teuthology.orchestra.run.smithi002.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-11-05T08:29:07.434 INFO:teuthology.orchestra.run.smithi090.stdout:Building dependency tree... 2023-11-05T08:29:07.434 INFO:teuthology.orchestra.run.smithi090.stdout:Reading state information... 2023-11-05T08:29:07.438 INFO:teuthology.orchestra.run.smithi002.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-11-05T08:29:07.452 INFO:teuthology.orchestra.run.smithi002.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-11-05T08:29:07.455 INFO:teuthology.orchestra.run.smithi002.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-11-05T08:29:07.584 INFO:teuthology.orchestra.run.smithi090.stdout:The following packages were automatically installed and are no longer required: 2023-11-05T08:29:07.585 INFO:teuthology.orchestra.run.smithi090.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-11-05T08:29:07.585 INFO:teuthology.orchestra.run.smithi090.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-11-05T08:29:07.585 INFO:teuthology.orchestra.run.smithi090.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-11-05T08:29:07.585 INFO:teuthology.orchestra.run.smithi090.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-11-05T08:29:07.587 INFO:teuthology.orchestra.run.smithi090.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-11-05T08:29:07.587 INFO:teuthology.orchestra.run.smithi090.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-05T08:29:07.587 INFO:teuthology.orchestra.run.smithi090.stdout:The following additional packages will be installed: 2023-11-05T08:29:07.587 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-11-05T08:29:07.588 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-11-05T08:29:07.588 INFO:teuthology.orchestra.run.smithi090.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-11-05T08:29:07.588 INFO:teuthology.orchestra.run.smithi090.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-11-05T08:29:07.588 INFO:teuthology.orchestra.run.smithi090.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-11-05T08:29:07.589 INFO:teuthology.orchestra.run.smithi090.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-11-05T08:29:07.589 INFO:teuthology.orchestra.run.smithi090.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-11-05T08:29:07.589 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-11-05T08:29:07.589 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2023-11-05T08:29:07.589 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-11-05T08:29:07.590 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-11-05T08:29:07.590 INFO:teuthology.orchestra.run.smithi090.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-11-05T08:29:07.590 INFO:teuthology.orchestra.run.smithi090.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-11-05T08:29:07.590 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-11-05T08:29:07.590 INFO:teuthology.orchestra.run.smithi090.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-11-05T08:29:07.591 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2023-11-05T08:29:07.591 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2023-11-05T08:29:07.591 INFO:teuthology.orchestra.run.smithi090.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-11-05T08:29:07.591 INFO:teuthology.orchestra.run.smithi090.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-11-05T08:29:07.591 INFO:teuthology.orchestra.run.smithi090.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-11-05T08:29:07.592 INFO:teuthology.orchestra.run.smithi090.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-11-05T08:29:07.592 INFO:teuthology.orchestra.run.smithi090.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2023-11-05T08:29:07.592 INFO:teuthology.orchestra.run.smithi090.stdout:Suggested packages: 2023-11-05T08:29:07.592 INFO:teuthology.orchestra.run.smithi090.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-11-05T08:29:07.593 INFO:teuthology.orchestra.run.smithi090.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-11-05T08:29:07.593 INFO:teuthology.orchestra.run.smithi090.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-11-05T08:29:07.593 INFO:teuthology.orchestra.run.smithi090.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-11-05T08:29:07.593 INFO:teuthology.orchestra.run.smithi090.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-11-05T08:29:07.593 INFO:teuthology.orchestra.run.smithi090.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-11-05T08:29:07.594 INFO:teuthology.orchestra.run.smithi090.stdout: python3-watchdog 2023-11-05T08:29:07.594 INFO:teuthology.orchestra.run.smithi090.stdout:Recommended packages: 2023-11-05T08:29:07.594 INFO:teuthology.orchestra.run.smithi090.stdout: btrfs-tools 2023-11-05T08:29:07.639 INFO:teuthology.orchestra.run.smithi002.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-11-05T08:29:07.681 INFO:teuthology.orchestra.run.smithi090.stdout:The following NEW packages will be installed: 2023-11-05T08:29:07.681 INFO:teuthology.orchestra.run.smithi090.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-11-05T08:29:07.682 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-11-05T08:29:07.682 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2023-11-05T08:29:07.682 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2023-11-05T08:29:07.682 INFO:teuthology.orchestra.run.smithi090.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2023-11-05T08:29:07.682 INFO:teuthology.orchestra.run.smithi090.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-11-05T08:29:07.683 INFO:teuthology.orchestra.run.smithi090.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-11-05T08:29:07.684 INFO:teuthology.orchestra.run.smithi090.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-11-05T08:29:07.684 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-11-05T08:29:07.684 INFO:teuthology.orchestra.run.smithi090.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2023-11-05T08:29:07.684 INFO:teuthology.orchestra.run.smithi090.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-11-05T08:29:07.685 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-11-05T08:29:07.685 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-11-05T08:29:07.685 INFO:teuthology.orchestra.run.smithi090.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-11-05T08:29:07.685 INFO:teuthology.orchestra.run.smithi090.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-11-05T08:29:07.685 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-11-05T08:29:07.686 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-11-05T08:29:07.686 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2023-11-05T08:29:07.686 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-11-05T08:29:07.686 INFO:teuthology.orchestra.run.smithi090.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-11-05T08:29:07.686 INFO:teuthology.orchestra.run.smithi090.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-11-05T08:29:07.687 INFO:teuthology.orchestra.run.smithi090.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-11-05T08:29:07.687 INFO:teuthology.orchestra.run.smithi090.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-11-05T08:29:07.687 INFO:teuthology.orchestra.run.smithi090.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-11-05T08:29:07.687 INFO:teuthology.orchestra.run.smithi090.stdout: socat xmlstarlet 2023-11-05T08:29:07.826 INFO:teuthology.orchestra.run.smithi002.stdout:Get:74 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6776-g420cd129-1jammy [31.7 MB] 2023-11-05T08:29:07.828 INFO:teuthology.orchestra.run.smithi002.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-11-05T08:29:07.891 INFO:teuthology.orchestra.run.smithi002.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-11-05T08:29:07.926 INFO:teuthology.orchestra.run.smithi002.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-11-05T08:29:07.985 INFO:teuthology.orchestra.run.smithi002.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-11-05T08:29:08.173 INFO:teuthology.orchestra.run.smithi090.stdout:0 upgraded, 109 newly installed, 0 to remove and 258 not upgraded. 2023-11-05T08:29:08.173 INFO:teuthology.orchestra.run.smithi090.stdout:Need to get 184 MB of archives. 2023-11-05T08:29:08.173 INFO:teuthology.orchestra.run.smithi090.stdout:After this operation, 825 MB of additional disk space will be used. 2023-11-05T08:29:08.173 INFO:teuthology.orchestra.run.smithi090.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6776-g420cd129-1jammy [3,449 kB] 2023-11-05T08:29:08.214 INFO:teuthology.orchestra.run.smithi002.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-11-05T08:29:08.218 INFO:teuthology.orchestra.run.smithi002.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-11-05T08:29:08.282 INFO:teuthology.orchestra.run.smithi002.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-11-05T08:29:08.284 INFO:teuthology.orchestra.run.smithi002.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-11-05T08:29:08.293 INFO:teuthology.orchestra.run.smithi002.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-11-05T08:29:08.312 INFO:teuthology.orchestra.run.smithi002.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-11-05T08:29:08.322 INFO:teuthology.orchestra.run.smithi053.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6776-g420cd129-1jammy [31.7 MB] 2023-11-05T08:29:08.530 INFO:teuthology.orchestra.run.smithi002.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-11-05T08:29:08.541 INFO:teuthology.orchestra.run.smithi069.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6776-g420cd129-1jammy [6,853 kB] 2023-11-05T08:29:08.829 INFO:teuthology.orchestra.run.smithi090.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-11-05T08:29:08.886 INFO:teuthology.orchestra.run.smithi090.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-11-05T08:29:08.922 INFO:teuthology.orchestra.run.smithi090.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-11-05T08:29:08.924 INFO:teuthology.orchestra.run.smithi090.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-11-05T08:29:08.939 INFO:teuthology.orchestra.run.smithi090.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-11-05T08:29:08.991 INFO:teuthology.orchestra.run.smithi090.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-11-05T08:29:08.995 INFO:teuthology.orchestra.run.smithi090.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-11-05T08:29:09.007 INFO:teuthology.orchestra.run.smithi090.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-11-05T08:29:09.010 INFO:teuthology.orchestra.run.smithi090.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-11-05T08:29:09.011 INFO:teuthology.orchestra.run.smithi090.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-11-05T08:29:09.011 INFO:teuthology.orchestra.run.smithi090.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-11-05T08:29:09.012 INFO:teuthology.orchestra.run.smithi002.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-11-05T08:29:09.013 INFO:teuthology.orchestra.run.smithi002.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-11-05T08:29:09.019 INFO:teuthology.orchestra.run.smithi090.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-11-05T08:29:09.020 INFO:teuthology.orchestra.run.smithi090.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-11-05T08:29:09.020 INFO:teuthology.orchestra.run.smithi090.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-11-05T08:29:09.022 INFO:teuthology.orchestra.run.smithi090.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-11-05T08:29:09.023 INFO:teuthology.orchestra.run.smithi002.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-11-05T08:29:09.026 INFO:teuthology.orchestra.run.smithi002.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-11-05T08:29:09.028 INFO:teuthology.orchestra.run.smithi090.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-11-05T08:29:09.029 INFO:teuthology.orchestra.run.smithi090.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-11-05T08:29:09.029 INFO:teuthology.orchestra.run.smithi090.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-11-05T08:29:09.045 INFO:teuthology.orchestra.run.smithi090.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-11-05T08:29:09.045 INFO:teuthology.orchestra.run.smithi090.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-11-05T08:29:09.046 INFO:teuthology.orchestra.run.smithi090.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-11-05T08:29:09.048 INFO:teuthology.orchestra.run.smithi090.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-11-05T08:29:09.049 INFO:teuthology.orchestra.run.smithi090.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-11-05T08:29:09.049 INFO:teuthology.orchestra.run.smithi090.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-11-05T08:29:09.050 INFO:teuthology.orchestra.run.smithi090.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-11-05T08:29:09.056 INFO:teuthology.orchestra.run.smithi002.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-11-05T08:29:09.061 INFO:teuthology.orchestra.run.smithi090.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-11-05T08:29:09.062 INFO:teuthology.orchestra.run.smithi090.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-11-05T08:29:09.062 INFO:teuthology.orchestra.run.smithi090.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-11-05T08:29:09.070 INFO:teuthology.orchestra.run.smithi002.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-11-05T08:29:09.078 INFO:teuthology.orchestra.run.smithi090.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-11-05T08:29:09.079 INFO:teuthology.orchestra.run.smithi090.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-11-05T08:29:09.079 INFO:teuthology.orchestra.run.smithi090.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-11-05T08:29:09.084 INFO:teuthology.orchestra.run.smithi090.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-11-05T08:29:09.085 INFO:teuthology.orchestra.run.smithi090.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-11-05T08:29:09.085 INFO:teuthology.orchestra.run.smithi090.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-11-05T08:29:09.086 INFO:teuthology.orchestra.run.smithi090.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-11-05T08:29:09.094 INFO:teuthology.orchestra.run.smithi090.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-11-05T08:29:09.096 INFO:teuthology.orchestra.run.smithi090.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-11-05T08:29:09.097 INFO:teuthology.orchestra.run.smithi090.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-11-05T08:29:09.111 INFO:teuthology.orchestra.run.smithi090.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-11-05T08:29:09.112 INFO:teuthology.orchestra.run.smithi090.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-11-05T08:29:09.114 INFO:teuthology.orchestra.run.smithi090.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-11-05T08:29:09.115 INFO:teuthology.orchestra.run.smithi090.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-11-05T08:29:09.117 INFO:teuthology.orchestra.run.smithi090.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-11-05T08:29:09.170 INFO:teuthology.orchestra.run.smithi090.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-11-05T08:29:09.170 INFO:teuthology.orchestra.run.smithi090.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-11-05T08:29:09.172 INFO:teuthology.orchestra.run.smithi090.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-11-05T08:29:09.172 INFO:teuthology.orchestra.run.smithi090.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-11-05T08:29:09.172 INFO:teuthology.orchestra.run.smithi090.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-11-05T08:29:09.173 INFO:teuthology.orchestra.run.smithi090.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-11-05T08:29:09.190 INFO:teuthology.orchestra.run.smithi090.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-11-05T08:29:09.192 INFO:teuthology.orchestra.run.smithi090.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-11-05T08:29:09.192 INFO:teuthology.orchestra.run.smithi090.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-11-05T08:29:09.197 INFO:teuthology.orchestra.run.smithi002.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-11-05T08:29:09.199 INFO:teuthology.orchestra.run.smithi002.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-11-05T08:29:09.200 INFO:teuthology.orchestra.run.smithi090.stdout:Get:54 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6776-g420cd129-1jammy [3,130 kB] 2023-11-05T08:29:09.201 INFO:teuthology.orchestra.run.smithi002.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-11-05T08:29:09.206 INFO:teuthology.orchestra.run.smithi090.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-11-05T08:29:09.207 INFO:teuthology.orchestra.run.smithi090.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-11-05T08:29:09.207 INFO:teuthology.orchestra.run.smithi090.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-11-05T08:29:09.208 INFO:teuthology.orchestra.run.smithi090.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-11-05T08:29:09.208 INFO:teuthology.orchestra.run.smithi090.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-11-05T08:29:09.208 INFO:teuthology.orchestra.run.smithi090.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-11-05T08:29:09.219 INFO:teuthology.orchestra.run.smithi002.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-11-05T08:29:09.220 INFO:teuthology.orchestra.run.smithi090.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-11-05T08:29:09.221 INFO:teuthology.orchestra.run.smithi090.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-11-05T08:29:09.224 INFO:teuthology.orchestra.run.smithi090.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-11-05T08:29:09.237 INFO:teuthology.orchestra.run.smithi090.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-11-05T08:29:09.238 INFO:teuthology.orchestra.run.smithi090.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-11-05T08:29:09.270 INFO:teuthology.orchestra.run.smithi090.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-11-05T08:29:09.274 INFO:teuthology.orchestra.run.smithi090.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-11-05T08:29:09.276 INFO:teuthology.orchestra.run.smithi090.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-11-05T08:29:09.281 INFO:teuthology.orchestra.run.smithi090.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-11-05T08:29:09.284 INFO:teuthology.orchestra.run.smithi090.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-11-05T08:29:09.284 INFO:teuthology.orchestra.run.smithi090.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-11-05T08:29:09.285 INFO:teuthology.orchestra.run.smithi090.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-11-05T08:29:09.285 INFO:teuthology.orchestra.run.smithi090.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-11-05T08:29:09.350 INFO:teuthology.orchestra.run.smithi090.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-11-05T08:29:09.351 INFO:teuthology.orchestra.run.smithi090.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-11-05T08:29:09.359 INFO:teuthology.orchestra.run.smithi090.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-11-05T08:29:09.359 INFO:teuthology.orchestra.run.smithi090.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-11-05T08:29:09.360 INFO:teuthology.orchestra.run.smithi090.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-11-05T08:29:09.362 INFO:teuthology.orchestra.run.smithi090.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-11-05T08:29:09.362 INFO:teuthology.orchestra.run.smithi090.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-11-05T08:29:09.387 INFO:teuthology.orchestra.run.smithi002.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-11-05T08:29:09.667 INFO:teuthology.orchestra.run.smithi090.stdout:Get:81 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6776-g420cd129-1jammy [688 kB] 2023-11-05T08:29:09.700 INFO:teuthology.orchestra.run.smithi069.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6776-g420cd129-1jammy [238 kB] 2023-11-05T08:29:09.704 INFO:teuthology.orchestra.run.smithi069.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6776-g420cd129-1jammy [128 kB] 2023-11-05T08:29:09.775 INFO:teuthology.orchestra.run.smithi069.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6776-g420cd129-1jammy [1,496 kB] 2023-11-05T08:29:09.780 INFO:teuthology.orchestra.run.smithi090.stdout:Get:82 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6776-g420cd129-1jammy [356 kB] 2023-11-05T08:29:09.826 INFO:teuthology.orchestra.run.smithi090.stdout:Get:83 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6776-g420cd129-1jammy [32.8 kB] 2023-11-05T08:29:09.830 INFO:teuthology.orchestra.run.smithi090.stdout:Get:84 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6776-g420cd129-1jammy [180 kB] 2023-11-05T08:29:09.849 INFO:teuthology.orchestra.run.smithi090.stdout:Get:85 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6776-g420cd129-1jammy [62.4 kB] 2023-11-05T08:29:09.890 INFO:teuthology.orchestra.run.smithi090.stdout:Get:86 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6776-g420cd129-1jammy [330 kB] 2023-11-05T08:29:09.922 INFO:teuthology.orchestra.run.smithi090.stdout:Get:87 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6776-g420cd129-1jammy [6,220 kB] 2023-11-05T08:29:09.966 INFO:teuthology.orchestra.run.smithi069.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6776-g420cd129-1jammy [9,863 kB] 2023-11-05T08:29:10.793 INFO:teuthology.orchestra.run.smithi053.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6776-g420cd129-1jammy [14.1 kB] 2023-11-05T08:29:10.794 INFO:teuthology.orchestra.run.smithi053.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6776-g420cd129-1jammy [839 kB] 2023-11-05T08:29:10.810 INFO:teuthology.orchestra.run.smithi053.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6776-g420cd129-1jammy [2,146 kB] 2023-11-05T08:29:10.922 INFO:teuthology.orchestra.run.smithi053.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6776-g420cd129-1jammy [284 kB] 2023-11-05T08:29:10.938 INFO:teuthology.orchestra.run.smithi053.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6776-g420cd129-1jammy [138 kB] 2023-11-05T08:29:10.942 INFO:teuthology.orchestra.run.smithi090.stdout:Get:88 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6776-g420cd129-1jammy [112 kB] 2023-11-05T08:29:10.952 INFO:teuthology.orchestra.run.smithi053.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6776-g420cd129-1jammy [1,906 kB] 2023-11-05T08:29:10.960 INFO:teuthology.orchestra.run.smithi090.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6776-g420cd129-1jammy [439 kB] 2023-11-05T08:29:11.031 INFO:teuthology.orchestra.run.smithi090.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6776-g420cd129-1jammy [22.6 MB] 2023-11-05T08:29:11.100 INFO:teuthology.orchestra.run.smithi053.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6776-g420cd129-1jammy [8,625 kB] 2023-11-05T08:29:11.803 INFO:teuthology.orchestra.run.smithi053.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6776-g420cd129-1jammy [14.2 kB] 2023-11-05T08:29:11.804 INFO:teuthology.orchestra.run.smithi053.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6776-g420cd129-1jammy [48.2 MB] 2023-11-05T08:29:12.072 INFO:teuthology.orchestra.run.smithi069.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6776-g420cd129-1jammy [31.7 MB] 2023-11-05T08:29:12.529 INFO:teuthology.orchestra.run.smithi002.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6776-g420cd129-1jammy [14.1 kB] 2023-11-05T08:29:12.530 INFO:teuthology.orchestra.run.smithi002.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6776-g420cd129-1jammy [839 kB] 2023-11-05T08:29:12.598 INFO:teuthology.orchestra.run.smithi002.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6776-g420cd129-1jammy [2,146 kB] 2023-11-05T08:29:12.930 INFO:teuthology.orchestra.run.smithi002.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6776-g420cd129-1jammy [284 kB] 2023-11-05T08:29:12.993 INFO:teuthology.orchestra.run.smithi002.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6776-g420cd129-1jammy [138 kB] 2023-11-05T08:29:13.003 INFO:teuthology.orchestra.run.smithi002.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6776-g420cd129-1jammy [1,906 kB] 2023-11-05T08:29:13.335 INFO:teuthology.orchestra.run.smithi002.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6776-g420cd129-1jammy [8,625 kB] 2023-11-05T08:29:14.397 INFO:teuthology.orchestra.run.smithi090.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6776-g420cd129-1jammy [6,853 kB] 2023-11-05T08:29:14.765 INFO:teuthology.orchestra.run.smithi002.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6776-g420cd129-1jammy [14.2 kB] 2023-11-05T08:29:14.768 INFO:teuthology.orchestra.run.smithi002.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6776-g420cd129-1jammy [48.2 MB] 2023-11-05T08:29:15.146 INFO:teuthology.orchestra.run.smithi090.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6776-g420cd129-1jammy [238 kB] 2023-11-05T08:29:15.165 INFO:teuthology.orchestra.run.smithi090.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6776-g420cd129-1jammy [128 kB] 2023-11-05T08:29:15.184 INFO:teuthology.orchestra.run.smithi090.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6776-g420cd129-1jammy [1,496 kB] 2023-11-05T08:29:15.349 INFO:teuthology.orchestra.run.smithi090.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6776-g420cd129-1jammy [9,863 kB] 2023-11-05T08:29:15.885 INFO:teuthology.orchestra.run.smithi053.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6776-g420cd129-1jammy [120 kB] 2023-11-05T08:29:15.887 INFO:teuthology.orchestra.run.smithi053.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6776-g420cd129-1jammy [38.3 kB] 2023-11-05T08:29:15.956 INFO:teuthology.orchestra.run.smithi053.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6776-g420cd129-1jammy [12.3 MB] 2023-11-05T08:29:16.559 INFO:teuthology.orchestra.run.smithi090.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6776-g420cd129-1jammy [31.7 MB] 2023-11-05T08:29:16.865 INFO:teuthology.orchestra.run.smithi053.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6776-g420cd129-1jammy [91.6 kB] 2023-11-05T08:29:17.442 INFO:teuthology.orchestra.run.smithi053.stdout:Fetched 184 MB in 14s (13.3 MB/s) 2023-11-05T08:29:17.766 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libfmt8:amd64. 2023-11-05T08:29:17.870 INFO:teuthology.orchestra.run.smithi053.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165985 files and directories currently installed.) 2023-11-05T08:29:17.876 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-11-05T08:29:17.966 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-11-05T08:29:18.075 INFO:teuthology.orchestra.run.smithi069.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6776-g420cd129-1jammy [14.1 kB] 2023-11-05T08:29:18.076 INFO:teuthology.orchestra.run.smithi069.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6776-g420cd129-1jammy [839 kB] 2023-11-05T08:29:18.092 INFO:teuthology.orchestra.run.smithi069.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6776-g420cd129-1jammy [2,146 kB] 2023-11-05T08:29:18.245 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-11-05T08:29:18.271 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-11-05T08:29:18.311 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-11-05T08:29:18.328 INFO:teuthology.orchestra.run.smithi069.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6776-g420cd129-1jammy [284 kB] 2023-11-05T08:29:18.352 INFO:teuthology.orchestra.run.smithi069.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6776-g420cd129-1jammy [138 kB] 2023-11-05T08:29:18.382 INFO:teuthology.orchestra.run.smithi069.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6776-g420cd129-1jammy [1,906 kB] 2023-11-05T08:29:18.623 INFO:teuthology.orchestra.run.smithi069.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6776-g420cd129-1jammy [8,625 kB] 2023-11-05T08:29:18.656 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-11-05T08:29:18.683 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-11-05T08:29:18.723 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-11-05T08:29:19.060 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-11-05T08:29:19.086 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-11-05T08:29:19.126 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-11-05T08:29:19.395 INFO:teuthology.orchestra.run.smithi090.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6776-g420cd129-1jammy [14.1 kB] 2023-11-05T08:29:19.395 INFO:teuthology.orchestra.run.smithi090.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6776-g420cd129-1jammy [839 kB] 2023-11-05T08:29:19.412 INFO:teuthology.orchestra.run.smithi090.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6776-g420cd129-1jammy [2,146 kB] 2023-11-05T08:29:19.505 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-11-05T08:29:19.532 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-11-05T08:29:19.571 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-05T08:29:19.579 INFO:teuthology.orchestra.run.smithi090.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6776-g420cd129-1jammy [284 kB] 2023-11-05T08:29:19.592 INFO:teuthology.orchestra.run.smithi090.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6776-g420cd129-1jammy [138 kB] 2023-11-05T08:29:19.610 INFO:teuthology.orchestra.run.smithi090.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6776-g420cd129-1jammy [1,906 kB] 2023-11-05T08:29:19.690 INFO:teuthology.orchestra.run.smithi069.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6776-g420cd129-1jammy [14.2 kB] 2023-11-05T08:29:19.707 INFO:teuthology.orchestra.run.smithi069.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6776-g420cd129-1jammy [48.2 MB] 2023-11-05T08:29:19.776 INFO:teuthology.orchestra.run.smithi090.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6776-g420cd129-1jammy [8,625 kB] 2023-11-05T08:29:19.958 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-11-05T08:29:19.985 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-11-05T08:29:20.025 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-05T08:29:20.341 INFO:teuthology.orchestra.run.smithi002.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6776-g420cd129-1jammy [120 kB] 2023-11-05T08:29:20.343 INFO:teuthology.orchestra.run.smithi002.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6776-g420cd129-1jammy [38.3 kB] 2023-11-05T08:29:20.344 INFO:teuthology.orchestra.run.smithi002.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6776-g420cd129-1jammy [12.3 MB] 2023-11-05T08:29:20.395 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-11-05T08:29:20.421 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-11-05T08:29:20.461 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-05T08:29:20.467 INFO:teuthology.orchestra.run.smithi090.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6776-g420cd129-1jammy [14.2 kB] 2023-11-05T08:29:20.468 INFO:teuthology.orchestra.run.smithi090.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6776-g420cd129-1jammy [48.2 MB] 2023-11-05T08:29:20.806 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-11-05T08:29:20.833 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-11-05T08:29:20.864 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-11-05T08:29:21.159 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package librados2. 2023-11-05T08:29:21.175 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../008-librados2_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:21.217 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking librados2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:21.371 INFO:teuthology.orchestra.run.smithi002.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6776-g420cd129-1jammy [91.6 kB] 2023-11-05T08:29:21.629 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package librbd1. 2023-11-05T08:29:21.645 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../009-librbd1_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:21.678 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking librbd1 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:21.950 INFO:teuthology.orchestra.run.smithi002.stdout:Fetched 184 MB in 18s (10.0 MB/s) 2023-11-05T08:29:22.067 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libfmt8:amd64. 2023-11-05T08:29:22.098 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libcephfs2. 2023-11-05T08:29:22.125 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../010-libcephfs2_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:22.164 INFO:teuthology.orchestra.run.smithi002.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165985 files and directories currently installed.) 2023-11-05T08:29:22.165 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libcephfs2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:22.170 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-11-05T08:29:22.241 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-11-05T08:29:22.451 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-rados. 2023-11-05T08:29:22.478 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../011-python3-rados_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:22.517 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-rados (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:22.552 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-11-05T08:29:22.579 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-11-05T08:29:22.619 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-11-05T08:29:22.854 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-11-05T08:29:22.881 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-05T08:29:22.912 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-ceph-argparse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:23.189 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-11-05T08:29:23.216 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-11-05T08:29:23.255 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-11-05T08:29:23.323 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-cephfs. 2023-11-05T08:29:23.350 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../013-python3-cephfs_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:23.390 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-cephfs (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:23.592 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-11-05T08:29:23.619 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-11-05T08:29:23.628 INFO:teuthology.orchestra.run.smithi090.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6776-g420cd129-1jammy [120 kB] 2023-11-05T08:29:23.631 INFO:teuthology.orchestra.run.smithi090.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6776-g420cd129-1jammy [38.3 kB] 2023-11-05T08:29:23.632 INFO:teuthology.orchestra.run.smithi090.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6776-g420cd129-1jammy [12.3 MB] 2023-11-05T08:29:23.658 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-11-05T08:29:23.668 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-ceph-common. 2023-11-05T08:29:23.694 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../014-python3-ceph-common_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-05T08:29:23.751 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-ceph-common (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:24.045 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-11-05T08:29:24.071 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-wcwidth. 2023-11-05T08:29:24.072 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-11-05T08:29:24.087 INFO:teuthology.orchestra.run.smithi090.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6776-g420cd129-1jammy [91.6 kB] 2023-11-05T08:29:24.097 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-11-05T08:29:24.121 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-05T08:29:24.137 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-11-05T08:29:24.457 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-prettytable. 2023-11-05T08:29:24.484 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2023-11-05T08:29:24.515 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-11-05T08:29:24.524 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-11-05T08:29:24.537 INFO:teuthology.orchestra.run.smithi069.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6776-g420cd129-1jammy [120 kB] 2023-11-05T08:29:24.542 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-11-05T08:29:24.581 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-05T08:29:24.599 INFO:teuthology.orchestra.run.smithi090.stdout:Fetched 184 MB in 16s (11.2 MB/s) 2023-11-05T08:29:24.614 INFO:teuthology.orchestra.run.smithi069.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6776-g420cd129-1jammy [38.3 kB] 2023-11-05T08:29:24.615 INFO:teuthology.orchestra.run.smithi069.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6776-g420cd129-1jammy [12.3 MB] 2023-11-05T08:29:24.710 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libfmt8:amd64. 2023-11-05T08:29:24.765 INFO:teuthology.orchestra.run.smithi090.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165985 files and directories currently installed.) 2023-11-05T08:29:24.769 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-11-05T08:29:24.784 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-rbd. 2023-11-05T08:29:24.801 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../017-python3-rbd_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:24.843 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-rbd (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:24.852 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-11-05T08:29:24.968 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-11-05T08:29:24.995 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-11-05T08:29:25.034 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-05T08:29:25.163 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-11-05T08:29:25.189 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-11-05T08:29:25.190 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-11-05T08:29:25.215 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-11-05T08:29:25.221 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-11-05T08:29:25.255 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-11-05T08:29:25.371 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-11-05T08:29:25.398 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-11-05T08:29:25.437 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-11-05T08:29:25.459 INFO:teuthology.orchestra.run.smithi069.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6776-g420cd129-1jammy [91.6 kB] 2023-11-05T08:29:25.558 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package librgw2. 2023-11-05T08:29:25.584 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../019-librgw2_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:25.609 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-11-05T08:29:25.624 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking librgw2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:25.635 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-11-05T08:29:25.676 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-11-05T08:29:25.765 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package librados2. 2023-11-05T08:29:25.792 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../008-librados2_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:25.832 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking librados2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:26.014 INFO:teuthology.orchestra.run.smithi069.stdout:Fetched 184 MB in 22s (8,187 kB/s) 2023-11-05T08:29:26.014 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-11-05T08:29:26.040 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-11-05T08:29:26.080 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-11-05T08:29:26.129 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libfmt8:amd64. 2023-11-05T08:29:26.151 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-rgw. 2023-11-05T08:29:26.167 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../020-python3-rgw_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:26.210 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-rgw (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:26.222 INFO:teuthology.orchestra.run.smithi069.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165985 files and directories currently installed.) 2023-11-05T08:29:26.227 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-11-05T08:29:26.302 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-11-05T08:29:26.401 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package librbd1. 2023-11-05T08:29:26.428 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../009-librbd1_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:26.468 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking librbd1 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:26.521 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package liboath0:amd64. 2023-11-05T08:29:26.549 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2023-11-05T08:29:26.551 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-11-05T08:29:26.577 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-11-05T08:29:26.579 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-11-05T08:29:26.621 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-11-05T08:29:26.626 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-05T08:29:26.648 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-11-05T08:29:26.688 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-11-05T08:29:26.871 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libcephfs2. 2023-11-05T08:29:26.874 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libradosstriper1. 2023-11-05T08:29:26.897 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../010-libcephfs2_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:26.901 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../022-libradosstriper1_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:26.929 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libcephfs2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:26.941 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libradosstriper1 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:27.040 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-11-05T08:29:27.041 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-11-05T08:29:27.066 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-11-05T08:29:27.067 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-11-05T08:29:27.106 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-05T08:29:27.108 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-11-05T08:29:27.190 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-rados. 2023-11-05T08:29:27.216 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../011-python3-rados_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:27.256 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-rados (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:27.395 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-common. 2023-11-05T08:29:27.422 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../023-ceph-common_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:27.453 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-11-05T08:29:27.461 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-common (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:27.479 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-11-05T08:29:27.485 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-11-05T08:29:27.511 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-11-05T08:29:27.519 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-11-05T08:29:27.551 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-05T08:29:27.576 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-11-05T08:29:27.602 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-05T08:29:27.642 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-ceph-argparse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:27.895 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-cephfs. 2023-11-05T08:29:27.906 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-11-05T08:29:27.922 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../013-python3-cephfs_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:27.933 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-11-05T08:29:27.947 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-11-05T08:29:27.961 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-cephfs (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:27.974 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-11-05T08:29:28.022 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-05T08:29:28.173 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-11-05T08:29:28.264 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-ceph-common. 2023-11-05T08:29:28.291 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../014-python3-ceph-common_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-05T08:29:28.330 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-ceph-common (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:28.417 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-11-05T08:29:28.443 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-11-05T08:29:28.492 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-05T08:29:28.518 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package librados2. 2023-11-05T08:29:28.544 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../008-librados2_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:28.584 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking librados2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:28.625 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-wcwidth. 2023-11-05T08:29:28.652 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-11-05T08:29:28.685 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-11-05T08:29:28.812 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-11-05T08:29:28.838 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-11-05T08:29:28.859 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-base. 2023-11-05T08:29:28.878 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-05T08:29:28.886 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../024-ceph-base_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:28.935 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-base (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:28.986 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-prettytable. 2023-11-05T08:29:29.013 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2023-11-05T08:29:29.053 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-11-05T08:29:29.071 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package librbd1. 2023-11-05T08:29:29.097 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../009-librbd1_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:29.129 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking librbd1 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:29.240 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-11-05T08:29:29.266 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-11-05T08:29:29.306 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-11-05T08:29:29.336 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-rbd. 2023-11-05T08:29:29.363 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../017-python3-rbd_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:29.405 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-rbd (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:29.584 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libcephfs2. 2023-11-05T08:29:29.610 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../010-libcephfs2_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:29.642 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package librados2. 2023-11-05T08:29:29.647 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-11-05T08:29:29.650 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libcephfs2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:29.668 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../008-librados2_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:29.673 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-11-05T08:29:29.709 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking librados2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:29.713 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-11-05T08:29:29.758 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-11-05T08:29:29.785 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-11-05T08:29:29.825 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-11-05T08:29:29.945 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-rados. 2023-11-05T08:29:29.960 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../011-python3-rados_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:29.974 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-cheroot. 2023-11-05T08:29:30.001 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-11-05T08:29:30.003 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-rados (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:30.041 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-11-05T08:29:30.119 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package librgw2. 2023-11-05T08:29:30.136 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../019-librgw2_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:30.162 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package librbd1. 2023-11-05T08:29:30.169 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking librgw2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:30.188 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../009-librbd1_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:30.228 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking librbd1 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:30.332 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-11-05T08:29:30.344 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-11-05T08:29:30.348 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-05T08:29:30.371 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-11-05T08:29:30.382 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-ceph-argparse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:30.411 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-11-05T08:29:30.610 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-cephfs. 2023-11-05T08:29:30.626 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../013-python3-cephfs_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:30.631 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libcephfs2. 2023-11-05T08:29:30.657 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../010-libcephfs2_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:30.668 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-cephfs (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:30.688 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-rgw. 2023-11-05T08:29:30.689 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libcephfs2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:30.714 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-jaraco.text. 2023-11-05T08:29:30.715 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../020-python3-rgw_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:30.741 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2023-11-05T08:29:30.755 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-rgw (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:30.780 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-11-05T08:29:30.967 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-rados. 2023-11-05T08:29:30.988 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-ceph-common. 2023-11-05T08:29:30.993 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../011-python3-rados_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:31.015 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../014-python3-ceph-common_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-05T08:29:31.033 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-rados (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:31.055 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-ceph-common (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:31.075 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-11-05T08:29:31.099 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package liboath0:amd64. 2023-11-05T08:29:31.102 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-11-05T08:29:31.126 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2023-11-05T08:29:31.133 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-11-05T08:29:31.166 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-11-05T08:29:31.368 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-wcwidth. 2023-11-05T08:29:31.369 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-11-05T08:29:31.394 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-11-05T08:29:31.395 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-05T08:29:31.435 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-11-05T08:29:31.436 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-ceph-argparse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:31.460 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libradosstriper1. 2023-11-05T08:29:31.470 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-tempora. 2023-11-05T08:29:31.487 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../022-libradosstriper1_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:31.497 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2023-11-05T08:29:31.527 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libradosstriper1 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:31.536 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-11-05T08:29:31.697 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-cephfs. 2023-11-05T08:29:31.712 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../013-python3-cephfs_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:31.755 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-prettytable. 2023-11-05T08:29:31.755 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-cephfs (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:31.781 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2023-11-05T08:29:31.821 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-11-05T08:29:31.839 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-portend. 2023-11-05T08:29:31.866 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2023-11-05T08:29:31.906 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-11-05T08:29:31.914 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-common. 2023-11-05T08:29:31.941 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../023-ceph-common_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:31.980 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-common (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:32.075 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-ceph-common. 2023-11-05T08:29:32.099 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-rbd. 2023-11-05T08:29:32.101 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../014-python3-ceph-common_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-05T08:29:32.115 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../017-python3-rbd_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:32.141 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-ceph-common (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:32.149 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-rbd (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:32.234 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-11-05T08:29:32.261 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2023-11-05T08:29:32.292 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-11-05T08:29:32.452 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-wcwidth. 2023-11-05T08:29:32.479 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-11-05T08:29:32.512 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-11-05T08:29:32.519 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-11-05T08:29:32.539 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-11-05T08:29:32.579 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-11-05T08:29:32.604 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-cherrypy3. 2023-11-05T08:29:32.630 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2023-11-05T08:29:32.670 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-11-05T08:29:32.813 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-prettytable. 2023-11-05T08:29:32.840 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2023-11-05T08:29:32.871 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-11-05T08:29:32.924 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package librgw2. 2023-11-05T08:29:32.950 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../019-librgw2_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:32.990 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking librgw2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:33.016 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-natsort. 2023-11-05T08:29:33.043 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2023-11-05T08:29:33.082 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-11-05T08:29:33.115 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-rbd. 2023-11-05T08:29:33.131 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../017-python3-rbd_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:33.174 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-rbd (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:33.294 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-base. 2023-11-05T08:29:33.321 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../024-ceph-base_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:33.370 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-base (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:33.377 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-logutils. 2023-11-05T08:29:33.404 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2023-11-05T08:29:33.527 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-11-05T08:29:33.554 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-11-05T08:29:33.588 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-11-05T08:29:33.593 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-11-05T08:29:33.695 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-rgw. 2023-11-05T08:29:33.721 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../020-python3-rgw_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:33.761 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-rgw (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:33.929 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package librgw2. 2023-11-05T08:29:33.930 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-markupsafe. 2023-11-05T08:29:33.956 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../019-librgw2_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:33.957 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-11-05T08:29:33.996 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking librgw2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:33.996 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-11-05T08:29:34.090 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package liboath0:amd64. 2023-11-05T08:29:34.116 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2023-11-05T08:29:34.156 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-11-05T08:29:34.248 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-11-05T08:29:34.266 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-mako. 2023-11-05T08:29:34.275 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-11-05T08:29:34.294 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-11-05T08:29:34.323 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-11-05T08:29:34.333 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-11-05T08:29:34.468 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libradosstriper1. 2023-11-05T08:29:34.490 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-rgw. 2023-11-05T08:29:34.494 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../022-libradosstriper1_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:34.505 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../020-python3-rgw_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:34.540 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-rgw (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:34.543 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libradosstriper1 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:34.693 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-cheroot. 2023-11-05T08:29:34.694 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-simplegeneric. 2023-11-05T08:29:34.719 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-11-05T08:29:34.722 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2023-11-05T08:29:34.761 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-11-05T08:29:34.767 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-11-05T08:29:34.859 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package liboath0:amd64. 2023-11-05T08:29:34.886 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2023-11-05T08:29:34.934 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-11-05T08:29:34.973 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-common. 2023-11-05T08:29:34.999 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../023-ceph-common_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:35.039 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-common (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:35.072 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-singledispatch. 2023-11-05T08:29:35.100 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-11-05T08:29:35.139 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-11-05T08:29:35.162 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-11-05T08:29:35.190 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-11-05T08:29:35.245 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-11-05T08:29:35.262 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libradosstriper1. 2023-11-05T08:29:35.289 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../022-libradosstriper1_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:35.329 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libradosstriper1 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:35.400 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-webob. 2023-11-05T08:29:35.427 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-11-05T08:29:35.467 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-11-05T08:29:35.565 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-jaraco.text. 2023-11-05T08:29:35.592 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2023-11-05T08:29:35.632 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-11-05T08:29:35.740 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-common. 2023-11-05T08:29:35.762 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-waitress. 2023-11-05T08:29:35.767 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../023-ceph-common_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:35.789 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-11-05T08:29:35.807 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-common (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:35.832 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-11-05T08:29:35.959 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-11-05T08:29:35.986 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-11-05T08:29:36.034 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-11-05T08:29:36.131 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-tempita. 2023-11-05T08:29:36.159 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-11-05T08:29:36.198 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-11-05T08:29:36.372 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-base. 2023-11-05T08:29:36.398 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../024-ceph-base_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:36.412 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-tempora. 2023-11-05T08:29:36.439 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2023-11-05T08:29:36.448 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-base (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:36.487 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-11-05T08:29:36.501 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-paste. 2023-11-05T08:29:36.528 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-11-05T08:29:36.568 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-11-05T08:29:36.815 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-portend. 2023-11-05T08:29:36.841 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2023-11-05T08:29:36.881 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-11-05T08:29:36.896 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-11-05T08:29:36.923 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-11-05T08:29:36.962 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-11-05T08:29:37.184 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-11-05T08:29:37.202 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-11-05T08:29:37.211 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2023-11-05T08:29:37.229 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-11-05T08:29:37.249 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-pastedeploy. 2023-11-05T08:29:37.251 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-11-05T08:29:37.269 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-11-05T08:29:37.277 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2023-11-05T08:29:37.315 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-11-05T08:29:37.565 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-cheroot. 2023-11-05T08:29:37.591 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-11-05T08:29:37.602 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-webtest. 2023-11-05T08:29:37.604 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-cherrypy3. 2023-11-05T08:29:37.629 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2023-11-05T08:29:37.631 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2023-11-05T08:29:37.631 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-11-05T08:29:37.668 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-11-05T08:29:37.679 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-11-05T08:29:37.754 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-base. 2023-11-05T08:29:37.781 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../024-ceph-base_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:37.830 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-base (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:37.952 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-11-05T08:29:37.955 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-pecan. 2023-11-05T08:29:37.978 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-11-05T08:29:37.982 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-11-05T08:29:38.018 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-11-05T08:29:38.021 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-11-05T08:29:38.049 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-natsort. 2023-11-05T08:29:38.076 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2023-11-05T08:29:38.115 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-11-05T08:29:38.330 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-jaraco.text. 2023-11-05T08:29:38.356 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2023-11-05T08:29:38.393 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-certifi. 2023-11-05T08:29:38.405 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-11-05T08:29:38.420 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2023-11-05T08:29:38.427 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-logutils. 2023-11-05T08:29:38.454 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2023-11-05T08:29:38.459 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-11-05T08:29:38.493 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-11-05T08:29:38.533 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-11-05T08:29:38.559 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-11-05T08:29:38.600 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-11-05T08:29:38.779 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-idna. 2023-11-05T08:29:38.796 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-markupsafe. 2023-11-05T08:29:38.806 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../049-python3-idna_3.3-1_all.deb ... 2023-11-05T08:29:38.812 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-11-05T08:29:38.846 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-11-05T08:29:38.863 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-idna (3.3-1) ... 2023-11-05T08:29:38.876 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-11-05T08:29:38.902 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-11-05T08:29:38.943 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-11-05T08:29:39.107 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-mako. 2023-11-05T08:29:39.134 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-11-05T08:29:39.153 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-cheroot. 2023-11-05T08:29:39.174 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-11-05T08:29:39.179 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-11-05T08:29:39.182 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-urllib3. 2023-11-05T08:29:39.210 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-11-05T08:29:39.219 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-11-05T08:29:39.238 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-tempora. 2023-11-05T08:29:39.249 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-11-05T08:29:39.264 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2023-11-05T08:29:39.304 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-11-05T08:29:39.469 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-simplegeneric. 2023-11-05T08:29:39.495 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2023-11-05T08:29:39.522 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-11-05T08:29:39.535 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-11-05T08:29:39.548 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-11-05T08:29:39.569 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-requests. 2023-11-05T08:29:39.589 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-11-05T08:29:39.596 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-11-05T08:29:39.600 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-portend. 2023-11-05T08:29:39.627 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2023-11-05T08:29:39.635 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-11-05T08:29:39.667 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-11-05T08:29:39.815 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-singledispatch. 2023-11-05T08:29:39.842 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-11-05T08:29:39.881 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-11-05T08:29:39.908 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-jaraco.text. 2023-11-05T08:29:39.934 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2023-11-05T08:29:39.947 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-werkzeug. 2023-11-05T08:29:39.962 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-11-05T08:29:39.974 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-11-05T08:29:39.975 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-11-05T08:29:39.988 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2023-11-05T08:29:40.013 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-11-05T08:29:40.029 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-11-05T08:29:40.151 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-webob. 2023-11-05T08:29:40.178 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-11-05T08:29:40.217 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-11-05T08:29:40.286 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-11-05T08:29:40.312 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-11-05T08:29:40.325 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-11-05T08:29:40.332 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-cherrypy3. 2023-11-05T08:29:40.352 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-11-05T08:29:40.353 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-05T08:29:40.358 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2023-11-05T08:29:40.391 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:40.407 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-11-05T08:29:40.554 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-waitress. 2023-11-05T08:29:40.580 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-11-05T08:29:40.624 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-11-05T08:29:40.638 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-tempora. 2023-11-05T08:29:40.665 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2023-11-05T08:29:40.705 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-11-05T08:29:40.746 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-natsort. 2023-11-05T08:29:40.772 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2023-11-05T08:29:40.782 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-11-05T08:29:40.809 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:40.812 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-11-05T08:29:40.848 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:40.940 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-tempita. 2023-11-05T08:29:40.966 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-portend. 2023-11-05T08:29:40.967 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-11-05T08:29:40.992 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2023-11-05T08:29:41.006 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-11-05T08:29:41.024 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-11-05T08:29:41.116 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-logutils. 2023-11-05T08:29:41.142 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2023-11-05T08:29:41.162 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-openssl. 2023-11-05T08:29:41.182 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-11-05T08:29:41.186 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../055-python3-openssl_21.0.0-1_all.deb ... 2023-11-05T08:29:41.226 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-11-05T08:29:41.301 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-paste. 2023-11-05T08:29:41.319 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-11-05T08:29:41.328 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-11-05T08:29:41.345 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2023-11-05T08:29:41.367 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-11-05T08:29:41.385 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-11-05T08:29:41.462 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-markupsafe. 2023-11-05T08:29:41.488 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-11-05T08:29:41.521 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-mgr. 2023-11-05T08:29:41.529 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-11-05T08:29:41.548 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../056-ceph-mgr_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:41.579 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-mgr (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:41.688 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-cherrypy3. 2023-11-05T08:29:41.695 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-11-05T08:29:41.712 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-11-05T08:29:41.714 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2023-11-05T08:29:41.753 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-11-05T08:29:41.754 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-11-05T08:29:41.848 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-mako. 2023-11-05T08:29:41.875 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-11-05T08:29:41.915 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-11-05T08:29:41.982 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-mon. 2023-11-05T08:29:42.009 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../057-ceph-mon_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:42.031 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-pastedeploy. 2023-11-05T08:29:42.049 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-mon (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:42.059 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2023-11-05T08:29:42.066 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-natsort. 2023-11-05T08:29:42.092 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2023-11-05T08:29:42.098 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-11-05T08:29:42.132 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-11-05T08:29:42.368 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-webtest. 2023-11-05T08:29:42.395 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2023-11-05T08:29:42.426 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-11-05T08:29:42.444 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-logutils. 2023-11-05T08:29:42.470 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2023-11-05T08:29:42.511 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-11-05T08:29:42.720 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-pecan. 2023-11-05T08:29:42.747 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-11-05T08:29:42.787 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-11-05T08:29:42.813 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-markupsafe. 2023-11-05T08:29:42.840 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-11-05T08:29:42.880 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-11-05T08:29:42.969 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-osd. 2023-11-05T08:29:42.996 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../058-ceph-osd_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:43.035 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-osd (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:43.079 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-simplegeneric. 2023-11-05T08:29:43.105 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2023-11-05T08:29:43.145 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-11-05T08:29:43.158 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-mako. 2023-11-05T08:29:43.183 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-certifi. 2023-11-05T08:29:43.184 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-11-05T08:29:43.210 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2023-11-05T08:29:43.224 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-11-05T08:29:43.250 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-11-05T08:29:43.432 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-singledispatch. 2023-11-05T08:29:43.458 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-11-05T08:29:43.498 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-11-05T08:29:43.536 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-simplegeneric. 2023-11-05T08:29:43.562 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2023-11-05T08:29:43.577 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-idna. 2023-11-05T08:29:43.594 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-11-05T08:29:43.604 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../049-python3-idna_3.3-1_all.deb ... 2023-11-05T08:29:43.644 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-idna (3.3-1) ... 2023-11-05T08:29:43.794 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-webob. 2023-11-05T08:29:43.823 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-11-05T08:29:43.880 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-singledispatch. 2023-11-05T08:29:43.907 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-11-05T08:29:43.946 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-11-05T08:29:43.963 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-urllib3. 2023-11-05T08:29:43.977 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-11-05T08:29:43.990 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-11-05T08:29:44.030 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-11-05T08:29:44.332 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-waitress. 2023-11-05T08:29:44.358 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-11-05T08:29:44.358 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-requests. 2023-11-05T08:29:44.366 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-webob. 2023-11-05T08:29:44.385 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-11-05T08:29:44.392 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-11-05T08:29:44.401 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-11-05T08:29:44.425 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-11-05T08:29:44.449 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-11-05T08:29:44.710 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-tempita. 2023-11-05T08:29:44.736 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-11-05T08:29:44.752 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-werkzeug. 2023-11-05T08:29:44.780 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-11-05T08:29:44.785 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-11-05T08:29:44.810 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-waitress. 2023-11-05T08:29:44.819 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-11-05T08:29:44.837 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-11-05T08:29:44.880 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-11-05T08:29:44.938 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph. 2023-11-05T08:29:44.954 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../059-ceph_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:44.988 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:45.080 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-paste. 2023-11-05T08:29:45.106 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-11-05T08:29:45.114 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-11-05T08:29:45.130 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-05T08:29:45.147 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-11-05T08:29:45.171 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-tempita. 2023-11-05T08:29:45.180 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:45.198 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-11-05T08:29:45.238 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-11-05T08:29:45.316 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-fuse. 2023-11-05T08:29:45.343 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../060-ceph-fuse_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:45.383 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-fuse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:45.476 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-11-05T08:29:45.491 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-paste. 2023-11-05T08:29:45.503 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-11-05T08:29:45.517 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-11-05T08:29:45.543 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-11-05T08:29:45.549 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-11-05T08:29:45.577 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-11-05T08:29:45.604 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:45.643 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:45.761 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-mds. 2023-11-05T08:29:45.789 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../061-ceph-mds_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:45.813 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-pastedeploy. 2023-11-05T08:29:45.827 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-mds (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:45.839 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2023-11-05T08:29:45.877 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-11-05T08:29:45.879 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-11-05T08:29:45.904 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-11-05T08:29:45.944 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-11-05T08:29:45.954 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-openssl. 2023-11-05T08:29:45.981 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../055-python3-openssl_21.0.0-1_all.deb ... 2023-11-05T08:29:46.021 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-11-05T08:29:46.176 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-webtest. 2023-11-05T08:29:46.202 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2023-11-05T08:29:46.205 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-pastedeploy. 2023-11-05T08:29:46.214 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package cephadm. 2023-11-05T08:29:46.231 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2023-11-05T08:29:46.241 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../062-cephadm_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:46.242 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-11-05T08:29:46.263 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-11-05T08:29:46.280 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking cephadm (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:46.366 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-mgr. 2023-11-05T08:29:46.393 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../056-ceph-mgr_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:46.432 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-mgr (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:46.542 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-asyncssh. 2023-11-05T08:29:46.554 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-pecan. 2023-11-05T08:29:46.566 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-webtest. 2023-11-05T08:29:46.569 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../063-python3-asyncssh_2.5.0-1_all.deb ... 2023-11-05T08:29:46.580 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-11-05T08:29:46.592 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2023-11-05T08:29:46.608 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-11-05T08:29:46.632 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-11-05T08:29:46.645 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-11-05T08:29:46.760 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-mon. 2023-11-05T08:29:46.787 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../057-ceph-mon_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:46.818 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-mon (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:46.928 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python-babel-localedata. 2023-11-05T08:29:46.955 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../064-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-11-05T08:29:46.969 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-pecan. 2023-11-05T08:29:46.994 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-11-05T08:29:46.995 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-11-05T08:29:47.011 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-certifi. 2023-11-05T08:29:47.035 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-11-05T08:29:47.037 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2023-11-05T08:29:47.085 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-11-05T08:29:47.406 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-certifi. 2023-11-05T08:29:47.433 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2023-11-05T08:29:47.464 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-idna. 2023-11-05T08:29:47.473 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-11-05T08:29:47.490 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../049-python3-idna_3.3-1_all.deb ... 2023-11-05T08:29:47.539 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-idna (3.3-1) ... 2023-11-05T08:29:47.780 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-osd. 2023-11-05T08:29:47.801 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-idna. 2023-11-05T08:29:47.807 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../058-ceph-osd_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:47.827 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../049-python3-idna_3.3-1_all.deb ... 2023-11-05T08:29:47.846 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-osd (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:47.867 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-idna (3.3-1) ... 2023-11-05T08:29:47.877 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-urllib3. 2023-11-05T08:29:47.903 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-11-05T08:29:47.951 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-11-05T08:29:47.991 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-babel. 2023-11-05T08:29:48.019 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../065-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-11-05T08:29:48.057 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-11-05T08:29:48.145 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-urllib3. 2023-11-05T08:29:48.161 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-11-05T08:29:48.195 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-11-05T08:29:48.314 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-requests. 2023-11-05T08:29:48.340 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-11-05T08:29:48.369 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-jinja2. 2023-11-05T08:29:48.389 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-11-05T08:29:48.397 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../066-python3-jinja2_3.0.3-1_all.deb ... 2023-11-05T08:29:48.435 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-11-05T08:29:48.473 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-requests. 2023-11-05T08:29:48.488 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-11-05T08:29:48.525 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-11-05T08:29:48.734 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-werkzeug. 2023-11-05T08:29:48.760 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-11-05T08:29:48.781 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-11-05T08:29:48.808 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../067-ceph-mgr-cephadm_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-05T08:29:48.809 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-11-05T08:29:48.809 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-werkzeug. 2023-11-05T08:29:48.835 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-11-05T08:29:48.855 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:48.875 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-11-05T08:29:49.122 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-11-05T08:29:49.137 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-05T08:29:49.154 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-11-05T08:29:49.169 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-05T08:29:49.180 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:49.201 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-bcrypt. 2023-11-05T08:29:49.203 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:49.228 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../068-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-11-05T08:29:49.267 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-11-05T08:29:49.559 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-11-05T08:29:49.585 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:49.594 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-11-05T08:29:49.621 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-repoze.lru. 2023-11-05T08:29:49.621 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:49.624 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:49.648 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../069-python3-repoze.lru_0.7-2_all.deb ... 2023-11-05T08:29:49.662 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:49.687 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-11-05T08:29:49.917 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph. 2023-11-05T08:29:49.919 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-openssl. 2023-11-05T08:29:49.944 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../059-ceph_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:49.945 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../055-python3-openssl_21.0.0-1_all.deb ... 2023-11-05T08:29:49.973 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-openssl. 2023-11-05T08:29:49.983 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:49.985 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-11-05T08:29:49.990 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-routes. 2023-11-05T08:29:50.000 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../055-python3-openssl_21.0.0-1_all.deb ... 2023-11-05T08:29:50.018 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../070-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-11-05T08:29:50.040 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-11-05T08:29:50.056 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-11-05T08:29:50.252 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-fuse. 2023-11-05T08:29:50.279 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../060-ceph-fuse_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:50.297 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-mgr. 2023-11-05T08:29:50.310 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-fuse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:50.324 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../056-ceph-mgr_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:50.365 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-mgr (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:50.385 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-mgr. 2023-11-05T08:29:50.410 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-11-05T08:29:50.412 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../056-ceph-mgr_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:50.438 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../071-ceph-mgr-dashboard_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-05T08:29:50.451 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-mgr (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:50.476 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:50.705 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-mds. 2023-11-05T08:29:50.732 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../061-ceph-mds_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:50.780 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-mds (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:50.783 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-mon. 2023-11-05T08:29:50.809 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../057-ceph-mon_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:50.832 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-mon. 2023-11-05T08:29:50.849 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-mon (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:50.858 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../057-ceph-mon_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:50.898 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-mon (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:50.930 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-11-05T08:29:50.957 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../072-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-11-05T08:29:51.005 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-11-05T08:29:51.167 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package cephadm. 2023-11-05T08:29:51.194 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../062-cephadm_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:51.233 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking cephadm (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:51.519 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-asyncssh. 2023-11-05T08:29:51.547 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../063-python3-asyncssh_2.5.0-1_all.deb ... 2023-11-05T08:29:51.550 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-joblib. 2023-11-05T08:29:51.578 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-11-05T08:29:51.578 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../073-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-11-05T08:29:51.625 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-11-05T08:29:51.803 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-osd. 2023-11-05T08:29:51.830 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../058-ceph-osd_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:51.869 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-osd (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:51.902 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-osd. 2023-11-05T08:29:51.905 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python-babel-localedata. 2023-11-05T08:29:51.929 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../058-ceph-osd_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:51.933 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../064-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-11-05T08:29:51.969 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-osd (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:51.980 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-11-05T08:29:52.012 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-11-05T08:29:52.040 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../074-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-11-05T08:29:52.078 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-11-05T08:29:52.415 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-sklearn. 2023-11-05T08:29:52.442 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../075-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-11-05T08:29:52.490 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-11-05T08:29:53.027 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-babel. 2023-11-05T08:29:53.055 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../065-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-11-05T08:29:53.094 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-11-05T08:29:53.190 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-11-05T08:29:53.215 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../076-ceph-mgr-diskprediction-local_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-05T08:29:53.256 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:53.413 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-jinja2. 2023-11-05T08:29:53.441 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../066-python3-jinja2_3.0.3-1_all.deb ... 2023-11-05T08:29:53.480 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-11-05T08:29:53.639 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph. 2023-11-05T08:29:53.654 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../059-ceph_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:53.689 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:53.766 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph. 2023-11-05T08:29:53.792 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../059-ceph_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:53.841 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:54.008 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-fuse. 2023-11-05T08:29:54.035 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../060-ceph-fuse_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:54.050 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-11-05T08:29:54.078 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../067-ceph-mgr-cephadm_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-05T08:29:54.117 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:54.125 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-fuse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:54.268 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-cachetools. 2023-11-05T08:29:54.295 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../077-python3-cachetools_5.0.0-1_all.deb ... 2023-11-05T08:29:54.334 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-11-05T08:29:54.360 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-fuse. 2023-11-05T08:29:54.387 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../060-ceph-fuse_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:54.427 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-fuse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:54.437 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-bcrypt. 2023-11-05T08:29:54.465 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../068-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-11-05T08:29:54.503 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-11-05T08:29:54.504 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-mds. 2023-11-05T08:29:54.530 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../061-ceph-mds_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:54.602 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-mds (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:54.947 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-mds. 2023-11-05T08:29:54.971 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-pyasn1. 2023-11-05T08:29:54.973 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-repoze.lru. 2023-11-05T08:29:54.974 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../061-ceph-mds_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:54.998 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../078-python3-pyasn1_0.4.8-1_all.deb ... 2023-11-05T08:29:55.001 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../069-python3-repoze.lru_0.7-2_all.deb ... 2023-11-05T08:29:55.014 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-mds (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:55.029 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-11-05T08:29:55.039 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-11-05T08:29:55.148 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package cephadm. 2023-11-05T08:29:55.174 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../062-cephadm_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:55.206 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking cephadm (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:55.301 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-routes. 2023-11-05T08:29:55.307 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-11-05T08:29:55.324 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../079-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-11-05T08:29:55.328 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../070-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-11-05T08:29:55.359 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-11-05T08:29:55.365 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-11-05T08:29:55.410 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package cephadm. 2023-11-05T08:29:55.426 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../062-cephadm_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:29:55.460 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking cephadm (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:55.492 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-asyncssh. 2023-11-05T08:29:55.519 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../063-python3-asyncssh_2.5.0-1_all.deb ... 2023-11-05T08:29:55.558 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-11-05T08:29:55.662 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-11-05T08:29:55.668 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-rsa. 2023-11-05T08:29:55.678 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../071-ceph-mgr-dashboard_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-05T08:29:55.696 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../080-python3-rsa_4.8-1_all.deb ... 2023-11-05T08:29:55.712 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:55.735 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-rsa (4.8-1) ... 2023-11-05T08:29:55.756 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-asyncssh. 2023-11-05T08:29:55.783 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../063-python3-asyncssh_2.5.0-1_all.deb ... 2023-11-05T08:29:55.822 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-11-05T08:29:55.872 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python-babel-localedata. 2023-11-05T08:29:55.898 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../064-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-11-05T08:29:55.938 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-11-05T08:29:56.072 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-google-auth. 2023-11-05T08:29:56.099 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../081-python3-google-auth_1.5.1-3_all.deb ... 2023-11-05T08:29:56.138 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-11-05T08:29:56.140 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-11-05T08:29:56.160 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python-babel-localedata. 2023-11-05T08:29:56.168 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../072-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-11-05T08:29:56.187 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../064-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-11-05T08:29:56.207 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-11-05T08:29:56.226 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-11-05T08:29:56.467 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-11-05T08:29:56.494 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../082-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-11-05T08:29:56.533 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-11-05T08:29:56.719 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-joblib. 2023-11-05T08:29:56.746 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../073-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-11-05T08:29:56.785 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-11-05T08:29:56.879 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-websocket. 2023-11-05T08:29:56.895 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../083-python3-websocket_1.2.3-1_all.deb ... 2023-11-05T08:29:56.937 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-11-05T08:29:56.993 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-babel. 2023-11-05T08:29:57.020 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../065-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-11-05T08:29:57.059 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-11-05T08:29:57.164 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-11-05T08:29:57.191 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../074-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-11-05T08:29:57.230 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-11-05T08:29:57.282 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-kubernetes. 2023-11-05T08:29:57.310 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../084-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-11-05T08:29:57.349 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-11-05T08:29:57.363 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-jinja2. 2023-11-05T08:29:57.386 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../066-python3-jinja2_3.0.3-1_all.deb ... 2023-11-05T08:29:57.421 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-11-05T08:29:57.508 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-sklearn. 2023-11-05T08:29:57.535 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../075-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-11-05T08:29:57.549 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-babel. 2023-11-05T08:29:57.566 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-11-05T08:29:57.576 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../065-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-11-05T08:29:57.616 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-11-05T08:29:57.783 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-11-05T08:29:57.810 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../067-ceph-mgr-cephadm_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-05T08:29:57.841 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:57.937 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-jinja2. 2023-11-05T08:29:57.965 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../066-python3-jinja2_3.0.3-1_all.deb ... 2023-11-05T08:29:58.004 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-11-05T08:29:58.012 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-11-05T08:29:58.029 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../085-ceph-mgr-k8sevents_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-05T08:29:58.070 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:58.178 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-bcrypt. 2023-11-05T08:29:58.204 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../068-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-11-05T08:29:58.244 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-11-05T08:29:58.258 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-11-05T08:29:58.286 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../076-ceph-mgr-diskprediction-local_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-05T08:29:58.325 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:58.390 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libonig5:amd64. 2023-11-05T08:29:58.399 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-11-05T08:29:58.418 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../086-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-11-05T08:29:58.427 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../067-ceph-mgr-cephadm_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-05T08:29:58.465 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-11-05T08:29:58.466 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:58.530 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-repoze.lru. 2023-11-05T08:29:58.546 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../069-python3-repoze.lru_0.7-2_all.deb ... 2023-11-05T08:29:58.580 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-11-05T08:29:58.776 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libjq1:amd64. 2023-11-05T08:29:58.793 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../087-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-11-05T08:29:58.826 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-11-05T08:29:58.850 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-routes. 2023-11-05T08:29:58.855 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-bcrypt. 2023-11-05T08:29:58.877 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../070-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-11-05T08:29:58.882 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../068-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-11-05T08:29:58.916 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-11-05T08:29:58.921 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-11-05T08:29:59.071 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package jq. 2023-11-05T08:29:59.098 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../088-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-11-05T08:29:59.137 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-11-05T08:29:59.203 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-11-05T08:29:59.216 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-repoze.lru. 2023-11-05T08:29:59.230 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../071-ceph-mgr-dashboard_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-05T08:29:59.243 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../069-python3-repoze.lru_0.7-2_all.deb ... 2023-11-05T08:29:59.261 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:29:59.282 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-11-05T08:29:59.286 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-cachetools. 2023-11-05T08:29:59.313 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../077-python3-cachetools_5.0.0-1_all.deb ... 2023-11-05T08:29:59.352 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-11-05T08:29:59.390 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package socat. 2023-11-05T08:29:59.417 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../089-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-11-05T08:29:59.456 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-11-05T08:29:59.587 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-routes. 2023-11-05T08:29:59.615 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../070-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-11-05T08:29:59.654 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-11-05T08:29:59.655 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-pyasn1. 2023-11-05T08:29:59.683 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../078-python3-pyasn1_0.4.8-1_all.deb ... 2023-11-05T08:29:59.698 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-11-05T08:29:59.721 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-11-05T08:29:59.725 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../072-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-11-05T08:29:59.764 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-11-05T08:29:59.810 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package xmlstarlet. 2023-11-05T08:29:59.837 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../090-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-11-05T08:29:59.876 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-11-05T08:29:59.983 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-11-05T08:30:00.009 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../071-ceph-mgr-dashboard_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-05T08:30:00.049 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:00.050 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-11-05T08:30:00.078 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../079-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-11-05T08:30:00.117 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-11-05T08:30:00.234 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-joblib. 2023-11-05T08:30:00.239 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-test. 2023-11-05T08:30:00.261 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../073-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-11-05T08:30:00.267 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../091-ceph-test_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:30:00.293 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-11-05T08:30:00.305 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-test (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:00.395 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-rsa. 2023-11-05T08:30:00.422 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../080-python3-rsa_4.8-1_all.deb ... 2023-11-05T08:30:00.461 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-rsa (4.8-1) ... 2023-11-05T08:30:00.537 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-11-05T08:30:00.564 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../072-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-11-05T08:30:00.604 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-11-05T08:30:00.638 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-11-05T08:30:00.665 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../074-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-11-05T08:30:00.704 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-11-05T08:30:00.781 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-google-auth. 2023-11-05T08:30:00.809 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../081-python3-google-auth_1.5.1-3_all.deb ... 2023-11-05T08:30:00.848 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-11-05T08:30:00.966 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-sklearn. 2023-11-05T08:30:00.993 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../075-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-11-05T08:30:01.024 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-11-05T08:30:01.125 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-joblib. 2023-11-05T08:30:01.152 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../073-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-11-05T08:30:01.167 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-11-05T08:30:01.192 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-11-05T08:30:01.195 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../082-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-11-05T08:30:01.234 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-11-05T08:30:01.555 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-11-05T08:30:01.582 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../074-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-11-05T08:30:01.622 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-11-05T08:30:01.655 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-websocket. 2023-11-05T08:30:01.683 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../083-python3-websocket_1.2.3-1_all.deb ... 2023-11-05T08:30:01.721 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-11-05T08:30:01.774 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-11-05T08:30:01.801 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../076-ceph-mgr-diskprediction-local_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-05T08:30:01.840 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:01.926 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-sklearn. 2023-11-05T08:30:01.953 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../075-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-11-05T08:30:01.992 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-11-05T08:30:02.042 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-kubernetes. 2023-11-05T08:30:02.069 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../084-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-11-05T08:30:02.108 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-11-05T08:30:02.626 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-volume. 2023-11-05T08:30:02.654 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../092-ceph-volume_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-05T08:30:02.692 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-volume (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:02.797 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-11-05T08:30:02.802 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-cachetools. 2023-11-05T08:30:02.813 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../085-ceph-mgr-k8sevents_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-05T08:30:02.820 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-11-05T08:30:02.829 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../077-python3-cachetools_5.0.0-1_all.deb ... 2023-11-05T08:30:02.848 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../076-ceph-mgr-diskprediction-local_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-05T08:30:02.855 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:02.868 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-11-05T08:30:02.895 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:03.005 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libcephfs-dev. 2023-11-05T08:30:03.032 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../093-libcephfs-dev_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:30:03.071 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libcephfs-dev (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:03.163 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-pyasn1. 2023-11-05T08:30:03.190 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../078-python3-pyasn1_0.4.8-1_all.deb ... 2023-11-05T08:30:03.191 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libonig5:amd64. 2023-11-05T08:30:03.219 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../086-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-11-05T08:30:03.229 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-11-05T08:30:03.258 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-11-05T08:30:03.491 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package nvme-cli. 2023-11-05T08:30:03.519 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../094-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-11-05T08:30:03.557 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-11-05T08:30:03.586 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libjq1:amd64. 2023-11-05T08:30:03.613 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../087-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-11-05T08:30:03.616 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-11-05T08:30:03.643 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../079-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-11-05T08:30:03.652 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-11-05T08:30:03.691 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-11-05T08:30:03.888 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package jq. 2023-11-05T08:30:03.904 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-11-05T08:30:03.905 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../088-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-11-05T08:30:03.921 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../095-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-11-05T08:30:03.938 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-11-05T08:30:03.962 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-11-05T08:30:03.999 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-cachetools. 2023-11-05T08:30:04.002 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-rsa. 2023-11-05T08:30:04.026 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../077-python3-cachetools_5.0.0-1_all.deb ... 2023-11-05T08:30:04.029 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../080-python3-rsa_4.8-1_all.deb ... 2023-11-05T08:30:04.066 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-11-05T08:30:04.068 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-rsa (4.8-1) ... 2023-11-05T08:30:04.165 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package socat. 2023-11-05T08:30:04.182 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../089-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-11-05T08:30:04.224 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-11-05T08:30:04.381 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-google-auth. 2023-11-05T08:30:04.399 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-iniconfig. 2023-11-05T08:30:04.407 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../081-python3-google-auth_1.5.1-3_all.deb ... 2023-11-05T08:30:04.427 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../096-python3-iniconfig_1.1.1-2_all.deb ... 2023-11-05T08:30:04.447 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-11-05T08:30:04.474 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-11-05T08:30:04.543 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package xmlstarlet. 2023-11-05T08:30:04.560 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../090-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-11-05T08:30:04.601 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-11-05T08:30:04.628 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-pyasn1. 2023-11-05T08:30:04.655 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../078-python3-pyasn1_0.4.8-1_all.deb ... 2023-11-05T08:30:04.694 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-11-05T08:30:04.735 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-pastescript. 2023-11-05T08:30:04.751 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../097-python3-pastescript_2.0.2-4_all.deb ... 2023-11-05T08:30:04.766 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-11-05T08:30:04.784 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-11-05T08:30:04.794 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../082-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-11-05T08:30:04.866 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-11-05T08:30:04.973 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-test. 2023-11-05T08:30:05.000 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../091-ceph-test_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:30:05.024 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-11-05T08:30:05.039 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-test (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:05.051 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../079-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-11-05T08:30:05.097 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-pluggy. 2023-11-05T08:30:05.098 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-11-05T08:30:05.124 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../098-python3-pluggy_0.13.0-7.1_all.deb ... 2023-11-05T08:30:05.163 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-11-05T08:30:05.313 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-websocket. 2023-11-05T08:30:05.340 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../083-python3-websocket_1.2.3-1_all.deb ... 2023-11-05T08:30:05.378 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-rsa. 2023-11-05T08:30:05.388 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-11-05T08:30:05.405 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../080-python3-rsa_4.8-1_all.deb ... 2023-11-05T08:30:05.445 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-rsa (4.8-1) ... 2023-11-05T08:30:05.508 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-psutil. 2023-11-05T08:30:05.536 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../099-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-11-05T08:30:05.574 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-11-05T08:30:05.757 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-google-auth. 2023-11-05T08:30:05.766 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-kubernetes. 2023-11-05T08:30:05.784 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../081-python3-google-auth_1.5.1-3_all.deb ... 2023-11-05T08:30:05.793 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../084-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-11-05T08:30:05.824 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-11-05T08:30:05.841 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-11-05T08:30:05.902 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-py. 2023-11-05T08:30:05.930 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../100-python3-py_1.10.0-1_all.deb ... 2023-11-05T08:30:05.968 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-py (1.10.0-1) ... 2023-11-05T08:30:06.153 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-11-05T08:30:06.181 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../082-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-11-05T08:30:06.220 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-11-05T08:30:06.289 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-pygments. 2023-11-05T08:30:06.317 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../101-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-11-05T08:30:06.355 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-11-05T08:30:06.497 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-11-05T08:30:06.513 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../085-ceph-mgr-k8sevents_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-05T08:30:06.555 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:06.609 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-websocket. 2023-11-05T08:30:06.636 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../083-python3-websocket_1.2.3-1_all.deb ... 2023-11-05T08:30:06.675 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-11-05T08:30:06.817 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-pyinotify. 2023-11-05T08:30:06.846 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../102-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-11-05T08:30:06.874 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libonig5:amd64. 2023-11-05T08:30:06.884 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-11-05T08:30:06.902 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../086-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-11-05T08:30:06.940 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-11-05T08:30:06.989 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-kubernetes. 2023-11-05T08:30:07.016 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../084-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-11-05T08:30:07.055 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-11-05T08:30:07.170 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-toml. 2023-11-05T08:30:07.198 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../103-python3-toml_0.10.2-1_all.deb ... 2023-11-05T08:30:07.237 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-11-05T08:30:07.243 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libjq1:amd64. 2023-11-05T08:30:07.270 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../087-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-11-05T08:30:07.310 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-11-05T08:30:07.452 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-volume. 2023-11-05T08:30:07.464 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-pytest. 2023-11-05T08:30:07.479 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../092-ceph-volume_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-05T08:30:07.481 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../104-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-11-05T08:30:07.509 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-volume (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:07.522 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-11-05T08:30:07.587 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package jq. 2023-11-05T08:30:07.603 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../088-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-11-05T08:30:07.646 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-11-05T08:30:07.822 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libcephfs-dev. 2023-11-05T08:30:07.828 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-11-05T08:30:07.850 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../093-libcephfs-dev_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:30:07.856 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../085-ceph-mgr-k8sevents_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-05T08:30:07.868 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-simplejson. 2023-11-05T08:30:07.880 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libcephfs-dev (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:07.896 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-11-05T08:30:07.896 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:07.932 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package socat. 2023-11-05T08:30:07.934 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-11-05T08:30:07.960 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../089-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-11-05T08:30:07.998 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-11-05T08:30:08.211 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-11-05T08:30:08.228 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-11-05T08:30:08.241 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libonig5:amd64. 2023-11-05T08:30:08.242 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package nvme-cli. 2023-11-05T08:30:08.268 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../086-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-11-05T08:30:08.269 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../094-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-11-05T08:30:08.270 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-11-05T08:30:08.293 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package xmlstarlet. 2023-11-05T08:30:08.308 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-11-05T08:30:08.309 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../090-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-11-05T08:30:08.316 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-11-05T08:30:08.351 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-11-05T08:30:08.655 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-11-05T08:30:08.671 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../095-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-11-05T08:30:08.713 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-11-05T08:30:08.736 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libjq1:amd64. 2023-11-05T08:30:08.747 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-test. 2023-11-05T08:30:08.763 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../087-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-11-05T08:30:08.764 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../091-ceph-test_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:30:08.805 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-test (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:08.811 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-11-05T08:30:09.057 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package radosgw. 2023-11-05T08:30:09.085 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../107-radosgw_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:30:09.090 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package jq. 2023-11-05T08:30:09.106 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../088-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-11-05T08:30:09.123 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking radosgw (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:09.141 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-iniconfig. 2023-11-05T08:30:09.148 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-11-05T08:30:09.169 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../096-python3-iniconfig_1.1.1-2_all.deb ... 2023-11-05T08:30:09.208 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-11-05T08:30:09.443 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package socat. 2023-11-05T08:30:09.459 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../089-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-11-05T08:30:09.510 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-11-05T08:30:09.511 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-pastescript. 2023-11-05T08:30:09.538 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../097-python3-pastescript_2.0.2-4_all.deb ... 2023-11-05T08:30:09.577 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-11-05T08:30:09.880 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-pluggy. 2023-11-05T08:30:09.902 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package rbd-fuse. 2023-11-05T08:30:09.906 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package xmlstarlet. 2023-11-05T08:30:09.908 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../098-python3-pluggy_0.13.0-7.1_all.deb ... 2023-11-05T08:30:09.919 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../108-rbd-fuse_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:30:09.933 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../090-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-11-05T08:30:09.946 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-11-05T08:30:09.960 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking rbd-fuse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:09.980 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-11-05T08:30:10.225 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-psutil. 2023-11-05T08:30:10.241 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../099-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-11-05T08:30:10.274 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-11-05T08:30:10.370 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-test. 2023-11-05T08:30:10.397 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../091-ceph-test_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:30:10.445 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-test (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:10.531 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-11-05T08:30:10.627 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-py. 2023-11-05T08:30:10.655 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../100-python3-py_1.10.0-1_all.deb ... 2023-11-05T08:30:10.702 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-py (1.10.0-1) ... 2023-11-05T08:30:10.789 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-11-05T08:30:10.907 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-11-05T08:30:11.022 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-pygments. 2023-11-05T08:30:11.050 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../101-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-11-05T08:30:11.088 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-11-05T08:30:11.160 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-11-05T08:30:11.168 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package ceph-volume. 2023-11-05T08:30:11.195 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../092-ceph-volume_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-05T08:30:11.234 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking ceph-volume (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:11.491 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-11-05T08:30:11.538 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package libcephfs-dev. 2023-11-05T08:30:11.566 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../093-libcephfs-dev_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:30:11.568 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-pyinotify. 2023-11-05T08:30:11.596 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../102-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-11-05T08:30:11.604 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking libcephfs-dev (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:11.634 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-11-05T08:30:11.945 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-toml. 2023-11-05T08:30:11.960 INFO:teuthology.orchestra.run.smithi053.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-11-05T08:30:11.966 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package nvme-cli. 2023-11-05T08:30:11.973 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../103-python3-toml_0.10.2-1_all.deb ... 2023-11-05T08:30:11.993 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../094-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-11-05T08:30:12.011 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-11-05T08:30:12.041 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-11-05T08:30:12.051 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up cephadm (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:12.314 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-pytest. 2023-11-05T08:30:12.342 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../104-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-11-05T08:30:12.379 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-11-05T08:30:12.380 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-11-05T08:30:12.395 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../095-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-11-05T08:30:12.429 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-11-05T08:30:12.569 INFO:teuthology.orchestra.run.smithi053.stdout:Adding system user cephadm....done 2023-11-05T08:30:12.707 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-11-05T08:30:12.718 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-simplejson. 2023-11-05T08:30:12.746 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-11-05T08:30:12.784 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-11-05T08:30:12.857 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-iniconfig. 2023-11-05T08:30:12.885 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../096-python3-iniconfig_1.1.1-2_all.deb ... 2023-11-05T08:30:12.918 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-volume. 2023-11-05T08:30:12.923 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-11-05T08:30:12.945 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../092-ceph-volume_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-05T08:30:12.992 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-11-05T08:30:12.992 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-volume (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:13.070 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-11-05T08:30:13.098 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-11-05T08:30:13.136 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-11-05T08:30:13.201 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-pastescript. 2023-11-05T08:30:13.229 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../097-python3-pastescript_2.0.2-4_all.deb ... 2023-11-05T08:30:13.259 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-11-05T08:30:13.268 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-11-05T08:30:13.314 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libcephfs-dev. 2023-11-05T08:30:13.341 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../093-libcephfs-dev_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:30:13.385 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-11-05T08:30:13.388 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libcephfs-dev (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:13.580 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-pluggy. 2023-11-05T08:30:13.607 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../098-python3-pluggy_0.13.0-7.1_all.deb ... 2023-11-05T08:30:13.645 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-11-05T08:30:13.646 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-11-05T08:30:13.835 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package nvme-cli. 2023-11-05T08:30:13.857 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package radosgw. 2023-11-05T08:30:13.862 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../094-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-11-05T08:30:13.885 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../107-radosgw_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:30:13.910 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-11-05T08:30:13.913 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-11-05T08:30:13.915 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking radosgw (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:13.957 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-psutil. 2023-11-05T08:30:13.985 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../099-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-11-05T08:30:14.024 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-11-05T08:30:14.039 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-py (1.10.0-1) ... 2023-11-05T08:30:14.291 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-11-05T08:30:14.307 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../095-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-11-05T08:30:14.340 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-11-05T08:30:14.357 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-11-05T08:30:14.377 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-py. 2023-11-05T08:30:14.404 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../100-python3-py_1.10.0-1_all.deb ... 2023-11-05T08:30:14.443 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-py (1.10.0-1) ... 2023-11-05T08:30:14.701 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-11-05T08:30:14.726 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package rbd-fuse. 2023-11-05T08:30:14.743 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../108-rbd-fuse_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:30:14.780 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-pygments. 2023-11-05T08:30:14.784 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking rbd-fuse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:14.807 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../101-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-11-05T08:30:14.846 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-11-05T08:30:14.960 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-11-05T08:30:15.081 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-11-05T08:30:15.113 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-iniconfig. 2023-11-05T08:30:15.140 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../096-python3-iniconfig_1.1.1-2_all.deb ... 2023-11-05T08:30:15.179 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-11-05T08:30:15.269 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-11-05T08:30:15.334 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-pyinotify. 2023-11-05T08:30:15.348 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-11-05T08:30:15.362 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../102-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-11-05T08:30:15.400 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-11-05T08:30:15.474 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-11-05T08:30:15.474 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-pastescript. 2023-11-05T08:30:15.501 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../097-python3-pastescript_2.0.2-4_all.deb ... 2023-11-05T08:30:15.540 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-11-05T08:30:15.587 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-11-05T08:30:15.711 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-toml. 2023-11-05T08:30:15.725 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-11-05T08:30:15.739 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../103-python3-toml_0.10.2-1_all.deb ... 2023-11-05T08:30:15.778 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-11-05T08:30:15.830 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-11-05T08:30:15.854 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-pluggy. 2023-11-05T08:30:15.881 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../098-python3-pluggy_0.13.0-7.1_all.deb ... 2023-11-05T08:30:15.920 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-11-05T08:30:16.031 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-pytest. 2023-11-05T08:30:16.058 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../104-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-11-05T08:30:16.089 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-11-05T08:30:16.089 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-ceph-argparse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:16.118 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-11-05T08:30:16.199 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-psutil. 2023-11-05T08:30:16.227 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../099-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-11-05T08:30:16.266 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-11-05T08:30:16.357 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-11-05T08:30:16.434 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package python3-simplejson. 2023-11-05T08:30:16.461 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-11-05T08:30:16.483 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-11-05T08:30:16.500 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-11-05T08:30:16.613 INFO:teuthology.orchestra.run.smithi002.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-11-05T08:30:16.627 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-py. 2023-11-05T08:30:16.652 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../100-python3-py_1.10.0-1_all.deb ... 2023-11-05T08:30:16.693 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-py (1.10.0-1) ... 2023-11-05T08:30:16.734 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up cephadm (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:16.742 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-11-05T08:30:16.761 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-11-05T08:30:16.777 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-11-05T08:30:16.819 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-11-05T08:30:16.868 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-11-05T08:30:17.007 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-pygments. 2023-11-05T08:30:17.035 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../101-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-11-05T08:30:17.074 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-11-05T08:30:17.286 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-11-05T08:30:17.336 INFO:teuthology.orchestra.run.smithi002.stdout:Adding system user cephadm....done 2023-11-05T08:30:17.412 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-11-05T08:30:17.450 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-11-05T08:30:17.523 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package radosgw. 2023-11-05T08:30:17.551 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../107-radosgw_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:30:17.589 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking radosgw (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:17.662 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-pyinotify. 2023-11-05T08:30:17.690 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../102-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-11-05T08:30:17.705 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-11-05T08:30:17.736 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-11-05T08:30:17.972 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-11-05T08:30:18.074 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-toml. 2023-11-05T08:30:18.102 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../103-python3-toml_0.10.2-1_all.deb ... 2023-11-05T08:30:18.135 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-11-05T08:30:18.140 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-11-05T08:30:18.273 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-11-05T08:30:18.351 INFO:teuthology.orchestra.run.smithi069.stdout:Selecting previously unselected package rbd-fuse. 2023-11-05T08:30:18.367 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing to unpack .../108-rbd-fuse_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:30:18.394 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-11-05T08:30:18.409 INFO:teuthology.orchestra.run.smithi069.stdout:Unpacking rbd-fuse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:18.435 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-pytest. 2023-11-05T08:30:18.464 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../104-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-11-05T08:30:18.510 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-11-05T08:30:18.520 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-11-05T08:30:18.704 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-11-05T08:30:18.780 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-11-05T08:30:18.857 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-simplejson. 2023-11-05T08:30:18.885 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-11-05T08:30:18.932 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-11-05T08:30:18.959 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-11-05T08:30:18.964 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-11-05T08:30:19.039 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-11-05T08:30:19.090 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-11-05T08:30:19.165 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-py (1.10.0-1) ... 2023-11-05T08:30:19.201 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-11-05T08:30:19.218 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-11-05T08:30:19.219 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-11-05T08:30:19.260 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-11-05T08:30:19.336 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-11-05T08:30:19.384 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-11-05T08:30:19.454 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-11-05T08:30:19.475 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-11-05T08:30:19.746 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-11-05T08:30:19.757 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-11-05T08:30:19.844 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-11-05T08:30:20.103 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-11-05T08:30:20.106 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package radosgw. 2023-11-05T08:30:20.135 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../107-radosgw_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:30:20.139 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-11-05T08:30:20.189 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking radosgw (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:20.227 INFO:teuthology.orchestra.run.smithi069.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-11-05T08:30:20.350 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up cephadm (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:20.391 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-11-05T08:30:20.496 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-11-05T08:30:20.747 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-11-05T08:30:20.784 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-11-05T08:30:20.893 INFO:teuthology.orchestra.run.smithi069.stdout:Adding system user cephadm....done 2023-11-05T08:30:20.969 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package rbd-fuse. 2023-11-05T08:30:20.997 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../108-rbd-fuse_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-05T08:30:21.035 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking rbd-fuse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:21.048 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-11-05T08:30:21.058 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-11-05T08:30:21.093 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-11-05T08:30:21.309 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-ceph-argparse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:21.333 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-11-05T08:30:21.356 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-11-05T08:30:21.445 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-11-05T08:30:21.569 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-11-05T08:30:21.592 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-11-05T08:30:21.634 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-11-05T08:30:21.694 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-11-05T08:30:21.696 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-11-05T08:30:21.718 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-11-05T08:30:21.735 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-11-05T08:30:21.805 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-ceph-common (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:21.954 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-11-05T08:30:21.977 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-11-05T08:30:22.000 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-11-05T08:30:22.080 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-11-05T08:30:22.098 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-idna (3.3-1) ... 2023-11-05T08:30:22.245 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-11-05T08:30:22.369 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-11-05T08:30:22.371 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-py (1.10.0-1) ... 2023-11-05T08:30:22.407 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-11-05T08:30:22.456 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-11-05T08:30:22.574 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-11-05T08:30:22.689 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-11-05T08:30:22.828 INFO:teuthology.orchestra.run.smithi090.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-11-05T08:30:22.841 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-11-05T08:30:22.963 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up cephadm (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:23.058 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-11-05T08:30:23.184 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-11-05T08:30:23.401 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-11-05T08:30:23.494 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-11-05T08:30:23.542 INFO:teuthology.orchestra.run.smithi090.stdout:Adding system user cephadm....done 2023-11-05T08:30:23.646 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-05T08:30:23.677 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-11-05T08:30:23.679 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-11-05T08:30:23.780 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-toml (0.10.2-1) ... 2023-11-05T08:30:23.953 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-11-05T08:30:23.956 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-11-05T08:30:24.031 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-11-05T08:30:24.149 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-11-05T08:30:24.187 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-11-05T08:30:24.216 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-11-05T08:30:24.229 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-11-05T08:30:24.275 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-11-05T08:30:24.342 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-11-05T08:30:24.447 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-11-05T08:30:24.488 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-ceph-argparse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:24.573 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-11-05T08:30:24.601 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-11-05T08:30:24.648 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-11-05T08:30:24.748 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-11-05T08:30:24.854 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-11-05T08:30:24.857 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-11-05T08:30:24.919 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-11-05T08:30:24.955 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-py (1.10.0-1) ... 2023-11-05T08:30:25.082 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-11-05T08:30:25.100 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-11-05T08:30:25.170 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-05T08:30:25.251 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-11-05T08:30:25.257 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-11-05T08:30:25.280 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-11-05T08:30:25.391 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-11-05T08:30:25.406 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-11-05T08:30:25.610 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-11-05T08:30:25.701 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-11-05T08:30:25.768 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-11-05T08:30:25.861 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-11-05T08:30:25.894 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-11-05T08:30:25.977 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-11-05T08:30:26.020 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-11-05T08:30:26.027 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-11-05T08:30:26.121 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-11-05T08:30:26.245 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-11-05T08:30:26.296 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-11-05T08:30:26.364 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-11-05T08:30:26.436 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-11-05T08:30:26.512 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-11-05T08:30:26.597 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-11-05T08:30:26.615 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-11-05T08:30:26.738 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-11-05T08:30:26.772 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-11-05T08:30:26.867 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-ceph-argparse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:26.898 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-11-05T08:30:27.023 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-11-05T08:30:27.089 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-11-05T08:30:27.124 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-11-05T08:30:27.160 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-11-05T08:30:27.286 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-11-05T08:30:27.332 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-11-05T08:30:27.400 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-11-05T08:30:27.458 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-ceph-common (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:27.534 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-11-05T08:30:27.537 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-11-05T08:30:27.567 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-11-05T08:30:27.665 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-11-05T08:30:27.759 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-idna (3.3-1) ... 2023-11-05T08:30:27.835 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-11-05T08:30:27.835 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-11-05T08:30:27.960 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-11-05T08:30:28.024 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-11-05T08:30:28.077 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-11-05T08:30:28.086 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-11-05T08:30:28.145 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-11-05T08:30:28.150 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-11-05T08:30:28.388 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-11-05T08:30:28.413 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-11-05T08:30:28.427 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-11-05T08:30:28.678 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-11-05T08:30:28.764 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-11-05T08:30:28.790 INFO:teuthology.orchestra.run.smithi053.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-11-05T08:30:28.971 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-11-05T08:30:29.015 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-11-05T08:30:29.074 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-11-05T08:30:29.200 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-11-05T08:30:29.349 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-05T08:30:29.383 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-11-05T08:30:29.484 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-toml (0.10.2-1) ... 2023-11-05T08:30:29.618 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-11-05T08:30:29.666 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-11-05T08:30:29.684 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-11-05T08:30:29.751 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-11-05T08:30:29.877 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-11-05T08:30:29.886 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-portend (3.0.0-1) ... 2023-11-05T08:30:29.926 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-11-05T08:30:30.003 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-11-05T08:30:30.045 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-11-05T08:30:30.052 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-11-05T08:30:30.128 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-05T08:30:30.170 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-11-05T08:30:30.254 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-11-05T08:30:30.295 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-11-05T08:30:30.296 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-11-05T08:30:30.421 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-ceph-common (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:30.539 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-11-05T08:30:30.580 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-11-05T08:30:30.714 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-idna (3.3-1) ... 2023-11-05T08:30:30.782 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-05T08:30:30.796 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-11-05T08:30:30.908 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-11-05T08:30:31.007 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-11-05T08:30:31.032 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-11-05T08:30:31.034 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-11-05T08:30:31.181 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-11-05T08:30:31.283 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-11-05T08:30:31.335 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-11-05T08:30:31.408 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-rsa (4.8-1) ... 2023-11-05T08:30:31.425 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-11-05T08:30:31.594 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-11-05T08:30:31.685 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-11-05T08:30:31.802 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-11-05T08:30:31.854 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-11-05T08:30:31.961 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-11-05T08:30:32.087 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-11-05T08:30:32.096 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-11-05T08:30:32.204 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-11-05T08:30:32.271 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-05T08:30:32.347 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-11-05T08:30:32.405 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-toml (0.10.2-1) ... 2023-11-05T08:30:32.413 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-11-05T08:30:32.580 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-11-05T08:30:32.590 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-11-05T08:30:32.640 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-11-05T08:30:32.656 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-11-05T08:30:32.700 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-11-05T08:30:32.766 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-ceph-common (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:32.782 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-11-05T08:30:32.865 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-11-05T08:30:32.925 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-11-05T08:30:32.967 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-11-05T08:30:33.060 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-idna (3.3-1) ... 2023-11-05T08:30:33.093 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-11-05T08:30:33.200 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-11-05T08:30:33.235 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-11-05T08:30:33.334 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up librados2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:33.386 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-11-05T08:30:33.387 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-11-05T08:30:33.461 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up librgw2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:33.507 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-11-05T08:30:33.570 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:33.671 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libcephfs2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:33.695 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-11-05T08:30:33.753 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-05T08:30:33.788 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libradosstriper1 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:33.871 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-11-05T08:30:33.955 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-11-05T08:30:33.997 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-11-05T08:30:34.008 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-11-05T08:30:34.293 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up librbd1 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:34.299 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-11-05T08:30:34.333 INFO:teuthology.orchestra.run.smithi002.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-11-05T08:30:34.419 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-mgr-modules-core (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:34.545 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-fuse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:34.553 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-05T08:30:34.558 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-11-05T08:30:34.608 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-11-05T08:30:34.687 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-toml (0.10.2-1) ... 2023-11-05T08:30:34.734 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-11-05T08:30:34.767 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-11-05T08:30:34.767 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-11-05T08:30:34.826 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-11-05T08:30:34.938 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-11-05T08:30:35.065 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-11-05T08:30:35.119 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-11-05T08:30:35.144 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-11-05T08:30:35.191 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-11-05T08:30:35.272 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libcephfs-dev (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:35.361 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-11-05T08:30:35.372 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-rados (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:35.411 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-portend (3.0.0-1) ... 2023-11-05T08:30:35.484 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-11-05T08:30:35.498 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-11-05T08:30:35.613 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-11-05T08:30:35.671 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-05T08:30:35.737 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-11-05T08:30:35.739 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-11-05T08:30:35.772 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-11-05T08:30:35.996 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-05T08:30:36.031 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-11-05T08:30:36.081 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-11-05T08:30:36.106 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-11-05T08:30:36.157 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-11-05T08:30:36.232 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-11-05T08:30:36.374 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-11-05T08:30:36.459 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-11-05T08:30:36.525 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-11-05T08:30:36.659 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-11-05T08:30:36.745 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-rbd (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:36.776 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-11-05T08:30:36.784 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-11-05T08:30:36.809 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-rsa (4.8-1) ... 2023-11-05T08:30:36.846 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up rbd-fuse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:36.972 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-rgw (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:37.046 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-11-05T08:30:37.052 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-11-05T08:30:37.085 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-11-05T08:30:37.098 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-cephfs (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:37.224 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-common (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:37.272 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-11-05T08:30:37.353 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-11-05T08:30:37.442 INFO:teuthology.orchestra.run.smithi069.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-11-05T08:30:37.543 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-11-05T08:30:37.638 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-11-05T08:30:37.697 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-11-05T08:30:37.766 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-11-05T08:30:37.767 INFO:teuthology.orchestra.run.smithi053.stdout:Adding group ceph....done 2023-11-05T08:30:37.823 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-11-05T08:30:37.892 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-11-05T08:30:38.056 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-11-05T08:30:38.126 INFO:teuthology.orchestra.run.smithi053.stdout:Adding system user ceph....done 2023-11-05T08:30:38.160 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-11-05T08:30:38.208 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-11-05T08:30:38.286 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-11-05T08:30:38.295 INFO:teuthology.orchestra.run.smithi053.stdout:Setting system user ceph properties....done 2023-11-05T08:30:38.309 INFO:teuthology.orchestra.run.smithi053.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-11-05T08:30:38.348 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-11-05T08:30:38.462 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-11-05T08:30:38.476 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-portend (3.0.0-1) ... 2023-11-05T08:30:38.564 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-11-05T08:30:38.684 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-11-05T08:30:38.768 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-05T08:30:38.810 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up librados2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:38.828 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-11-05T08:30:38.840 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-11-05T08:30:38.894 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-11-05T08:30:38.936 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up librgw2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:39.062 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:39.083 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-11-05T08:30:39.188 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libcephfs2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:39.204 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-11-05T08:30:39.314 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libradosstriper1 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:39.366 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-test (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:39.488 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-11-05T08:30:39.492 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up radosgw (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:39.534 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-11-05T08:30:39.544 INFO:teuthology.orchestra.run.smithi090.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-11-05T08:30:39.756 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-11-05T08:30:39.778 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-11-05T08:30:39.810 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up librbd1 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:39.882 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-rsa (4.8-1) ... 2023-11-05T08:30:39.896 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-11-05T08:30:39.936 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-mgr-modules-core (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:39.981 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-11-05T08:30:39.982 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-11-05T08:30:40.062 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-fuse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:40.158 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-11-05T08:30:40.264 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-11-05T08:30:40.264 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-11-05T08:30:40.282 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-11-05T08:30:40.459 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-11-05T08:30:40.486 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-base (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:40.533 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-portend (3.0.0-1) ... 2023-11-05T08:30:40.719 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-11-05T08:30:40.784 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-05T08:30:40.788 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libcephfs-dev (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:40.817 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-11-05T08:30:40.898 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-rados (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:40.999 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-11-05T08:30:41.087 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-11-05T08:30:41.094 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-11-05T08:30:41.321 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-mds (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:41.363 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-11-05T08:30:41.470 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-11-05T08:30:41.561 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-11-05T08:30:41.561 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-11-05T08:30:41.707 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-11-05T08:30:41.771 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-11-05T08:30:41.833 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up librados2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:41.959 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up librgw2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:42.031 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-11-05T08:30:42.064 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-mgr (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:42.085 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:42.158 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-rsa (4.8-1) ... 2023-11-05T08:30:42.211 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libcephfs2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:42.262 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-rbd (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:42.266 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-11-05T08:30:42.267 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-11-05T08:30:42.328 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libradosstriper1 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:42.371 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up rbd-fuse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:42.426 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-11-05T08:30:42.472 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-rgw (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:42.533 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-11-05T08:30:42.573 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-cephfs (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:42.685 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-11-05T08:30:42.699 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-common (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:42.807 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-osd (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:42.818 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up librbd1 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:42.944 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-mgr-modules-core (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:42.945 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-11-05T08:30:43.070 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-fuse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:43.117 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-11-05T08:30:43.117 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-11-05T08:30:43.267 INFO:teuthology.orchestra.run.smithi002.stdout:Adding group ceph....done 2023-11-05T08:30:43.280 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-11-05T08:30:43.281 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-11-05T08:30:43.330 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-11-05T08:30:43.598 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-11-05T08:30:43.610 INFO:teuthology.orchestra.run.smithi002.stdout:Adding system user ceph....done 2023-11-05T08:30:43.676 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:43.737 INFO:teuthology.orchestra.run.smithi002.stdout:Setting system user ceph properties....done 2023-11-05T08:30:43.752 INFO:teuthology.orchestra.run.smithi002.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-11-05T08:30:43.771 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up libcephfs-dev (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:43.802 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:43.898 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-rados (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:43.906 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-11-05T08:30:43.952 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-11-05T08:30:43.953 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-mon (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:44.050 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-11-05T08:30:44.078 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up librados2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:44.175 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-11-05T08:30:44.176 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-11-05T08:30:44.204 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up librgw2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:44.261 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-11-05T08:30:44.330 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:44.456 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libcephfs2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:44.557 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libradosstriper1 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:44.712 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-mgr-cephadm (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:44.770 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-11-05T08:30:44.842 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-test (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:44.847 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:44.948 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-mgr-dashboard (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:44.968 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up radosgw (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:45.038 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up librbd1 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:45.074 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-volume (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:45.156 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-mgr-modules-core (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:45.250 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-rbd (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:45.282 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-fuse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:45.376 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up rbd-fuse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:45.464 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-11-05T08:30:45.464 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-11-05T08:30:45.494 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-11-05T08:30:45.495 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-11-05T08:30:45.502 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-rgw (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:45.543 INFO:teuthology.orchestra.run.smithi053.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-11-05T08:30:45.611 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up python3-cephfs (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:45.712 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-common (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:46.003 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-base (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:46.034 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libcephfs-dev (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:46.160 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-rados (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:46.196 INFO:teuthology.orchestra.run.smithi069.stdout:Adding group ceph....done 2023-11-05T08:30:46.296 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-11-05T08:30:46.364 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-11-05T08:30:46.532 INFO:teuthology.orchestra.run.smithi069.stdout:Adding system user ceph....done 2023-11-05T08:30:46.709 INFO:teuthology.orchestra.run.smithi069.stdout:Setting system user ceph properties....done 2023-11-05T08:30:46.724 INFO:teuthology.orchestra.run.smithi069.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-11-05T08:30:46.745 INFO:teuthology.orchestra.run.smithi053.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-11-05T08:30:46.869 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-11-05T08:30:46.871 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-mds (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:47.099 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-11-05T08:30:47.099 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-11-05T08:30:47.205 INFO:teuthology.orchestra.run.smithi053.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-11-05T08:30:47.208 DEBUG:teuthology.parallel:result is None 2023-11-05T08:30:47.209 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-11-05T08:30:47.568 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-rbd (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:47.614 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-mgr (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:47.693 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up rbd-fuse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:47.764 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-test (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:47.821 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-rgw (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:47.850 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-11-05T08:30:47.850 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-11-05T08:30:47.914 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up radosgw (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:47.947 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-cephfs (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:48.073 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-common (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:48.398 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-11-05T08:30:48.399 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-11-05T08:30:48.407 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-osd (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:48.574 INFO:teuthology.orchestra.run.smithi090.stdout:Adding group ceph....done 2023-11-05T08:30:48.736 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-11-05T08:30:48.736 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-11-05T08:30:48.883 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-base (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:48.944 INFO:teuthology.orchestra.run.smithi090.stdout:Adding system user ceph....done 2023-11-05T08:30:49.096 INFO:teuthology.orchestra.run.smithi090.stdout:Setting system user ceph properties....done 2023-11-05T08:30:49.115 INFO:teuthology.orchestra.run.smithi090.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-11-05T08:30:49.225 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-11-05T08:30:49.251 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-11-05T08:30:49.275 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:49.402 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:49.561 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-mon (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:49.621 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-11-05T08:30:49.734 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-mds (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:49.780 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-11-05T08:30:49.781 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-11-05T08:30:49.965 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-11-05T08:30:49.965 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-11-05T08:30:50.151 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-test (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:50.278 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up radosgw (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:50.345 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-mgr-cephadm (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:50.453 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-mgr (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:50.488 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:50.589 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-mgr-dashboard (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:50.668 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-11-05T08:30:50.669 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-11-05T08:30:50.795 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-11-05T08:30:50.795 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-11-05T08:30:50.910 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-volume (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:51.179 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-osd (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:51.271 INFO:teuthology.orchestra.run.smithi002.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-11-05T08:30:51.322 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-base (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:51.502 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-11-05T08:30:51.502 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-11-05T08:30:51.641 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-11-05T08:30:52.043 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:52.169 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:52.193 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-mds (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:52.295 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-mon (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:52.422 INFO:teuthology.orchestra.run.smithi002.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-11-05T08:30:52.490 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-11-05T08:30:52.490 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-11-05T08:30:52.553 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-11-05T08:30:52.553 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-11-05T08:30:52.870 INFO:teuthology.orchestra.run.smithi002.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-11-05T08:30:52.873 DEBUG:teuthology.parallel:result is None 2023-11-05T08:30:53.012 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-mgr-cephadm (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:53.096 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-mgr (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:53.172 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:53.298 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-mgr-dashboard (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:53.308 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-11-05T08:30:53.308 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-11-05T08:30:53.455 INFO:teuthology.orchestra.run.smithi069.stdout:Setting up ceph-volume (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:53.856 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-osd (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:53.910 INFO:teuthology.orchestra.run.smithi069.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-11-05T08:30:54.166 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-11-05T08:30:54.166 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-11-05T08:30:54.685 INFO:teuthology.orchestra.run.smithi069.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-11-05T08:30:54.726 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:54.852 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:55.003 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-mon (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:55.120 INFO:teuthology.orchestra.run.smithi069.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-11-05T08:30:55.123 DEBUG:teuthology.parallel:result is None 2023-11-05T08:30:55.216 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-11-05T08:30:55.216 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-11-05T08:30:55.721 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-mgr-cephadm (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:55.872 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:56.078 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-mgr-dashboard (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:56.316 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-volume (18.0.0-6776-g420cd129-1jammy) ... 2023-11-05T08:30:56.655 INFO:teuthology.orchestra.run.smithi090.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-11-05T08:30:57.407 INFO:teuthology.orchestra.run.smithi090.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-11-05T08:30:57.879 INFO:teuthology.orchestra.run.smithi090.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-11-05T08:30:57.882 DEBUG:teuthology.parallel:result is None 2023-11-05T08:30:57.883 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=420cd129397301b1521ceb74e1bdbd9b855d2189 2023-11-05T08:30:58.029 DEBUG:teuthology.orchestra.run.smithi002:> dpkg-query -W -f '${Version}' ceph 2023-11-05T08:30:58.049 INFO:teuthology.orchestra.run.smithi002.stdout:18.0.0-6776-g420cd129-1jammy 2023-11-05T08:30:58.049 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6776-g420cd129-1jammy 2023-11-05T08:30:58.050 INFO:teuthology.task.install:The correct ceph version 18.0.0-6776-g420cd129-1jammy is installed. 2023-11-05T08:30:58.050 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=420cd129397301b1521ceb74e1bdbd9b855d2189 2023-11-05T08:30:58.274 DEBUG:teuthology.orchestra.run.smithi053:> dpkg-query -W -f '${Version}' ceph 2023-11-05T08:30:58.295 INFO:teuthology.orchestra.run.smithi053.stdout:18.0.0-6776-g420cd129-1jammy 2023-11-05T08:30:58.295 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6776-g420cd129-1jammy 2023-11-05T08:30:58.295 INFO:teuthology.task.install:The correct ceph version 18.0.0-6776-g420cd129-1jammy is installed. 2023-11-05T08:30:58.296 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=420cd129397301b1521ceb74e1bdbd9b855d2189 2023-11-05T08:30:58.433 DEBUG:teuthology.orchestra.run.smithi069:> dpkg-query -W -f '${Version}' ceph 2023-11-05T08:30:58.452 INFO:teuthology.orchestra.run.smithi069.stdout:18.0.0-6776-g420cd129-1jammy 2023-11-05T08:30:58.453 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6776-g420cd129-1jammy 2023-11-05T08:30:58.453 INFO:teuthology.task.install:The correct ceph version 18.0.0-6776-g420cd129-1jammy is installed. 2023-11-05T08:30:58.453 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=420cd129397301b1521ceb74e1bdbd9b855d2189 2023-11-05T08:30:58.670 DEBUG:teuthology.orchestra.run.smithi090:> dpkg-query -W -f '${Version}' ceph 2023-11-05T08:30:58.689 INFO:teuthology.orchestra.run.smithi090.stdout:18.0.0-6776-g420cd129-1jammy 2023-11-05T08:30:58.689 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6776-g420cd129-1jammy 2023-11-05T08:30:58.690 INFO:teuthology.task.install:The correct ceph version 18.0.0-6776-g420cd129-1jammy is installed. 2023-11-05T08:30:58.690 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-05T08:30:58.691 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-05T08:30:58.691 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-05T08:30:58.706 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-11-05T08:30:58.706 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-05T08:30:58.721 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-05T08:30:58.722 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-05T08:30:58.736 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-11-05T08:30:58.737 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-05T08:30:58.751 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-05T08:30:58.751 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-05T08:30:58.751 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/usr/bin/daemon-helper 2023-11-05T08:30:58.765 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-05T08:30:58.818 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-11-05T08:30:58.818 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/daemon-helper 2023-11-05T08:30:58.832 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-05T08:30:58.886 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-05T08:30:58.886 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/daemon-helper 2023-11-05T08:30:58.900 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-05T08:30:58.956 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-11-05T08:30:58.956 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/daemon-helper 2023-11-05T08:30:58.969 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-05T08:30:59.025 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-05T08:30:59.025 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-05T08:30:59.025 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-05T08:30:59.040 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-05T08:30:59.094 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-11-05T08:30:59.094 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-05T08:30:59.108 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-05T08:30:59.161 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-05T08:30:59.162 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-05T08:30:59.175 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-05T08:30:59.229 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-11-05T08:30:59.229 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-05T08:30:59.243 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-05T08:30:59.296 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-05T08:30:59.296 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-05T08:30:59.296 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/usr/bin/stdin-killer 2023-11-05T08:30:59.310 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-05T08:30:59.365 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-11-05T08:30:59.366 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/stdin-killer 2023-11-05T08:30:59.380 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-05T08:30:59.434 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-05T08:30:59.434 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/stdin-killer 2023-11-05T08:30:59.450 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-05T08:30:59.504 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-11-05T08:30:59.504 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/stdin-killer 2023-11-05T08:30:59.518 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-05T08:30:59.573 INFO:teuthology.run_tasks:Running task ceph... 2023-11-05T08:30:59.639 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-11-05T08:30:59.640 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 777 /var/log/ceph 2023-11-05T08:30:59.643 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 777 /var/log/ceph 2023-11-05T08:30:59.646 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 777 /var/log/ceph 2023-11-05T08:30:59.648 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 777 /var/log/ceph 2023-11-05T08:30:59.661 INFO:tasks.ceph:Disabling ceph logrotate... 2023-11-05T08:30:59.662 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-05T08:30:59.699 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-05T08:30:59.702 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-05T08:30:59.704 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-05T08:30:59.716 INFO:tasks.ceph:Creating extra log directories... 2023-11-05T08:30:59.716 DEBUG:teuthology.orchestra.run.smithi002:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-05T08:30:59.755 DEBUG:teuthology.orchestra.run.smithi053:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-05T08:30:59.758 DEBUG:teuthology.orchestra.run.smithi069:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-05T08:30:59.760 DEBUG:teuthology.orchestra.run.smithi090:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-05T08:30:59.786 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-11-05T08:30:59.786 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 5000, 'osd_async_recovery_min_cost': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-11-05T08:30:59.786 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/benhanokh-2023-10-31_16:46:33-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442398', 'branch': 'DBG_maintain_prefix_itr_2', 'description': 'rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}', 'email': None, 'first_in_suite': False, 'job_id': '7442398', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'benhanokh-2023-10-31_16:46:33-rados-DBG_maintain_prefix_itr_2-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'DBG_maintain_prefix_itr_2'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 5000, 'osd_async_recovery_min_cost': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'sha1': '420cd129397301b1521ceb74e1bdbd9b855d2189'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '420cd129397301b1521ceb74e1bdbd9b855d2189'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'DBG_maintain_prefix_itr_2', 'sha1': '420cd129397301b1521ceb74e1bdbd9b855d2189'}}, 'owner': 'scheduled_benhanokh@teuthology', 'priority': 100, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'], ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'], ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'], ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0']], 'seed': 9009, 'sha1': '420cd129397301b1521ceb74e1bdbd9b855d2189', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'DBG_maintain_prefix_itr_2', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '420cd129397301b1521ceb74e1bdbd9b855d2189', 'targets': {'smithi002.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFYnh2HerGKUWiNK+JZ1+sOjYIB226bYu1t+bYcFbrDbwRbToSpMKjOKohouju5rpo4JGem8VCkcd8WCrEfDO90=', 'smithi053.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJhqMGwSC/AaSpescnzcOcLSXMVxyFf8TWoJBg6tZz9J7np1OvfHqj0Q6WgZ/zHx4+f+RgNMtOA5jvGv7ZeAjsU=', 'smithi069.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDUJGpZU1aZPH20MVSpQexFWypB4vrahbDUJdquA7RrNdQ0snsbTNu0jtgDOtsAX775dxnSa84npB5kPsKt0MRM=', 'smithi090.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNtyBiM3I8tIXFthmH5r8xkuDBDlJ50iU5mm6jCAXzGFO6LLWNfz+LKJeC7G8VVF6lJYYzPO5xDEdjKdlVey0UI='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 5000, 'osd_async_recovery_min_cost': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'sha1': '420cd129397301b1521ceb74e1bdbd9b855d2189', 'cluster': 'ceph'}}, {'thrashosds': {'bluestore_new_sharding': 'random', 'chance_bluestore_reshard': 1, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200}}, {'rados': {'clients': ['client.0'], 'ec_pool': True, 'erasure_code_profile': {'c': 2, 'crush-failure-domain': 'osd', 'k': 4, 'm': 3, 'name': 'shecprofile', 'plugin': 'shec'}, 'objects': 50, 'op_weights': {'append': 100, 'copy_from': 50, 'delete': 50, 'read': 100, 'rmattr': 25, 'rollback': 50, 'setattr': 25, 'snap_create': 50, 'snap_remove': 50, 'write': 0}, 'ops': 400, 'write_append_excl': False}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6899cd26fceddb2fec83dc1a1349394b28c8998e', 'timestamp': '2023-10-31_16:46:33', 'tube': 'smithi', 'user': 'benhanokh', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3084822'} 2023-11-05T08:30:59.787 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-05T08:30:59.812 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-05T08:30:59.814 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-05T08:30:59.817 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-05T08:30:59.823 DEBUG:teuthology.orchestra.run.smithi002:> sudo install -d -m0777 -- /var/run/ceph 2023-11-05T08:30:59.859 DEBUG:teuthology.orchestra.run.smithi053:> sudo install -d -m0777 -- /var/run/ceph 2023-11-05T08:30:59.867 DEBUG:teuthology.orchestra.run.smithi069:> sudo install -d -m0777 -- /var/run/ceph 2023-11-05T08:30:59.870 DEBUG:teuthology.orchestra.run.smithi090:> sudo install -d -m0777 -- /var/run/ceph 2023-11-05T08:30:59.882 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-05T08:30:59.883 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/scratch_devs of=/dev/stdout 2023-11-05T08:30:59.919 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-05T08:30:59.919 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_1 2023-11-05T08:30:59.967 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-05T08:30:59.967 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-05T08:30:59.967 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 889 Links: 1 2023-11-05T08:30:59.967 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-05T08:30:59.968 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2023-11-05 08:25:31.470406279 +0000 2023-11-05T08:30:59.968 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2023-11-05 08:25:31.138409244 +0000 2023-11-05T08:30:59.968 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2023-11-05 08:25:31.138409244 +0000 2023-11-05T08:30:59.968 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: - 2023-11-05T08:30:59.969 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-05T08:31:00.021 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2023-11-05T08:31:00.022 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2023-11-05T08:31:00.022 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000257291 s, 2.0 MB/s 2023-11-05T08:31:00.023 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-05T08:31:00.072 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_2 2023-11-05T08:31:00.118 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-05T08:31:00.119 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-05T08:31:00.119 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 901 Links: 1 2023-11-05T08:31:00.119 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-05T08:31:00.119 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2023-11-05 08:25:31.698404243 +0000 2023-11-05T08:31:00.119 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2023-11-05 08:25:31.694404279 +0000 2023-11-05T08:31:00.120 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2023-11-05 08:25:31.694404279 +0000 2023-11-05T08:31:00.120 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: - 2023-11-05T08:31:00.120 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-05T08:31:00.173 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2023-11-05T08:31:00.173 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2023-11-05T08:31:00.173 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000311937 s, 1.6 MB/s 2023-11-05T08:31:00.174 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-05T08:31:00.224 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_3 2023-11-05T08:31:00.271 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-05T08:31:00.271 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-05T08:31:00.271 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 922 Links: 1 2023-11-05T08:31:00.271 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-05T08:31:00.272 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2023-11-05 08:25:32.666395597 +0000 2023-11-05T08:31:00.272 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2023-11-05 08:25:32.290398955 +0000 2023-11-05T08:31:00.272 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2023-11-05 08:25:32.290398955 +0000 2023-11-05T08:31:00.272 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: - 2023-11-05T08:31:00.272 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-05T08:31:00.325 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2023-11-05T08:31:00.325 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2023-11-05T08:31:00.326 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000313264 s, 1.6 MB/s 2023-11-05T08:31:00.327 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-05T08:31:00.376 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_4 2023-11-05T08:31:00.423 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-05T08:31:00.423 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-05T08:31:00.423 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 932 Links: 1 2023-11-05T08:31:00.423 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-05T08:31:00.423 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2023-11-05 08:25:33.222390630 +0000 2023-11-05T08:31:00.424 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2023-11-05 08:25:32.882393667 +0000 2023-11-05T08:31:00.424 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2023-11-05 08:25:32.882393667 +0000 2023-11-05T08:31:00.424 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: - 2023-11-05T08:31:00.424 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-05T08:31:00.477 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2023-11-05T08:31:00.477 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2023-11-05T08:31:00.477 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000311665 s, 1.6 MB/s 2023-11-05T08:31:00.478 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-05T08:31:00.527 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2023-11-05T08:31:00.528 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-11-05T08:31:00.528 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/scratch_devs of=/dev/stdout 2023-11-05T08:31:00.535 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-05T08:31:00.535 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_1 2023-11-05T08:31:00.582 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-05T08:31:00.583 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-05T08:31:00.583 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 889 Links: 1 2023-11-05T08:31:00.583 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-05T08:31:00.583 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2023-11-05 08:25:22.142862635 +0000 2023-11-05T08:31:00.584 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2023-11-05 08:25:21.762866028 +0000 2023-11-05T08:31:00.584 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2023-11-05 08:25:21.762866028 +0000 2023-11-05T08:31:00.584 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: - 2023-11-05T08:31:00.584 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-05T08:31:00.637 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2023-11-05T08:31:00.638 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2023-11-05T08:31:00.638 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000306375 s, 1.7 MB/s 2023-11-05T08:31:00.639 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-05T08:31:00.688 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_2 2023-11-05T08:31:00.734 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-05T08:31:00.735 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-05T08:31:00.735 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 905 Links: 1 2023-11-05T08:31:00.735 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-05T08:31:00.735 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2023-11-05 08:25:22.354860740 +0000 2023-11-05T08:31:00.736 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2023-11-05 08:25:22.350860777 +0000 2023-11-05T08:31:00.736 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2023-11-05 08:25:22.350860777 +0000 2023-11-05T08:31:00.736 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: - 2023-11-05T08:31:00.737 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-05T08:31:00.789 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2023-11-05T08:31:00.789 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2023-11-05T08:31:00.789 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000289548 s, 1.8 MB/s 2023-11-05T08:31:00.790 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-05T08:31:00.840 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_3 2023-11-05T08:31:00.886 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-05T08:31:00.887 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-05T08:31:00.887 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 920 Links: 1 2023-11-05T08:31:00.887 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-05T08:31:00.887 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2023-11-05 08:25:23.302852274 +0000 2023-11-05T08:31:00.888 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2023-11-05 08:25:22.946855453 +0000 2023-11-05T08:31:00.888 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2023-11-05 08:25:22.946855453 +0000 2023-11-05T08:31:00.888 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: - 2023-11-05T08:31:00.888 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-05T08:31:00.941 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2023-11-05T08:31:00.942 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2023-11-05T08:31:00.942 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000306348 s, 1.7 MB/s 2023-11-05T08:31:00.944 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-05T08:31:00.992 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_4 2023-11-05T08:31:01.039 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-05T08:31:01.039 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-05T08:31:01.039 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 938 Links: 1 2023-11-05T08:31:01.040 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-05T08:31:01.040 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2023-11-05 08:25:23.530850238 +0000 2023-11-05T08:31:01.041 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2023-11-05 08:25:23.526850272 +0000 2023-11-05T08:31:01.041 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2023-11-05 08:25:23.526850272 +0000 2023-11-05T08:31:01.041 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: - 2023-11-05T08:31:01.042 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-05T08:31:01.093 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2023-11-05T08:31:01.093 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2023-11-05T08:31:01.093 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000302022 s, 1.7 MB/s 2023-11-05T08:31:01.095 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-05T08:31:01.144 INFO:tasks.ceph:osd dev map: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2023-11-05T08:31:01.144 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-05T08:31:01.145 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/scratch_devs of=/dev/stdout 2023-11-05T08:31:01.152 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-05T08:31:01.153 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_1 2023-11-05T08:31:01.197 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-05T08:31:01.198 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-05T08:31:01.198 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 887 Links: 1 2023-11-05T08:31:01.198 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-05T08:31:01.198 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2023-11-05 08:25:24.965545071 +0000 2023-11-05T08:31:01.199 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2023-11-05 08:25:24.605548286 +0000 2023-11-05T08:31:01.199 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2023-11-05 08:25:24.605548286 +0000 2023-11-05T08:31:01.199 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: - 2023-11-05T08:31:01.199 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-05T08:31:01.251 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2023-11-05T08:31:01.252 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2023-11-05T08:31:01.252 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000445704 s, 1.1 MB/s 2023-11-05T08:31:01.253 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-05T08:31:01.307 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_2 2023-11-05T08:31:01.353 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-05T08:31:01.353 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-05T08:31:01.354 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 900 Links: 1 2023-11-05T08:31:01.354 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-05T08:31:01.354 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2023-11-05 08:25:25.557539782 +0000 2023-11-05T08:31:01.354 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2023-11-05 08:25:25.209542891 +0000 2023-11-05T08:31:01.355 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2023-11-05 08:25:25.209542891 +0000 2023-11-05T08:31:01.355 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: - 2023-11-05T08:31:01.355 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-05T08:31:01.407 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2023-11-05T08:31:01.408 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2023-11-05T08:31:01.408 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000422177 s, 1.2 MB/s 2023-11-05T08:31:01.409 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-05T08:31:01.458 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_3 2023-11-05T08:31:01.505 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-05T08:31:01.505 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-05T08:31:01.505 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 922 Links: 1 2023-11-05T08:31:01.506 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-05T08:31:01.506 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2023-11-05 08:25:26.149534494 +0000 2023-11-05T08:31:01.506 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2023-11-05 08:25:25.813537496 +0000 2023-11-05T08:31:01.506 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2023-11-05 08:25:25.813537496 +0000 2023-11-05T08:31:01.506 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: - 2023-11-05T08:31:01.507 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-05T08:31:01.559 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2023-11-05T08:31:01.560 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2023-11-05T08:31:01.560 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.00041777 s, 1.2 MB/s 2023-11-05T08:31:01.561 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-05T08:31:01.610 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_4 2023-11-05T08:31:01.657 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-05T08:31:01.657 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-05T08:31:01.657 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 938 Links: 1 2023-11-05T08:31:01.657 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-05T08:31:01.658 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2023-11-05 08:25:26.741529207 +0000 2023-11-05T08:31:01.658 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2023-11-05 08:25:26.413532137 +0000 2023-11-05T08:31:01.658 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2023-11-05 08:25:26.413532137 +0000 2023-11-05T08:31:01.658 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: - 2023-11-05T08:31:01.658 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-05T08:31:01.710 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2023-11-05T08:31:01.711 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2023-11-05T08:31:01.711 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000306402 s, 1.7 MB/s 2023-11-05T08:31:01.712 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-05T08:31:01.758 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2023-11-05T08:31:01.759 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-11-05T08:31:01.759 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/scratch_devs of=/dev/stdout 2023-11-05T08:31:01.765 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-05T08:31:01.766 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_1 2023-11-05T08:31:01.814 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-05T08:31:01.814 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-05T08:31:01.814 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 889 Links: 1 2023-11-05T08:31:01.815 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-05T08:31:01.815 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2023-11-05 08:25:23.635872795 +0000 2023-11-05T08:31:01.815 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2023-11-05 08:25:23.267876065 +0000 2023-11-05T08:31:01.815 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2023-11-05 08:25:23.267876065 +0000 2023-11-05T08:31:01.815 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: - 2023-11-05T08:31:01.816 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-05T08:31:01.868 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2023-11-05T08:31:01.868 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2023-11-05T08:31:01.869 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000372345 s, 1.4 MB/s 2023-11-05T08:31:01.869 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-05T08:31:01.919 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_2 2023-11-05T08:31:01.965 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-05T08:31:01.966 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-05T08:31:01.966 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 905 Links: 1 2023-11-05T08:31:01.966 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-05T08:31:01.966 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2023-11-05 08:25:24.235867462 +0000 2023-11-05T08:31:01.967 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2023-11-05 08:25:23.863870769 +0000 2023-11-05T08:31:01.967 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2023-11-05 08:25:23.863870769 +0000 2023-11-05T08:31:01.967 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: - 2023-11-05T08:31:01.967 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-05T08:31:02.020 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2023-11-05T08:31:02.020 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2023-11-05T08:31:02.020 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000345274 s, 1.5 MB/s 2023-11-05T08:31:02.021 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-05T08:31:02.071 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_3 2023-11-05T08:31:02.118 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-05T08:31:02.118 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-05T08:31:02.118 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 923 Links: 1 2023-11-05T08:31:02.118 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-05T08:31:02.118 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2023-11-05 08:25:24.795862483 +0000 2023-11-05T08:31:02.119 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2023-11-05 08:25:24.455865505 +0000 2023-11-05T08:31:02.119 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2023-11-05 08:25:24.455865505 +0000 2023-11-05T08:31:02.119 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: - 2023-11-05T08:31:02.119 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-05T08:31:02.172 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2023-11-05T08:31:02.172 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2023-11-05T08:31:02.172 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000351553 s, 1.5 MB/s 2023-11-05T08:31:02.173 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-05T08:31:02.223 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_4 2023-11-05T08:31:02.269 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-05T08:31:02.269 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-05T08:31:02.270 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 932 Links: 1 2023-11-05T08:31:02.270 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-05T08:31:02.270 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2023-11-05 08:25:25.375857328 +0000 2023-11-05T08:31:02.270 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2023-11-05 08:25:25.015860528 +0000 2023-11-05T08:31:02.270 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2023-11-05 08:25:25.015860528 +0000 2023-11-05T08:31:02.271 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: - 2023-11-05T08:31:02.271 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-05T08:31:02.325 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2023-11-05T08:31:02.325 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2023-11-05T08:31:02.326 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000370448 s, 1.4 MB/s 2023-11-05T08:31:02.326 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-05T08:31:02.375 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2023-11-05T08:31:02.375 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi002.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi053.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi069.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi090.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2023-11-05T08:31:02.375 INFO:tasks.ceph:Generating config... 2023-11-05T08:31:02.377 INFO:tasks.ceph:[osd] debug monc = 20 2023-11-05T08:31:02.377 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-11-05T08:31:02.377 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2023-11-05T08:31:02.377 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-11-05T08:31:02.378 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-11-05T08:31:02.378 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2023-11-05T08:31:02.378 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-11-05T08:31:02.378 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-11-05T08:31:02.379 INFO:tasks.ceph:[osd] debug ms = 1 2023-11-05T08:31:02.379 INFO:tasks.ceph:[osd] debug osd = 20 2023-11-05T08:31:02.379 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-11-05T08:31:02.379 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-11-05T08:31:02.380 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-11-05T08:31:02.380 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-11-05T08:31:02.380 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2023-11-05T08:31:02.380 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.1 2023-11-05T08:31:02.381 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-11-05T08:31:02.381 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-11-05T08:31:02.381 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-11-05T08:31:02.381 INFO:tasks.ceph:[osd] osd heartbeat use min delay socket = True 2023-11-05T08:31:02.382 INFO:tasks.ceph:[osd] osd max backfills = 3 2023-11-05T08:31:02.382 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2023-11-05T08:31:02.382 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2023-11-05T08:31:02.382 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2023-11-05T08:31:02.383 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-11-05T08:31:02.383 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-11-05T08:31:02.383 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-11-05T08:31:02.383 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2023-11-05T08:31:02.384 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2023-11-05T08:31:02.384 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-11-05T08:31:02.384 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-11-05T08:31:02.384 INFO:tasks.ceph:[global] mon election default strategy = 3 2023-11-05T08:31:02.385 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-11-05T08:31:02.385 INFO:tasks.ceph:[global] osd_async_recovery_min_cost = 1 2023-11-05T08:31:02.385 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-11-05T08:31:02.385 INFO:tasks.ceph:[mgr] debug ms = 1 2023-11-05T08:31:02.386 INFO:tasks.ceph:[mon] debug mon = 20 2023-11-05T08:31:02.386 INFO:tasks.ceph:[mon] debug ms = 1 2023-11-05T08:31:02.386 INFO:tasks.ceph:[mon] debug paxos = 20 2023-11-05T08:31:02.386 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-11-05T08:31:02.387 INFO:tasks.ceph:Setting up mon.a... 2023-11-05T08:31:02.387 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-11-05T08:31:02.423 INFO:teuthology.orchestra.run.smithi002.stdout:creating /etc/ceph/ceph.keyring 2023-11-05T08:31:02.435 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-11-05T08:31:02.518 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-05T08:31:02.575 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.2'), ('mon.b', '172.21.15.53'), ('mon.c', '172.21.15.69')] 2023-11-05T08:31:02.576 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.2,172.21.15.53,172.21.15.69', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 5000, 'osd_async_recovery_min_cost': 1}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2023-11-05T08:31:02.577 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-05T08:31:02.577 DEBUG:teuthology.orchestra.run.smithi002:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-11-05T08:31:02.622 DEBUG:teuthology.orchestra.run.smithi002:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.2 --add b 172.21.15.53 --add c 172.21.15.69 --print /home/ubuntu/cephtest/ceph.monmap 2023-11-05T08:31:02.709 INFO:teuthology.orchestra.run.smithi002.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-11-05T08:31:02.710 INFO:teuthology.orchestra.run.smithi002.stdout:monmaptool: generated fsid 8f1fa9b5-06de-4da7-985f-30d4ad103e2d 2023-11-05T08:31:02.710 INFO:teuthology.orchestra.run.smithi002.stdout:setting min_mon_release = pacific 2023-11-05T08:31:02.710 INFO:teuthology.orchestra.run.smithi002.stdout:epoch 0 2023-11-05T08:31:02.710 INFO:teuthology.orchestra.run.smithi002.stdout:fsid 8f1fa9b5-06de-4da7-985f-30d4ad103e2d 2023-11-05T08:31:02.710 INFO:teuthology.orchestra.run.smithi002.stdout:last_changed 2023-11-05T08:31:02.712167+0000 2023-11-05T08:31:02.711 INFO:teuthology.orchestra.run.smithi002.stdout:created 2023-11-05T08:31:02.712167+0000 2023-11-05T08:31:02.711 INFO:teuthology.orchestra.run.smithi002.stdout:min_mon_release 16 (pacific) 2023-11-05T08:31:02.711 INFO:teuthology.orchestra.run.smithi002.stdout:election_strategy: 3 2023-11-05T08:31:02.711 INFO:teuthology.orchestra.run.smithi002.stdout:0: [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] mon.a 2023-11-05T08:31:02.711 INFO:teuthology.orchestra.run.smithi002.stdout:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2023-11-05T08:31:02.712 INFO:teuthology.orchestra.run.smithi002.stdout:2: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.c 2023-11-05T08:31:02.712 INFO:teuthology.orchestra.run.smithi002.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-11-05T08:31:02.713 DEBUG:teuthology.orchestra.run.smithi002:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-11-05T08:31:02.758 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 8f1fa9b5-06de-4da7-985f-30d4ad103e2d... 2023-11-05T08:31:02.759 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-05T08:31:02.803 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-05T08:31:02.806 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-05T08:31:02.808 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-05T08:31:02.833 INFO:teuthology.orchestra.run.smithi002.stdout:[global] 2023-11-05T08:31:02.834 INFO:teuthology.orchestra.run.smithi002.stdout: chdir = "" 2023-11-05T08:31:02.834 INFO:teuthology.orchestra.run.smithi002.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-05T08:31:02.834 INFO:teuthology.orchestra.run.smithi002.stdout: auth supported = cephx 2023-11-05T08:31:02.834 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:31:02.834 INFO:teuthology.orchestra.run.smithi002.stdout: filestore xattr use omap = true 2023-11-05T08:31:02.835 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:31:02.835 INFO:teuthology.orchestra.run.smithi002.stdout: mon clock drift allowed = 1.000 2023-11-05T08:31:02.835 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:31:02.836 INFO:teuthology.orchestra.run.smithi002.stdout: osd crush chooseleaf type = 0 2023-11-05T08:31:02.836 INFO:teuthology.orchestra.run.smithi002.stdout: auth debug = true 2023-11-05T08:31:02.836 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:31:02.836 INFO:teuthology.orchestra.run.smithi002.stdout: ms die on old message = true 2023-11-05T08:31:02.837 INFO:teuthology.orchestra.run.smithi002.stdout: ms die on bug = true 2023-11-05T08:31:02.837 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:31:02.837 INFO:teuthology.orchestra.run.smithi002.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-05T08:31:02.837 INFO:teuthology.orchestra.run.smithi002.stdout: mon pg warn max object skew = 0 2023-11-05T08:31:02.837 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:31:02.838 INFO:teuthology.orchestra.run.smithi002.stdout: # disable pg_autoscaler by default for new pools 2023-11-05T08:31:02.838 INFO:teuthology.orchestra.run.smithi002.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-05T08:31:02.838 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:31:02.838 INFO:teuthology.orchestra.run.smithi002.stdout: osd pool default size = 2 2023-11-05T08:31:02.838 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:31:02.839 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd allow primary affinity = true 2023-11-05T08:31:02.839 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd allow pg remap = true 2023-11-05T08:31:02.839 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on legacy crush tunables = false 2023-11-05T08:31:02.839 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on crush straw calc version zero = false 2023-11-05T08:31:02.839 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on no sortbitwise = false 2023-11-05T08:31:02.840 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on osd down out interval zero = false 2023-11-05T08:31:02.840 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on too few osds = false 2023-11-05T08:31:02.840 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-05T08:31:02.840 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-05T08:31:02.841 INFO:teuthology.orchestra.run.smithi002.stdout: mon_allow_pool_size_one = true 2023-11-05T08:31:02.841 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:31:02.841 INFO:teuthology.orchestra.run.smithi002.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-05T08:31:02.841 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:31:02.841 INFO:teuthology.orchestra.run.smithi002.stdout: osd default data pool replay window = 5 2023-11-05T08:31:02.842 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:31:02.842 INFO:teuthology.orchestra.run.smithi002.stdout: mon allow pool delete = true 2023-11-05T08:31:02.842 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:31:02.842 INFO:teuthology.orchestra.run.smithi002.stdout: mon cluster log file level = debug 2023-11-05T08:31:02.842 INFO:teuthology.orchestra.run.smithi002.stdout: debug asserts on shutdown = true 2023-11-05T08:31:02.843 INFO:teuthology.orchestra.run.smithi002.stdout: mon health detail to clog = false 2023-11-05T08:31:02.843 INFO:teuthology.orchestra.run.smithi002.stdout: mon host = "172.21.15.2,172.21.15.53,172.21.15.69" 2023-11-05T08:31:02.843 INFO:teuthology.orchestra.run.smithi002.stdout: mon client directed command retry = 5 2023-11-05T08:31:02.843 INFO:teuthology.orchestra.run.smithi002.stdout: mon election default strategy = 3 2023-11-05T08:31:02.843 INFO:teuthology.orchestra.run.smithi002.stdout: ms inject socket failures = 5000 2023-11-05T08:31:02.844 INFO:teuthology.orchestra.run.smithi002.stdout: osd_async_recovery_min_cost = 1 2023-11-05T08:31:02.844 INFO:teuthology.orchestra.run.smithi002.stdout: fsid = 8f1fa9b5-06de-4da7-985f-30d4ad103e2d 2023-11-05T08:31:02.844 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:31:02.844 INFO:teuthology.orchestra.run.smithi002.stdout:[osd] 2023-11-05T08:31:02.844 INFO:teuthology.orchestra.run.smithi002.stdout: osd journal size = 100 2023-11-05T08:31:02.845 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:31:02.845 INFO:teuthology.orchestra.run.smithi002.stdout: osd scrub load threshold = 5.0 2023-11-05T08:31:02.845 INFO:teuthology.orchestra.run.smithi002.stdout: osd scrub max interval = 120 2023-11-05T08:31:02.845 INFO:teuthology.orchestra.run.smithi002.stdout: osd mclock profile = high_recovery_ops 2023-11-05T08:31:02.845 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:31:02.846 INFO:teuthology.orchestra.run.smithi002.stdout: osd recover clone overlap = true 2023-11-05T08:31:02.846 INFO:teuthology.orchestra.run.smithi002.stdout: osd recovery max chunk = 1048576 2023-11-05T08:31:02.846 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:31:02.846 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug shutdown = true 2023-11-05T08:31:02.846 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug op order = true 2023-11-05T08:31:02.846 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug verify stray on activate = true 2023-11-05T08:31:02.847 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug trim objects = true 2023-11-05T08:31:02.847 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:31:02.847 INFO:teuthology.orchestra.run.smithi002.stdout: osd open classes on start = true 2023-11-05T08:31:02.847 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug pg log writeout = true 2023-11-05T08:31:02.847 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:31:02.848 INFO:teuthology.orchestra.run.smithi002.stdout: osd deep scrub update digest min age = 30 2023-11-05T08:31:02.848 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:31:02.848 INFO:teuthology.orchestra.run.smithi002.stdout: osd map max advance = 10 2023-11-05T08:31:02.848 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:31:02.848 INFO:teuthology.orchestra.run.smithi002.stdout: journal zero on create = true 2023-11-05T08:31:02.849 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:31:02.849 INFO:teuthology.orchestra.run.smithi002.stdout: filestore ondisk finisher threads = 3 2023-11-05T08:31:02.849 INFO:teuthology.orchestra.run.smithi002.stdout: filestore apply finisher threads = 3 2023-11-05T08:31:02.849 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:31:02.849 INFO:teuthology.orchestra.run.smithi002.stdout: bdev debug aio = true 2023-11-05T08:31:02.849 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug misdirected ops = true 2023-11-05T08:31:02.850 INFO:teuthology.orchestra.run.smithi002.stdout: debug monc = 20 2023-11-05T08:31:02.850 INFO:teuthology.orchestra.run.smithi002.stdout: bluestore block size = 96636764160 2023-11-05T08:31:02.850 INFO:teuthology.orchestra.run.smithi002.stdout: bluestore compression algorithm = zstd 2023-11-05T08:31:02.850 INFO:teuthology.orchestra.run.smithi002.stdout: bluestore compression mode = aggressive 2023-11-05T08:31:02.851 INFO:teuthology.orchestra.run.smithi002.stdout: bluestore fsck on mount = True 2023-11-05T08:31:02.851 INFO:teuthology.orchestra.run.smithi002.stdout: bluestore zero block detection = True 2023-11-05T08:31:02.851 INFO:teuthology.orchestra.run.smithi002.stdout: debug bluefs = 20 2023-11-05T08:31:02.851 INFO:teuthology.orchestra.run.smithi002.stdout: debug bluestore = 20 2023-11-05T08:31:02.851 INFO:teuthology.orchestra.run.smithi002.stdout: debug ms = 1 2023-11-05T08:31:02.852 INFO:teuthology.orchestra.run.smithi002.stdout: debug osd = 20 2023-11-05T08:31:02.852 INFO:teuthology.orchestra.run.smithi002.stdout: debug rocksdb = 10 2023-11-05T08:31:02.852 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd backfillfull_ratio = 0.85 2023-11-05T08:31:02.852 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd full ratio = 0.9 2023-11-05T08:31:02.852 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd nearfull ratio = 0.8 2023-11-05T08:31:02.852 INFO:teuthology.orchestra.run.smithi002.stdout: osd blocked scrub grace period = 3600 2023-11-05T08:31:02.853 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug reject backfill probability = 0.1 2023-11-05T08:31:02.853 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug verify cached snaps = True 2023-11-05T08:31:02.853 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug verify missing on start = True 2023-11-05T08:31:02.853 INFO:teuthology.orchestra.run.smithi002.stdout: osd failsafe full ratio = 0.95 2023-11-05T08:31:02.853 INFO:teuthology.orchestra.run.smithi002.stdout: osd heartbeat use min delay socket = True 2023-11-05T08:31:02.854 INFO:teuthology.orchestra.run.smithi002.stdout: osd max backfills = 3 2023-11-05T08:31:02.854 INFO:teuthology.orchestra.run.smithi002.stdout: osd max markdown count = 1000 2023-11-05T08:31:02.854 INFO:teuthology.orchestra.run.smithi002.stdout: osd mclock override recovery settings = True 2023-11-05T08:31:02.854 INFO:teuthology.orchestra.run.smithi002.stdout: osd objectstore = bluestore 2023-11-05T08:31:02.854 INFO:teuthology.orchestra.run.smithi002.stdout: osd op queue = debug_random 2023-11-05T08:31:02.855 INFO:teuthology.orchestra.run.smithi002.stdout: osd op queue cut off = debug_random 2023-11-05T08:31:02.855 INFO:teuthology.orchestra.run.smithi002.stdout: osd scrub min interval = 60 2023-11-05T08:31:02.855 INFO:teuthology.orchestra.run.smithi002.stdout: osd shutdown pgref assert = True 2023-11-05T08:31:02.855 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:31:02.855 INFO:teuthology.orchestra.run.smithi002.stdout:[mgr] 2023-11-05T08:31:02.856 INFO:teuthology.orchestra.run.smithi002.stdout: debug ms = 1 2023-11-05T08:31:02.856 INFO:teuthology.orchestra.run.smithi002.stdout: debug mgr = 20 2023-11-05T08:31:02.856 INFO:teuthology.orchestra.run.smithi002.stdout: debug mon = 20 2023-11-05T08:31:02.856 INFO:teuthology.orchestra.run.smithi002.stdout: debug auth = 20 2023-11-05T08:31:02.856 INFO:teuthology.orchestra.run.smithi002.stdout: mon reweight min pgs per osd = 4 2023-11-05T08:31:02.857 INFO:teuthology.orchestra.run.smithi002.stdout: mon reweight min bytes per osd = 10 2023-11-05T08:31:02.857 INFO:teuthology.orchestra.run.smithi002.stdout: mgr/telemetry/nag = false 2023-11-05T08:31:02.857 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:31:02.857 INFO:teuthology.orchestra.run.smithi002.stdout:[mon] 2023-11-05T08:31:02.857 INFO:teuthology.orchestra.run.smithi002.stdout: debug ms = 1 2023-11-05T08:31:02.858 INFO:teuthology.orchestra.run.smithi002.stdout: debug mon = 20 2023-11-05T08:31:02.858 INFO:teuthology.orchestra.run.smithi002.stdout: debug paxos = 20 2023-11-05T08:31:02.858 INFO:teuthology.orchestra.run.smithi002.stdout: debug auth = 20 2023-11-05T08:31:02.858 INFO:teuthology.orchestra.run.smithi002.stdout: mon data avail warn = 5 2023-11-05T08:31:02.858 INFO:teuthology.orchestra.run.smithi002.stdout: mon mgr mkfs grace = 240 2023-11-05T08:31:02.859 INFO:teuthology.orchestra.run.smithi002.stdout: mon reweight min pgs per osd = 4 2023-11-05T08:31:02.859 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd reporter subtree level = osd 2023-11-05T08:31:02.859 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd prime pg temp = true 2023-11-05T08:31:02.859 INFO:teuthology.orchestra.run.smithi002.stdout: mon reweight min bytes per osd = 10 2023-11-05T08:31:02.859 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:31:02.859 INFO:teuthology.orchestra.run.smithi002.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-05T08:31:02.860 INFO:teuthology.orchestra.run.smithi002.stdout: auth mon ticket ttl = 660 # 11m 2023-11-05T08:31:02.860 INFO:teuthology.orchestra.run.smithi002.stdout: auth service ticket ttl = 240 # 4m 2023-11-05T08:31:02.860 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:31:02.860 INFO:teuthology.orchestra.run.smithi002.stdout: # don't complain about insecure global_id in the test suite 2023-11-05T08:31:02.860 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-05T08:31:02.861 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-05T08:31:02.861 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:31:02.861 INFO:teuthology.orchestra.run.smithi002.stdout: # 1m isn't quite enough 2023-11-05T08:31:02.861 INFO:teuthology.orchestra.run.smithi002.stdout: mon_down_mkfs_grace = 2m 2023-11-05T08:31:02.861 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:31:02.862 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_filestore_osds = false 2023-11-05T08:31:02.862 INFO:teuthology.orchestra.run.smithi002.stdout: mon scrub interval = 300 2023-11-05T08:31:02.862 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:31:02.862 INFO:teuthology.orchestra.run.smithi002.stdout:[client] 2023-11-05T08:31:02.862 INFO:teuthology.orchestra.run.smithi002.stdout: rgw cache enabled = true 2023-11-05T08:31:02.863 INFO:teuthology.orchestra.run.smithi002.stdout: rgw enable ops log = true 2023-11-05T08:31:02.863 INFO:teuthology.orchestra.run.smithi002.stdout: rgw enable usage log = true 2023-11-05T08:31:02.863 INFO:teuthology.orchestra.run.smithi002.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-05T08:31:02.863 INFO:teuthology.orchestra.run.smithi002.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-05T08:31:02.863 INFO:teuthology.orchestra.run.smithi002.stdout:[mon.a] 2023-11-05T08:31:02.863 INFO:teuthology.orchestra.run.smithi002.stdout:[mon.b] 2023-11-05T08:31:02.864 INFO:teuthology.orchestra.run.smithi002.stdout:[mon.c] 2023-11-05T08:31:02.866 INFO:teuthology.orchestra.run.smithi090.stdout:[global] 2023-11-05T08:31:02.866 INFO:teuthology.orchestra.run.smithi090.stdout: chdir = "" 2023-11-05T08:31:02.867 INFO:teuthology.orchestra.run.smithi090.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-05T08:31:02.867 INFO:teuthology.orchestra.run.smithi090.stdout: auth supported = cephx 2023-11-05T08:31:02.867 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-05T08:31:02.867 INFO:teuthology.orchestra.run.smithi090.stdout: filestore xattr use omap = true 2023-11-05T08:31:02.867 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-05T08:31:02.867 INFO:teuthology.orchestra.run.smithi090.stdout: mon clock drift allowed = 1.000 2023-11-05T08:31:02.868 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-05T08:31:02.868 INFO:teuthology.orchestra.run.smithi090.stdout: osd crush chooseleaf type = 0 2023-11-05T08:31:02.868 INFO:teuthology.orchestra.run.smithi090.stdout: auth debug = true 2023-11-05T08:31:02.868 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-05T08:31:02.868 INFO:teuthology.orchestra.run.smithi090.stdout: ms die on old message = true 2023-11-05T08:31:02.869 INFO:teuthology.orchestra.run.smithi090.stdout: ms die on bug = true 2023-11-05T08:31:02.869 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-05T08:31:02.869 INFO:teuthology.orchestra.run.smithi090.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-05T08:31:02.869 INFO:teuthology.orchestra.run.smithi090.stdout: mon pg warn max object skew = 0 2023-11-05T08:31:02.869 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-05T08:31:02.869 INFO:teuthology.orchestra.run.smithi090.stdout: # disable pg_autoscaler by default for new pools 2023-11-05T08:31:02.870 INFO:teuthology.orchestra.run.smithi090.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-05T08:31:02.870 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-05T08:31:02.870 INFO:teuthology.orchestra.run.smithi090.stdout: osd pool default size = 2 2023-11-05T08:31:02.870 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-05T08:31:02.870 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd allow primary affinity = true 2023-11-05T08:31:02.871 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd allow pg remap = true 2023-11-05T08:31:02.871 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on legacy crush tunables = false 2023-11-05T08:31:02.871 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on crush straw calc version zero = false 2023-11-05T08:31:02.871 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on no sortbitwise = false 2023-11-05T08:31:02.871 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on osd down out interval zero = false 2023-11-05T08:31:02.872 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on too few osds = false 2023-11-05T08:31:02.872 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-05T08:31:02.872 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-05T08:31:02.872 INFO:teuthology.orchestra.run.smithi090.stdout: mon_allow_pool_size_one = true 2023-11-05T08:31:02.872 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-05T08:31:02.872 INFO:teuthology.orchestra.run.smithi090.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-05T08:31:02.873 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-05T08:31:02.873 INFO:teuthology.orchestra.run.smithi090.stdout: osd default data pool replay window = 5 2023-11-05T08:31:02.873 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-05T08:31:02.873 INFO:teuthology.orchestra.run.smithi090.stdout: mon allow pool delete = true 2023-11-05T08:31:02.873 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-05T08:31:02.874 INFO:teuthology.orchestra.run.smithi090.stdout: mon cluster log file level = debug 2023-11-05T08:31:02.874 INFO:teuthology.orchestra.run.smithi090.stdout: debug asserts on shutdown = true 2023-11-05T08:31:02.874 INFO:teuthology.orchestra.run.smithi090.stdout: mon health detail to clog = false 2023-11-05T08:31:02.874 INFO:teuthology.orchestra.run.smithi090.stdout: mon host = "172.21.15.2,172.21.15.53,172.21.15.69" 2023-11-05T08:31:02.874 INFO:teuthology.orchestra.run.smithi090.stdout: mon client directed command retry = 5 2023-11-05T08:31:02.875 INFO:teuthology.orchestra.run.smithi090.stdout: mon election default strategy = 3 2023-11-05T08:31:02.875 INFO:teuthology.orchestra.run.smithi090.stdout: ms inject socket failures = 5000 2023-11-05T08:31:02.875 INFO:teuthology.orchestra.run.smithi090.stdout: osd_async_recovery_min_cost = 1 2023-11-05T08:31:02.875 INFO:teuthology.orchestra.run.smithi090.stdout: fsid = 8f1fa9b5-06de-4da7-985f-30d4ad103e2d 2023-11-05T08:31:02.875 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-05T08:31:02.875 INFO:teuthology.orchestra.run.smithi090.stdout:[osd] 2023-11-05T08:31:02.876 INFO:teuthology.orchestra.run.smithi090.stdout: osd journal size = 100 2023-11-05T08:31:02.876 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-05T08:31:02.876 INFO:teuthology.orchestra.run.smithi090.stdout: osd scrub load threshold = 5.0 2023-11-05T08:31:02.876 INFO:teuthology.orchestra.run.smithi090.stdout: osd scrub max interval = 120 2023-11-05T08:31:02.876 INFO:teuthology.orchestra.run.smithi090.stdout: osd mclock profile = high_recovery_ops 2023-11-05T08:31:02.877 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-05T08:31:02.877 INFO:teuthology.orchestra.run.smithi090.stdout: osd recover clone overlap = true 2023-11-05T08:31:02.877 INFO:teuthology.orchestra.run.smithi090.stdout: osd recovery max chunk = 1048576 2023-11-05T08:31:02.877 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-05T08:31:02.877 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug shutdown = true 2023-11-05T08:31:02.878 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug op order = true 2023-11-05T08:31:02.878 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug verify stray on activate = true 2023-11-05T08:31:02.878 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug trim objects = true 2023-11-05T08:31:02.878 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-05T08:31:02.878 INFO:teuthology.orchestra.run.smithi090.stdout: osd open classes on start = true 2023-11-05T08:31:02.879 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug pg log writeout = true 2023-11-05T08:31:02.879 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-05T08:31:02.879 INFO:teuthology.orchestra.run.smithi090.stdout: osd deep scrub update digest min age = 30 2023-11-05T08:31:02.879 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-05T08:31:02.879 INFO:teuthology.orchestra.run.smithi090.stdout: osd map max advance = 10 2023-11-05T08:31:02.879 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-05T08:31:02.880 INFO:teuthology.orchestra.run.smithi090.stdout: journal zero on create = true 2023-11-05T08:31:02.880 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-05T08:31:02.880 INFO:teuthology.orchestra.run.smithi090.stdout: filestore ondisk finisher threads = 3 2023-11-05T08:31:02.880 INFO:teuthology.orchestra.run.smithi090.stdout: filestore apply finisher threads = 3 2023-11-05T08:31:02.880 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-05T08:31:02.881 INFO:teuthology.orchestra.run.smithi090.stdout: bdev debug aio = true 2023-11-05T08:31:02.881 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug misdirected ops = true 2023-11-05T08:31:02.881 INFO:teuthology.orchestra.run.smithi090.stdout: debug monc = 20 2023-11-05T08:31:02.881 INFO:teuthology.orchestra.run.smithi090.stdout: bluestore block size = 96636764160 2023-11-05T08:31:02.881 INFO:teuthology.orchestra.run.smithi090.stdout: bluestore compression algorithm = zstd 2023-11-05T08:31:02.882 INFO:teuthology.orchestra.run.smithi090.stdout: bluestore compression mode = aggressive 2023-11-05T08:31:02.882 INFO:teuthology.orchestra.run.smithi090.stdout: bluestore fsck on mount = True 2023-11-05T08:31:02.882 INFO:teuthology.orchestra.run.smithi090.stdout: bluestore zero block detection = True 2023-11-05T08:31:02.882 INFO:teuthology.orchestra.run.smithi090.stdout: debug bluefs = 20 2023-11-05T08:31:02.882 INFO:teuthology.orchestra.run.smithi090.stdout: debug bluestore = 20 2023-11-05T08:31:02.883 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2023-11-05T08:31:02.883 INFO:teuthology.orchestra.run.smithi090.stdout: debug osd = 20 2023-11-05T08:31:02.883 INFO:teuthology.orchestra.run.smithi090.stdout: debug rocksdb = 10 2023-11-05T08:31:02.883 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd backfillfull_ratio = 0.85 2023-11-05T08:31:02.883 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd full ratio = 0.9 2023-11-05T08:31:02.883 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd nearfull ratio = 0.8 2023-11-05T08:31:02.884 INFO:teuthology.orchestra.run.smithi090.stdout: osd blocked scrub grace period = 3600 2023-11-05T08:31:02.884 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug reject backfill probability = 0.1 2023-11-05T08:31:02.884 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug verify cached snaps = True 2023-11-05T08:31:02.884 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug verify missing on start = True 2023-11-05T08:31:02.884 INFO:teuthology.orchestra.run.smithi090.stdout: osd failsafe full ratio = 0.95 2023-11-05T08:31:02.884 INFO:teuthology.orchestra.run.smithi090.stdout: osd heartbeat use min delay socket = True 2023-11-05T08:31:02.885 INFO:teuthology.orchestra.run.smithi090.stdout: osd max backfills = 3 2023-11-05T08:31:02.885 INFO:teuthology.orchestra.run.smithi090.stdout: osd max markdown count = 1000 2023-11-05T08:31:02.885 INFO:teuthology.orchestra.run.smithi090.stdout: osd mclock override recovery settings = True 2023-11-05T08:31:02.885 INFO:teuthology.orchestra.run.smithi090.stdout: osd objectstore = bluestore 2023-11-05T08:31:02.885 INFO:teuthology.orchestra.run.smithi090.stdout: osd op queue = debug_random 2023-11-05T08:31:02.886 INFO:teuthology.orchestra.run.smithi090.stdout: osd op queue cut off = debug_random 2023-11-05T08:31:02.886 INFO:teuthology.orchestra.run.smithi090.stdout: osd scrub min interval = 60 2023-11-05T08:31:02.886 INFO:teuthology.orchestra.run.smithi090.stdout: osd shutdown pgref assert = True 2023-11-05T08:31:02.886 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-05T08:31:02.886 INFO:teuthology.orchestra.run.smithi090.stdout:[mgr] 2023-11-05T08:31:02.887 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2023-11-05T08:31:02.887 INFO:teuthology.orchestra.run.smithi090.stdout: debug mgr = 20 2023-11-05T08:31:02.887 INFO:teuthology.orchestra.run.smithi090.stdout: debug mon = 20 2023-11-05T08:31:02.887 INFO:teuthology.orchestra.run.smithi090.stdout: debug auth = 20 2023-11-05T08:31:02.887 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min pgs per osd = 4 2023-11-05T08:31:02.887 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min bytes per osd = 10 2023-11-05T08:31:02.888 INFO:teuthology.orchestra.run.smithi090.stdout: mgr/telemetry/nag = false 2023-11-05T08:31:02.888 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-05T08:31:02.888 INFO:teuthology.orchestra.run.smithi090.stdout:[mon] 2023-11-05T08:31:02.888 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2023-11-05T08:31:02.888 INFO:teuthology.orchestra.run.smithi090.stdout: debug mon = 20 2023-11-05T08:31:02.889 INFO:teuthology.orchestra.run.smithi090.stdout: debug paxos = 20 2023-11-05T08:31:02.889 INFO:teuthology.orchestra.run.smithi090.stdout: debug auth = 20 2023-11-05T08:31:02.889 INFO:teuthology.orchestra.run.smithi090.stdout: mon data avail warn = 5 2023-11-05T08:31:02.889 INFO:teuthology.orchestra.run.smithi090.stdout: mon mgr mkfs grace = 240 2023-11-05T08:31:02.889 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min pgs per osd = 4 2023-11-05T08:31:02.889 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd reporter subtree level = osd 2023-11-05T08:31:02.890 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd prime pg temp = true 2023-11-05T08:31:02.890 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min bytes per osd = 10 2023-11-05T08:31:02.890 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-05T08:31:02.890 INFO:teuthology.orchestra.run.smithi090.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-05T08:31:02.890 INFO:teuthology.orchestra.run.smithi090.stdout: auth mon ticket ttl = 660 # 11m 2023-11-05T08:31:02.891 INFO:teuthology.orchestra.run.smithi090.stdout: auth service ticket ttl = 240 # 4m 2023-11-05T08:31:02.891 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-05T08:31:02.891 INFO:teuthology.orchestra.run.smithi090.stdout: # don't complain about insecure global_id in the test suite 2023-11-05T08:31:02.891 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-05T08:31:02.891 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-05T08:31:02.892 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-05T08:31:02.892 INFO:teuthology.orchestra.run.smithi090.stdout: # 1m isn't quite enough 2023-11-05T08:31:02.892 INFO:teuthology.orchestra.run.smithi090.stdout: mon_down_mkfs_grace = 2m 2023-11-05T08:31:02.892 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-05T08:31:02.893 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_filestore_osds = false 2023-11-05T08:31:02.893 INFO:teuthology.orchestra.run.smithi090.stdout: mon scrub interval = 300 2023-11-05T08:31:02.893 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-11-05T08:31:02.893 INFO:teuthology.orchestra.run.smithi090.stdout:[client] 2023-11-05T08:31:02.893 INFO:teuthology.orchestra.run.smithi090.stdout: rgw cache enabled = true 2023-11-05T08:31:02.894 INFO:teuthology.orchestra.run.smithi090.stdout: rgw enable ops log = true 2023-11-05T08:31:02.894 INFO:teuthology.orchestra.run.smithi090.stdout: rgw enable usage log = true 2023-11-05T08:31:02.894 INFO:teuthology.orchestra.run.smithi090.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-05T08:31:02.894 INFO:teuthology.orchestra.run.smithi090.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-05T08:31:02.894 INFO:teuthology.orchestra.run.smithi090.stdout:[mon.a] 2023-11-05T08:31:02.895 INFO:teuthology.orchestra.run.smithi090.stdout:[mon.b] 2023-11-05T08:31:02.895 INFO:teuthology.orchestra.run.smithi090.stdout:[mon.c] 2023-11-05T08:31:02.895 INFO:teuthology.orchestra.run.smithi053.stdout:[global] 2023-11-05T08:31:02.895 INFO:teuthology.orchestra.run.smithi053.stdout: chdir = "" 2023-11-05T08:31:02.895 INFO:teuthology.orchestra.run.smithi053.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-05T08:31:02.896 INFO:teuthology.orchestra.run.smithi053.stdout: auth supported = cephx 2023-11-05T08:31:02.896 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-11-05T08:31:02.896 INFO:teuthology.orchestra.run.smithi053.stdout: filestore xattr use omap = true 2023-11-05T08:31:02.896 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-11-05T08:31:02.896 INFO:teuthology.orchestra.run.smithi053.stdout: mon clock drift allowed = 1.000 2023-11-05T08:31:02.897 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-11-05T08:31:02.897 INFO:teuthology.orchestra.run.smithi053.stdout: osd crush chooseleaf type = 0 2023-11-05T08:31:02.897 INFO:teuthology.orchestra.run.smithi053.stdout: auth debug = true 2023-11-05T08:31:02.897 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-11-05T08:31:02.897 INFO:teuthology.orchestra.run.smithi053.stdout: ms die on old message = true 2023-11-05T08:31:02.897 INFO:teuthology.orchestra.run.smithi053.stdout: ms die on bug = true 2023-11-05T08:31:02.898 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-11-05T08:31:02.898 INFO:teuthology.orchestra.run.smithi053.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-05T08:31:02.898 INFO:teuthology.orchestra.run.smithi053.stdout: mon pg warn max object skew = 0 2023-11-05T08:31:02.898 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-11-05T08:31:02.898 INFO:teuthology.orchestra.run.smithi053.stdout: # disable pg_autoscaler by default for new pools 2023-11-05T08:31:02.899 INFO:teuthology.orchestra.run.smithi053.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-05T08:31:02.899 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-11-05T08:31:02.899 INFO:teuthology.orchestra.run.smithi053.stdout: osd pool default size = 2 2023-11-05T08:31:02.899 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-11-05T08:31:02.899 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd allow primary affinity = true 2023-11-05T08:31:02.899 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd allow pg remap = true 2023-11-05T08:31:02.900 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on legacy crush tunables = false 2023-11-05T08:31:02.900 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on crush straw calc version zero = false 2023-11-05T08:31:02.900 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on no sortbitwise = false 2023-11-05T08:31:02.900 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on osd down out interval zero = false 2023-11-05T08:31:02.900 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on too few osds = false 2023-11-05T08:31:02.901 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-05T08:31:02.901 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-05T08:31:02.901 INFO:teuthology.orchestra.run.smithi053.stdout: mon_allow_pool_size_one = true 2023-11-05T08:31:02.901 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-11-05T08:31:02.901 INFO:teuthology.orchestra.run.smithi053.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-05T08:31:02.902 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-11-05T08:31:02.902 INFO:teuthology.orchestra.run.smithi053.stdout: osd default data pool replay window = 5 2023-11-05T08:31:02.902 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-11-05T08:31:02.902 INFO:teuthology.orchestra.run.smithi053.stdout: mon allow pool delete = true 2023-11-05T08:31:02.902 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-11-05T08:31:02.902 INFO:teuthology.orchestra.run.smithi053.stdout: mon cluster log file level = debug 2023-11-05T08:31:02.903 INFO:teuthology.orchestra.run.smithi053.stdout: debug asserts on shutdown = true 2023-11-05T08:31:02.903 INFO:teuthology.orchestra.run.smithi053.stdout: mon health detail to clog = false 2023-11-05T08:31:02.903 INFO:teuthology.orchestra.run.smithi053.stdout: mon host = "172.21.15.2,172.21.15.53,172.21.15.69" 2023-11-05T08:31:02.903 INFO:teuthology.orchestra.run.smithi053.stdout: mon client directed command retry = 5 2023-11-05T08:31:02.903 INFO:teuthology.orchestra.run.smithi053.stdout: mon election default strategy = 3 2023-11-05T08:31:02.904 INFO:teuthology.orchestra.run.smithi053.stdout: ms inject socket failures = 5000 2023-11-05T08:31:02.904 INFO:teuthology.orchestra.run.smithi053.stdout: osd_async_recovery_min_cost = 1 2023-11-05T08:31:02.904 INFO:teuthology.orchestra.run.smithi053.stdout: fsid = 8f1fa9b5-06de-4da7-985f-30d4ad103e2d 2023-11-05T08:31:02.904 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-11-05T08:31:02.904 INFO:teuthology.orchestra.run.smithi053.stdout:[osd] 2023-11-05T08:31:02.905 INFO:teuthology.orchestra.run.smithi053.stdout: osd journal size = 100 2023-11-05T08:31:02.905 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-11-05T08:31:02.905 INFO:teuthology.orchestra.run.smithi053.stdout: osd scrub load threshold = 5.0 2023-11-05T08:31:02.905 INFO:teuthology.orchestra.run.smithi053.stdout: osd scrub max interval = 120 2023-11-05T08:31:02.905 INFO:teuthology.orchestra.run.smithi053.stdout: osd mclock profile = high_recovery_ops 2023-11-05T08:31:02.906 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-11-05T08:31:02.906 INFO:teuthology.orchestra.run.smithi053.stdout: osd recover clone overlap = true 2023-11-05T08:31:02.906 INFO:teuthology.orchestra.run.smithi053.stdout: osd recovery max chunk = 1048576 2023-11-05T08:31:02.906 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-11-05T08:31:02.906 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug shutdown = true 2023-11-05T08:31:02.907 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug op order = true 2023-11-05T08:31:02.907 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug verify stray on activate = true 2023-11-05T08:31:02.907 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug trim objects = true 2023-11-05T08:31:02.907 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-11-05T08:31:02.907 INFO:teuthology.orchestra.run.smithi053.stdout: osd open classes on start = true 2023-11-05T08:31:02.908 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug pg log writeout = true 2023-11-05T08:31:02.908 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-11-05T08:31:02.908 INFO:teuthology.orchestra.run.smithi053.stdout: osd deep scrub update digest min age = 30 2023-11-05T08:31:02.908 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-11-05T08:31:02.908 INFO:teuthology.orchestra.run.smithi053.stdout: osd map max advance = 10 2023-11-05T08:31:02.909 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-11-05T08:31:02.909 INFO:teuthology.orchestra.run.smithi053.stdout: journal zero on create = true 2023-11-05T08:31:02.909 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-11-05T08:31:02.909 INFO:teuthology.orchestra.run.smithi053.stdout: filestore ondisk finisher threads = 3 2023-11-05T08:31:02.909 INFO:teuthology.orchestra.run.smithi053.stdout: filestore apply finisher threads = 3 2023-11-05T08:31:02.909 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-11-05T08:31:02.910 INFO:teuthology.orchestra.run.smithi053.stdout: bdev debug aio = true 2023-11-05T08:31:02.910 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug misdirected ops = true 2023-11-05T08:31:02.910 INFO:teuthology.orchestra.run.smithi053.stdout: debug monc = 20 2023-11-05T08:31:02.910 INFO:teuthology.orchestra.run.smithi053.stdout: bluestore block size = 96636764160 2023-11-05T08:31:02.910 INFO:teuthology.orchestra.run.smithi053.stdout: bluestore compression algorithm = zstd 2023-11-05T08:31:02.911 INFO:teuthology.orchestra.run.smithi053.stdout: bluestore compression mode = aggressive 2023-11-05T08:31:02.911 INFO:teuthology.orchestra.run.smithi053.stdout: bluestore fsck on mount = True 2023-11-05T08:31:02.911 INFO:teuthology.orchestra.run.smithi053.stdout: bluestore zero block detection = True 2023-11-05T08:31:02.911 INFO:teuthology.orchestra.run.smithi053.stdout: debug bluefs = 20 2023-11-05T08:31:02.911 INFO:teuthology.orchestra.run.smithi053.stdout: debug bluestore = 20 2023-11-05T08:31:02.911 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2023-11-05T08:31:02.912 INFO:teuthology.orchestra.run.smithi053.stdout: debug osd = 20 2023-11-05T08:31:02.912 INFO:teuthology.orchestra.run.smithi053.stdout: debug rocksdb = 10 2023-11-05T08:31:02.912 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd backfillfull_ratio = 0.85 2023-11-05T08:31:02.912 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd full ratio = 0.9 2023-11-05T08:31:02.912 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd nearfull ratio = 0.8 2023-11-05T08:31:02.912 INFO:teuthology.orchestra.run.smithi053.stdout: osd blocked scrub grace period = 3600 2023-11-05T08:31:02.913 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug reject backfill probability = 0.1 2023-11-05T08:31:02.913 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug verify cached snaps = True 2023-11-05T08:31:02.913 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug verify missing on start = True 2023-11-05T08:31:02.913 INFO:teuthology.orchestra.run.smithi053.stdout: osd failsafe full ratio = 0.95 2023-11-05T08:31:02.913 INFO:teuthology.orchestra.run.smithi053.stdout: osd heartbeat use min delay socket = True 2023-11-05T08:31:02.914 INFO:teuthology.orchestra.run.smithi053.stdout: osd max backfills = 3 2023-11-05T08:31:02.914 INFO:teuthology.orchestra.run.smithi053.stdout: osd max markdown count = 1000 2023-11-05T08:31:02.914 INFO:teuthology.orchestra.run.smithi053.stdout: osd mclock override recovery settings = True 2023-11-05T08:31:02.914 INFO:teuthology.orchestra.run.smithi053.stdout: osd objectstore = bluestore 2023-11-05T08:31:02.914 INFO:teuthology.orchestra.run.smithi053.stdout: osd op queue = debug_random 2023-11-05T08:31:02.915 INFO:teuthology.orchestra.run.smithi053.stdout: osd op queue cut off = debug_random 2023-11-05T08:31:02.915 INFO:teuthology.orchestra.run.smithi053.stdout: osd scrub min interval = 60 2023-11-05T08:31:02.915 INFO:teuthology.orchestra.run.smithi053.stdout: osd shutdown pgref assert = True 2023-11-05T08:31:02.915 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-11-05T08:31:02.915 INFO:teuthology.orchestra.run.smithi053.stdout:[mgr] 2023-11-05T08:31:02.916 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2023-11-05T08:31:02.916 INFO:teuthology.orchestra.run.smithi053.stdout: debug mgr = 20 2023-11-05T08:31:02.916 INFO:teuthology.orchestra.run.smithi053.stdout: debug mon = 20 2023-11-05T08:31:02.916 INFO:teuthology.orchestra.run.smithi053.stdout: debug auth = 20 2023-11-05T08:31:02.916 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min pgs per osd = 4 2023-11-05T08:31:02.916 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min bytes per osd = 10 2023-11-05T08:31:02.917 INFO:teuthology.orchestra.run.smithi053.stdout: mgr/telemetry/nag = false 2023-11-05T08:31:02.917 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-11-05T08:31:02.917 INFO:teuthology.orchestra.run.smithi053.stdout:[mon] 2023-11-05T08:31:02.917 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2023-11-05T08:31:02.917 INFO:teuthology.orchestra.run.smithi053.stdout: debug mon = 20 2023-11-05T08:31:02.918 INFO:teuthology.orchestra.run.smithi053.stdout: debug paxos = 20 2023-11-05T08:31:02.918 INFO:teuthology.orchestra.run.smithi053.stdout: debug auth = 20 2023-11-05T08:31:02.918 INFO:teuthology.orchestra.run.smithi053.stdout: mon data avail warn = 5 2023-11-05T08:31:02.918 INFO:teuthology.orchestra.run.smithi053.stdout: mon mgr mkfs grace = 240 2023-11-05T08:31:02.918 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min pgs per osd = 4 2023-11-05T08:31:02.918 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd reporter subtree level = osd 2023-11-05T08:31:02.919 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd prime pg temp = true 2023-11-05T08:31:02.919 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min bytes per osd = 10 2023-11-05T08:31:02.919 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-11-05T08:31:02.919 INFO:teuthology.orchestra.run.smithi053.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-05T08:31:02.919 INFO:teuthology.orchestra.run.smithi053.stdout: auth mon ticket ttl = 660 # 11m 2023-11-05T08:31:02.920 INFO:teuthology.orchestra.run.smithi053.stdout: auth service ticket ttl = 240 # 4m 2023-11-05T08:31:02.920 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-11-05T08:31:02.920 INFO:teuthology.orchestra.run.smithi053.stdout: # don't complain about insecure global_id in the test suite 2023-11-05T08:31:02.920 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-05T08:31:02.920 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-05T08:31:02.921 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-11-05T08:31:02.921 INFO:teuthology.orchestra.run.smithi053.stdout: # 1m isn't quite enough 2023-11-05T08:31:02.921 INFO:teuthology.orchestra.run.smithi053.stdout: mon_down_mkfs_grace = 2m 2023-11-05T08:31:02.921 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-11-05T08:31:02.921 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_filestore_osds = false 2023-11-05T08:31:02.921 INFO:teuthology.orchestra.run.smithi053.stdout: mon scrub interval = 300 2023-11-05T08:31:02.922 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-11-05T08:31:02.922 INFO:teuthology.orchestra.run.smithi053.stdout:[client] 2023-11-05T08:31:02.922 INFO:teuthology.orchestra.run.smithi053.stdout: rgw cache enabled = true 2023-11-05T08:31:02.922 INFO:teuthology.orchestra.run.smithi053.stdout: rgw enable ops log = true 2023-11-05T08:31:02.923 INFO:teuthology.orchestra.run.smithi053.stdout: rgw enable usage log = true 2023-11-05T08:31:02.923 INFO:teuthology.orchestra.run.smithi053.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-05T08:31:02.923 INFO:teuthology.orchestra.run.smithi053.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-05T08:31:02.923 INFO:teuthology.orchestra.run.smithi053.stdout:[mon.a] 2023-11-05T08:31:02.923 INFO:teuthology.orchestra.run.smithi053.stdout:[mon.b] 2023-11-05T08:31:02.923 INFO:teuthology.orchestra.run.smithi053.stdout:[mon.c] 2023-11-05T08:31:02.924 INFO:teuthology.orchestra.run.smithi069.stdout:[global] 2023-11-05T08:31:02.924 INFO:teuthology.orchestra.run.smithi069.stdout: chdir = "" 2023-11-05T08:31:02.924 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-05T08:31:02.924 INFO:teuthology.orchestra.run.smithi069.stdout: auth supported = cephx 2023-11-05T08:31:02.925 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-05T08:31:02.925 INFO:teuthology.orchestra.run.smithi069.stdout: filestore xattr use omap = true 2023-11-05T08:31:02.925 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-05T08:31:02.925 INFO:teuthology.orchestra.run.smithi069.stdout: mon clock drift allowed = 1.000 2023-11-05T08:31:02.925 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-05T08:31:02.926 INFO:teuthology.orchestra.run.smithi069.stdout: osd crush chooseleaf type = 0 2023-11-05T08:31:02.926 INFO:teuthology.orchestra.run.smithi069.stdout: auth debug = true 2023-11-05T08:31:02.926 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-05T08:31:02.926 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on old message = true 2023-11-05T08:31:02.926 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on bug = true 2023-11-05T08:31:02.927 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-05T08:31:02.927 INFO:teuthology.orchestra.run.smithi069.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-05T08:31:02.927 INFO:teuthology.orchestra.run.smithi069.stdout: mon pg warn max object skew = 0 2023-11-05T08:31:02.927 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-05T08:31:02.927 INFO:teuthology.orchestra.run.smithi069.stdout: # disable pg_autoscaler by default for new pools 2023-11-05T08:31:02.928 INFO:teuthology.orchestra.run.smithi069.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-05T08:31:02.928 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-05T08:31:02.928 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default size = 2 2023-11-05T08:31:02.928 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-05T08:31:02.928 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow primary affinity = true 2023-11-05T08:31:02.929 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow pg remap = true 2023-11-05T08:31:02.929 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on legacy crush tunables = false 2023-11-05T08:31:02.929 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on crush straw calc version zero = false 2023-11-05T08:31:02.929 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on no sortbitwise = false 2023-11-05T08:31:02.929 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on osd down out interval zero = false 2023-11-05T08:31:02.929 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on too few osds = false 2023-11-05T08:31:02.930 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-05T08:31:02.930 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-05T08:31:02.930 INFO:teuthology.orchestra.run.smithi069.stdout: mon_allow_pool_size_one = true 2023-11-05T08:31:02.930 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-05T08:31:02.930 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-05T08:31:02.931 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-05T08:31:02.931 INFO:teuthology.orchestra.run.smithi069.stdout: osd default data pool replay window = 5 2023-11-05T08:31:02.931 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-05T08:31:02.931 INFO:teuthology.orchestra.run.smithi069.stdout: mon allow pool delete = true 2023-11-05T08:31:02.931 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-05T08:31:02.932 INFO:teuthology.orchestra.run.smithi069.stdout: mon cluster log file level = debug 2023-11-05T08:31:02.932 INFO:teuthology.orchestra.run.smithi069.stdout: debug asserts on shutdown = true 2023-11-05T08:31:02.932 INFO:teuthology.orchestra.run.smithi069.stdout: mon health detail to clog = false 2023-11-05T08:31:02.932 INFO:teuthology.orchestra.run.smithi069.stdout: mon host = "172.21.15.2,172.21.15.53,172.21.15.69" 2023-11-05T08:31:02.932 INFO:teuthology.orchestra.run.smithi069.stdout: mon client directed command retry = 5 2023-11-05T08:31:02.932 INFO:teuthology.orchestra.run.smithi069.stdout: mon election default strategy = 3 2023-11-05T08:31:02.933 INFO:teuthology.orchestra.run.smithi069.stdout: ms inject socket failures = 5000 2023-11-05T08:31:02.933 INFO:teuthology.orchestra.run.smithi069.stdout: osd_async_recovery_min_cost = 1 2023-11-05T08:31:02.933 INFO:teuthology.orchestra.run.smithi069.stdout: fsid = 8f1fa9b5-06de-4da7-985f-30d4ad103e2d 2023-11-05T08:31:02.933 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-05T08:31:02.933 INFO:teuthology.orchestra.run.smithi069.stdout:[osd] 2023-11-05T08:31:02.934 INFO:teuthology.orchestra.run.smithi069.stdout: osd journal size = 100 2023-11-05T08:31:02.934 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-05T08:31:02.934 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub load threshold = 5.0 2023-11-05T08:31:02.934 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub max interval = 120 2023-11-05T08:31:02.934 INFO:teuthology.orchestra.run.smithi069.stdout: osd mclock profile = high_recovery_ops 2023-11-05T08:31:02.935 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-05T08:31:02.935 INFO:teuthology.orchestra.run.smithi069.stdout: osd recover clone overlap = true 2023-11-05T08:31:02.935 INFO:teuthology.orchestra.run.smithi069.stdout: osd recovery max chunk = 1048576 2023-11-05T08:31:02.935 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-05T08:31:02.935 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug shutdown = true 2023-11-05T08:31:02.936 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug op order = true 2023-11-05T08:31:02.936 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug verify stray on activate = true 2023-11-05T08:31:02.936 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug trim objects = true 2023-11-05T08:31:02.936 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-05T08:31:02.936 INFO:teuthology.orchestra.run.smithi069.stdout: osd open classes on start = true 2023-11-05T08:31:02.937 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug pg log writeout = true 2023-11-05T08:31:02.937 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-05T08:31:02.937 INFO:teuthology.orchestra.run.smithi069.stdout: osd deep scrub update digest min age = 30 2023-11-05T08:31:02.937 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-05T08:31:02.937 INFO:teuthology.orchestra.run.smithi069.stdout: osd map max advance = 10 2023-11-05T08:31:02.938 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-05T08:31:02.938 INFO:teuthology.orchestra.run.smithi069.stdout: journal zero on create = true 2023-11-05T08:31:02.938 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-05T08:31:02.938 INFO:teuthology.orchestra.run.smithi069.stdout: filestore ondisk finisher threads = 3 2023-11-05T08:31:02.938 INFO:teuthology.orchestra.run.smithi069.stdout: filestore apply finisher threads = 3 2023-11-05T08:31:02.939 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-05T08:31:02.939 INFO:teuthology.orchestra.run.smithi069.stdout: bdev debug aio = true 2023-11-05T08:31:02.939 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug misdirected ops = true 2023-11-05T08:31:02.939 INFO:teuthology.orchestra.run.smithi069.stdout: debug monc = 20 2023-11-05T08:31:02.939 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore block size = 96636764160 2023-11-05T08:31:02.939 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore compression algorithm = zstd 2023-11-05T08:31:02.940 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore compression mode = aggressive 2023-11-05T08:31:02.940 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore fsck on mount = True 2023-11-05T08:31:02.940 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore zero block detection = True 2023-11-05T08:31:02.940 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluefs = 20 2023-11-05T08:31:02.940 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluestore = 20 2023-11-05T08:31:02.941 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2023-11-05T08:31:02.941 INFO:teuthology.orchestra.run.smithi069.stdout: debug osd = 20 2023-11-05T08:31:02.941 INFO:teuthology.orchestra.run.smithi069.stdout: debug rocksdb = 10 2023-11-05T08:31:02.941 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd backfillfull_ratio = 0.85 2023-11-05T08:31:02.941 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd full ratio = 0.9 2023-11-05T08:31:02.942 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd nearfull ratio = 0.8 2023-11-05T08:31:02.942 INFO:teuthology.orchestra.run.smithi069.stdout: osd blocked scrub grace period = 3600 2023-11-05T08:31:02.942 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug reject backfill probability = 0.1 2023-11-05T08:31:02.942 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug verify cached snaps = True 2023-11-05T08:31:02.942 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug verify missing on start = True 2023-11-05T08:31:02.942 INFO:teuthology.orchestra.run.smithi069.stdout: osd failsafe full ratio = 0.95 2023-11-05T08:31:02.943 INFO:teuthology.orchestra.run.smithi069.stdout: osd heartbeat use min delay socket = True 2023-11-05T08:31:02.943 INFO:teuthology.orchestra.run.smithi069.stdout: osd max backfills = 3 2023-11-05T08:31:02.943 INFO:teuthology.orchestra.run.smithi069.stdout: osd max markdown count = 1000 2023-11-05T08:31:02.943 INFO:teuthology.orchestra.run.smithi069.stdout: osd mclock override recovery settings = True 2023-11-05T08:31:02.943 INFO:teuthology.orchestra.run.smithi069.stdout: osd objectstore = bluestore 2023-11-05T08:31:02.944 INFO:teuthology.orchestra.run.smithi069.stdout: osd op queue = debug_random 2023-11-05T08:31:02.944 INFO:teuthology.orchestra.run.smithi069.stdout: osd op queue cut off = debug_random 2023-11-05T08:31:02.944 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub min interval = 60 2023-11-05T08:31:02.944 INFO:teuthology.orchestra.run.smithi069.stdout: osd shutdown pgref assert = True 2023-11-05T08:31:02.944 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-05T08:31:02.945 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr] 2023-11-05T08:31:02.945 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2023-11-05T08:31:02.945 INFO:teuthology.orchestra.run.smithi069.stdout: debug mgr = 20 2023-11-05T08:31:02.945 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2023-11-05T08:31:02.945 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2023-11-05T08:31:02.945 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2023-11-05T08:31:02.946 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2023-11-05T08:31:02.946 INFO:teuthology.orchestra.run.smithi069.stdout: mgr/telemetry/nag = false 2023-11-05T08:31:02.946 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-05T08:31:02.946 INFO:teuthology.orchestra.run.smithi069.stdout:[mon] 2023-11-05T08:31:02.946 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2023-11-05T08:31:02.947 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2023-11-05T08:31:02.947 INFO:teuthology.orchestra.run.smithi069.stdout: debug paxos = 20 2023-11-05T08:31:02.947 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2023-11-05T08:31:02.947 INFO:teuthology.orchestra.run.smithi069.stdout: mon data avail warn = 5 2023-11-05T08:31:02.947 INFO:teuthology.orchestra.run.smithi069.stdout: mon mgr mkfs grace = 240 2023-11-05T08:31:02.948 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2023-11-05T08:31:02.948 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd reporter subtree level = osd 2023-11-05T08:31:02.948 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd prime pg temp = true 2023-11-05T08:31:02.948 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2023-11-05T08:31:02.948 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-05T08:31:02.948 INFO:teuthology.orchestra.run.smithi069.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-05T08:31:02.949 INFO:teuthology.orchestra.run.smithi069.stdout: auth mon ticket ttl = 660 # 11m 2023-11-05T08:31:02.949 INFO:teuthology.orchestra.run.smithi069.stdout: auth service ticket ttl = 240 # 4m 2023-11-05T08:31:02.949 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-05T08:31:02.949 INFO:teuthology.orchestra.run.smithi069.stdout: # don't complain about insecure global_id in the test suite 2023-11-05T08:31:02.949 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-05T08:31:02.950 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-05T08:31:02.950 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-05T08:31:02.950 INFO:teuthology.orchestra.run.smithi069.stdout: # 1m isn't quite enough 2023-11-05T08:31:02.950 INFO:teuthology.orchestra.run.smithi069.stdout: mon_down_mkfs_grace = 2m 2023-11-05T08:31:02.950 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-05T08:31:02.951 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_filestore_osds = false 2023-11-05T08:31:02.951 INFO:teuthology.orchestra.run.smithi069.stdout: mon scrub interval = 300 2023-11-05T08:31:02.951 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-11-05T08:31:02.951 INFO:teuthology.orchestra.run.smithi069.stdout:[client] 2023-11-05T08:31:02.951 INFO:teuthology.orchestra.run.smithi069.stdout: rgw cache enabled = true 2023-11-05T08:31:02.952 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable ops log = true 2023-11-05T08:31:02.952 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable usage log = true 2023-11-05T08:31:02.952 INFO:teuthology.orchestra.run.smithi069.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-05T08:31:02.952 INFO:teuthology.orchestra.run.smithi069.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-05T08:31:02.952 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.a] 2023-11-05T08:31:02.952 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.b] 2023-11-05T08:31:02.953 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.c] 2023-11-05T08:31:02.953 INFO:tasks.ceph:Creating admin key on mon.a... 2023-11-05T08:31:02.953 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-11-05T08:31:03.007 INFO:tasks.ceph:Copying monmap to all nodes... 2023-11-05T08:31:03.007 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-05T08:31:03.007 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-11-05T08:31:03.013 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-05T08:31:03.013 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-11-05T08:31:03.058 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi002.front.sepia.ceph.com 2023-11-05T08:31:03.059 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-05T08:31:03.059 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-05T08:31:03.059 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-05T08:31:03.123 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-05T08:31:03.123 DEBUG:teuthology.orchestra.run.smithi002:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-05T08:31:03.170 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi053.front.sepia.ceph.com 2023-11-05T08:31:03.171 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-11-05T08:31:03.171 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-05T08:31:03.171 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-05T08:31:03.195 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-11-05T08:31:03.195 DEBUG:teuthology.orchestra.run.smithi053:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-05T08:31:03.243 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi069.front.sepia.ceph.com 2023-11-05T08:31:03.243 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-05T08:31:03.243 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-05T08:31:03.244 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-05T08:31:03.267 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-05T08:31:03.268 DEBUG:teuthology.orchestra.run.smithi069:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-05T08:31:03.313 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi090.front.sepia.ceph.com 2023-11-05T08:31:03.314 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-11-05T08:31:03.314 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-05T08:31:03.314 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-05T08:31:03.337 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-11-05T08:31:03.338 DEBUG:teuthology.orchestra.run.smithi090:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-05T08:31:03.385 INFO:tasks.ceph:Setting up mon nodes... 2023-11-05T08:31:03.386 INFO:tasks.ceph:Setting up mgr nodes... 2023-11-05T08:31:03.386 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-11-05T08:31:03.431 INFO:teuthology.orchestra.run.smithi002.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-11-05T08:31:03.435 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-11-05T08:31:03.477 INFO:teuthology.orchestra.run.smithi090.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-11-05T08:31:03.513 INFO:tasks.ceph:Setting up mds nodes... 2023-11-05T08:31:03.514 INFO:tasks.ceph_client:Setting up client nodes... 2023-11-05T08:31:03.514 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-11-05T08:31:03.593 INFO:teuthology.orchestra.run.smithi090.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-11-05T08:31:03.604 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-11-05T08:31:03.604 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi002.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi053.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi069.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi090.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2023-11-05T08:31:03.605 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-11-05T08:31:03.619 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2023-11-05T08:31:03.619 INFO:tasks.ceph:role: osd.0 2023-11-05T08:31:03.620 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi002.front.sepia.ceph.com 2023-11-05T08:31:03.620 DEBUG:teuthology.orchestra.run.smithi002:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-05T08:31:03.675 INFO:teuthology.orchestra.run.smithi002.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-05T08:31:03.675 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-05T08:31:03.675 INFO:teuthology.orchestra.run.smithi002.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-05T08:31:03.676 INFO:teuthology.orchestra.run.smithi002.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-05T08:31:03.676 INFO:teuthology.orchestra.run.smithi002.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-05T08:31:03.676 INFO:teuthology.orchestra.run.smithi002.stdout: = sunit=0 swidth=0 blks 2023-11-05T08:31:03.676 INFO:teuthology.orchestra.run.smithi002.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-05T08:31:03.676 INFO:teuthology.orchestra.run.smithi002.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-05T08:31:03.677 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-05T08:31:03.677 INFO:teuthology.orchestra.run.smithi002.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-05T08:31:03.698 INFO:teuthology.orchestra.run.smithi002.stdout:Discarding blocks...Done. 2023-11-05T08:31:03.700 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi002.front.sepia.ceph.com -o noatime 2023-11-05T08:31:03.701 DEBUG:teuthology.orchestra.run.smithi002:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-11-05T08:31:03.763 DEBUG:teuthology.orchestra.run.smithi002:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-11-05T08:31:03.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-05T08:31:03.815 INFO:teuthology.orchestra.run.smithi002.stderr:sudo: /sbin/restorecon: command not found 2023-11-05T08:31:03.815 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-11-05T08:31:03.868 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2023-11-05T08:31:03.868 INFO:tasks.ceph:role: osd.4 2023-11-05T08:31:03.868 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi002.front.sepia.ceph.com 2023-11-05T08:31:03.869 DEBUG:teuthology.orchestra.run.smithi002:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-05T08:31:03.922 INFO:teuthology.orchestra.run.smithi002.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-05T08:31:03.922 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-05T08:31:03.923 INFO:teuthology.orchestra.run.smithi002.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-05T08:31:03.923 INFO:teuthology.orchestra.run.smithi002.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-05T08:31:03.923 INFO:teuthology.orchestra.run.smithi002.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-05T08:31:03.923 INFO:teuthology.orchestra.run.smithi002.stdout: = sunit=0 swidth=0 blks 2023-11-05T08:31:03.923 INFO:teuthology.orchestra.run.smithi002.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-05T08:31:03.924 INFO:teuthology.orchestra.run.smithi002.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-05T08:31:03.924 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-05T08:31:03.924 INFO:teuthology.orchestra.run.smithi002.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-05T08:31:03.946 INFO:teuthology.orchestra.run.smithi002.stdout:Discarding blocks...Done. 2023-11-05T08:31:03.947 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi002.front.sepia.ceph.com -o noatime 2023-11-05T08:31:03.948 DEBUG:teuthology.orchestra.run.smithi002:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2023-11-05T08:31:04.006 DEBUG:teuthology.orchestra.run.smithi002:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-11-05T08:31:04.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-05T08:31:04.060 INFO:teuthology.orchestra.run.smithi002.stderr:sudo: /sbin/restorecon: command not found 2023-11-05T08:31:04.060 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/osd/ceph-8 2023-11-05T08:31:04.114 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2023-11-05T08:31:04.114 INFO:tasks.ceph:role: osd.8 2023-11-05T08:31:04.114 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi002.front.sepia.ceph.com 2023-11-05T08:31:04.115 DEBUG:teuthology.orchestra.run.smithi002:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-05T08:31:04.170 INFO:teuthology.orchestra.run.smithi002.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-05T08:31:04.170 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-05T08:31:04.171 INFO:teuthology.orchestra.run.smithi002.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-05T08:31:04.171 INFO:teuthology.orchestra.run.smithi002.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-05T08:31:04.171 INFO:teuthology.orchestra.run.smithi002.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-05T08:31:04.171 INFO:teuthology.orchestra.run.smithi002.stdout: = sunit=0 swidth=0 blks 2023-11-05T08:31:04.171 INFO:teuthology.orchestra.run.smithi002.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-05T08:31:04.172 INFO:teuthology.orchestra.run.smithi002.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-05T08:31:04.172 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-05T08:31:04.172 INFO:teuthology.orchestra.run.smithi002.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-05T08:31:04.194 INFO:teuthology.orchestra.run.smithi002.stdout:Discarding blocks...Done. 2023-11-05T08:31:04.196 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi002.front.sepia.ceph.com -o noatime 2023-11-05T08:31:04.196 DEBUG:teuthology.orchestra.run.smithi002:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-8 2023-11-05T08:31:04.253 DEBUG:teuthology.orchestra.run.smithi002:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-8 2023-11-05T08:31:04.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-05T08:31:04.307 INFO:teuthology.orchestra.run.smithi002.stderr:sudo: /sbin/restorecon: command not found 2023-11-05T08:31:04.308 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/osd/ceph-12 2023-11-05T08:31:04.360 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2023-11-05T08:31:04.360 INFO:tasks.ceph:role: osd.12 2023-11-05T08:31:04.361 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi002.front.sepia.ceph.com 2023-11-05T08:31:04.361 DEBUG:teuthology.orchestra.run.smithi002:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-11-05T08:31:04.415 INFO:teuthology.orchestra.run.smithi002.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-11-05T08:31:04.415 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-05T08:31:04.415 INFO:teuthology.orchestra.run.smithi002.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-05T08:31:04.416 INFO:teuthology.orchestra.run.smithi002.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-05T08:31:04.416 INFO:teuthology.orchestra.run.smithi002.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-05T08:31:04.416 INFO:teuthology.orchestra.run.smithi002.stdout: = sunit=0 swidth=0 blks 2023-11-05T08:31:04.416 INFO:teuthology.orchestra.run.smithi002.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-05T08:31:04.416 INFO:teuthology.orchestra.run.smithi002.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-05T08:31:04.416 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-05T08:31:04.417 INFO:teuthology.orchestra.run.smithi002.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-05T08:31:04.439 INFO:teuthology.orchestra.run.smithi002.stdout:Discarding blocks...Done. 2023-11-05T08:31:04.441 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi002.front.sepia.ceph.com -o noatime 2023-11-05T08:31:04.441 DEBUG:teuthology.orchestra.run.smithi002:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-12 2023-11-05T08:31:04.498 DEBUG:teuthology.orchestra.run.smithi002:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-12 2023-11-05T08:31:04.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-05T08:31:04.558 INFO:teuthology.orchestra.run.smithi002.stderr:sudo: /sbin/restorecon: command not found 2023-11-05T08:31:04.559 DEBUG:teuthology.orchestra.run.smithi002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-05T08:31:04.671 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-05T08:31:04.670+0000 7f16da1ef640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-11-05T08:31:04.671 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-05T08:31:04.670+0000 7f16da1ef640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-11-05T08:31:04.671 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-05T08:31:04.670+0000 7f16da1ef640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-05T08:31:04.671 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-05T08:31:04.670+0000 7f16da1ef640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-05T08:31:04.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-05T08:31:04.670+0000 7f16da1ef640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-05T08:31:04.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-05T08:31:04.670+0000 7f16da1ef640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-11-05T08:31:07.679 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-11-05T08:31:07.694 DEBUG:teuthology.orchestra.run.smithi002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-05T08:31:07.783 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-05T08:31:07.782+0000 7f979e47f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-11-05T08:31:07.783 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-05T08:31:07.782+0000 7f979e47f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-11-05T08:31:07.784 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-05T08:31:07.782+0000 7f979e47f640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-05T08:31:07.784 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-05T08:31:07.782+0000 7f979e47f640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-05T08:31:07.784 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-05T08:31:07.782+0000 7f979e47f640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-05T08:31:07.784 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-05T08:31:07.782+0000 7f979e47f640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-11-05T08:31:10.809 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-11-05T08:31:10.867 DEBUG:teuthology.orchestra.run.smithi002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 8 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-05T08:31:10.951 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-05T08:31:10.950+0000 7fa85367e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-8/keyring: can't open /var/lib/ceph/osd/ceph-8/keyring: (2) No such file or directory 2023-11-05T08:31:10.951 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-05T08:31:10.950+0000 7fa85367e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-8/keyring 2023-11-05T08:31:10.951 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-05T08:31:10.950+0000 7fa85367e640 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2023-11-05T08:31:10.951 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-05T08:31:10.950+0000 7fa85367e640 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2023-11-05T08:31:10.952 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-05T08:31:10.950+0000 7fa85367e640 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2023-11-05T08:31:10.952 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-05T08:31:10.950+0000 7fa85367e640 -1 bluestore(/var/lib/ceph/osd/ceph-8) _read_fsid unparsable uuid 2023-11-05T08:31:13.949 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2023-11-05T08:31:14.003 DEBUG:teuthology.orchestra.run.smithi002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 12 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-05T08:31:14.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-05T08:31:14.090+0000 7f780ce4f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-12/keyring: can't open /var/lib/ceph/osd/ceph-12/keyring: (2) No such file or directory 2023-11-05T08:31:14.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-05T08:31:14.090+0000 7f780ce4f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-12/keyring 2023-11-05T08:31:14.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-05T08:31:14.090+0000 7f780ce4f640 -1 bluestore(/var/lib/ceph/osd/ceph-12/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-12/block: (2) No such file or directory 2023-11-05T08:31:14.092 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-05T08:31:14.090+0000 7f780ce4f640 -1 bluestore(/var/lib/ceph/osd/ceph-12/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-12/block: (2) No such file or directory 2023-11-05T08:31:14.092 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-05T08:31:14.090+0000 7f780ce4f640 -1 bluestore(/var/lib/ceph/osd/ceph-12/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-12/block: (2) No such file or directory 2023-11-05T08:31:14.092 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-05T08:31:14.090+0000 7f780ce4f640 -1 bluestore(/var/lib/ceph/osd/ceph-12) _read_fsid unparsable uuid 2023-11-05T08:31:17.111 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-12 2023-11-05T08:31:17.167 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-11-05T08:31:17.182 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2023-11-05T08:31:17.182 INFO:tasks.ceph:role: osd.1 2023-11-05T08:31:17.182 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi053.front.sepia.ceph.com 2023-11-05T08:31:17.183 DEBUG:teuthology.orchestra.run.smithi053:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-05T08:31:17.238 INFO:teuthology.orchestra.run.smithi053.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-05T08:31:17.238 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-05T08:31:17.238 INFO:teuthology.orchestra.run.smithi053.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-05T08:31:17.239 INFO:teuthology.orchestra.run.smithi053.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-05T08:31:17.239 INFO:teuthology.orchestra.run.smithi053.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-05T08:31:17.239 INFO:teuthology.orchestra.run.smithi053.stdout: = sunit=0 swidth=0 blks 2023-11-05T08:31:17.239 INFO:teuthology.orchestra.run.smithi053.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-05T08:31:17.239 INFO:teuthology.orchestra.run.smithi053.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-05T08:31:17.240 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-05T08:31:17.240 INFO:teuthology.orchestra.run.smithi053.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-05T08:31:17.262 INFO:teuthology.orchestra.run.smithi053.stdout:Discarding blocks...Done. 2023-11-05T08:31:17.264 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi053.front.sepia.ceph.com -o noatime 2023-11-05T08:31:17.264 DEBUG:teuthology.orchestra.run.smithi053:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-1 2023-11-05T08:31:17.323 DEBUG:teuthology.orchestra.run.smithi053:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-11-05T08:31:17.376 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: /sbin/restorecon: command not found 2023-11-05T08:31:17.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-05T08:31:17.377 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-11-05T08:31:17.434 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2023-11-05T08:31:17.435 INFO:tasks.ceph:role: osd.5 2023-11-05T08:31:17.435 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi053.front.sepia.ceph.com 2023-11-05T08:31:17.435 DEBUG:teuthology.orchestra.run.smithi053:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-05T08:31:17.490 INFO:teuthology.orchestra.run.smithi053.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-05T08:31:17.490 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-05T08:31:17.490 INFO:teuthology.orchestra.run.smithi053.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-05T08:31:17.491 INFO:teuthology.orchestra.run.smithi053.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-05T08:31:17.491 INFO:teuthology.orchestra.run.smithi053.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-05T08:31:17.491 INFO:teuthology.orchestra.run.smithi053.stdout: = sunit=0 swidth=0 blks 2023-11-05T08:31:17.491 INFO:teuthology.orchestra.run.smithi053.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-05T08:31:17.491 INFO:teuthology.orchestra.run.smithi053.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-05T08:31:17.492 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-05T08:31:17.492 INFO:teuthology.orchestra.run.smithi053.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-05T08:31:17.513 INFO:teuthology.orchestra.run.smithi053.stdout:Discarding blocks...Done. 2023-11-05T08:31:17.515 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi053.front.sepia.ceph.com -o noatime 2023-11-05T08:31:17.515 DEBUG:teuthology.orchestra.run.smithi053:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-11-05T08:31:17.576 DEBUG:teuthology.orchestra.run.smithi053:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-11-05T08:31:17.626 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: /sbin/restorecon: command not found 2023-11-05T08:31:17.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-05T08:31:17.627 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/osd/ceph-9 2023-11-05T08:31:17.680 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2023-11-05T08:31:17.680 INFO:tasks.ceph:role: osd.9 2023-11-05T08:31:17.680 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi053.front.sepia.ceph.com 2023-11-05T08:31:17.681 DEBUG:teuthology.orchestra.run.smithi053:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-05T08:31:17.734 INFO:teuthology.orchestra.run.smithi053.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-05T08:31:17.734 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-05T08:31:17.735 INFO:teuthology.orchestra.run.smithi053.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-05T08:31:17.735 INFO:teuthology.orchestra.run.smithi053.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-05T08:31:17.735 INFO:teuthology.orchestra.run.smithi053.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-05T08:31:17.735 INFO:teuthology.orchestra.run.smithi053.stdout: = sunit=0 swidth=0 blks 2023-11-05T08:31:17.735 INFO:teuthology.orchestra.run.smithi053.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-05T08:31:17.736 INFO:teuthology.orchestra.run.smithi053.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-05T08:31:17.736 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-05T08:31:17.736 INFO:teuthology.orchestra.run.smithi053.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-05T08:31:17.757 INFO:teuthology.orchestra.run.smithi053.stdout:Discarding blocks...Done. 2023-11-05T08:31:17.759 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi053.front.sepia.ceph.com -o noatime 2023-11-05T08:31:17.759 DEBUG:teuthology.orchestra.run.smithi053:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-9 2023-11-05T08:31:17.818 DEBUG:teuthology.orchestra.run.smithi053:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-9 2023-11-05T08:31:17.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-05T08:31:17.871 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: /sbin/restorecon: command not found 2023-11-05T08:31:17.871 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/osd/ceph-13 2023-11-05T08:31:17.923 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2023-11-05T08:31:17.924 INFO:tasks.ceph:role: osd.13 2023-11-05T08:31:17.924 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi053.front.sepia.ceph.com 2023-11-05T08:31:17.924 DEBUG:teuthology.orchestra.run.smithi053:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-11-05T08:31:17.977 INFO:teuthology.orchestra.run.smithi053.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-11-05T08:31:17.977 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-05T08:31:17.978 INFO:teuthology.orchestra.run.smithi053.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-05T08:31:17.978 INFO:teuthology.orchestra.run.smithi053.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-05T08:31:17.978 INFO:teuthology.orchestra.run.smithi053.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-05T08:31:17.978 INFO:teuthology.orchestra.run.smithi053.stdout: = sunit=0 swidth=0 blks 2023-11-05T08:31:17.979 INFO:teuthology.orchestra.run.smithi053.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-05T08:31:17.979 INFO:teuthology.orchestra.run.smithi053.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-05T08:31:17.979 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-05T08:31:17.979 INFO:teuthology.orchestra.run.smithi053.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-05T08:31:18.000 INFO:teuthology.orchestra.run.smithi053.stdout:Discarding blocks...Done. 2023-11-05T08:31:18.002 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi053.front.sepia.ceph.com -o noatime 2023-11-05T08:31:18.003 DEBUG:teuthology.orchestra.run.smithi053:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-13 2023-11-05T08:31:18.062 DEBUG:teuthology.orchestra.run.smithi053:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-13 2023-11-05T08:31:18.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-05T08:31:18.115 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: /sbin/restorecon: command not found 2023-11-05T08:31:18.115 DEBUG:teuthology.orchestra.run.smithi053:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-05T08:31:18.221 INFO:teuthology.orchestra.run.smithi053.stderr:2023-11-05T08:31:18.215+0000 7fdaf9b37640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-11-05T08:31:18.222 INFO:teuthology.orchestra.run.smithi053.stderr:2023-11-05T08:31:18.215+0000 7fdaf9b37640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-11-05T08:31:18.222 INFO:teuthology.orchestra.run.smithi053.stderr:2023-11-05T08:31:18.215+0000 7fdaf9b37640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-05T08:31:18.222 INFO:teuthology.orchestra.run.smithi053.stderr:2023-11-05T08:31:18.215+0000 7fdaf9b37640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-05T08:31:18.222 INFO:teuthology.orchestra.run.smithi053.stderr:2023-11-05T08:31:18.215+0000 7fdaf9b37640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-05T08:31:18.223 INFO:teuthology.orchestra.run.smithi053.stderr:2023-11-05T08:31:18.219+0000 7fdaf9b37640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-11-05T08:31:21.235 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-11-05T08:31:21.249 DEBUG:teuthology.orchestra.run.smithi053:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-05T08:31:21.335 INFO:teuthology.orchestra.run.smithi053.stderr:2023-11-05T08:31:21.331+0000 7f8cb1b24640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-11-05T08:31:21.335 INFO:teuthology.orchestra.run.smithi053.stderr:2023-11-05T08:31:21.331+0000 7f8cb1b24640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-11-05T08:31:21.336 INFO:teuthology.orchestra.run.smithi053.stderr:2023-11-05T08:31:21.331+0000 7f8cb1b24640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-05T08:31:21.336 INFO:teuthology.orchestra.run.smithi053.stderr:2023-11-05T08:31:21.331+0000 7f8cb1b24640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-05T08:31:21.336 INFO:teuthology.orchestra.run.smithi053.stderr:2023-11-05T08:31:21.331+0000 7f8cb1b24640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-05T08:31:21.336 INFO:teuthology.orchestra.run.smithi053.stderr:2023-11-05T08:31:21.331+0000 7f8cb1b24640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-11-05T08:31:24.366 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-11-05T08:31:24.423 DEBUG:teuthology.orchestra.run.smithi053:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 9 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-05T08:31:24.507 INFO:teuthology.orchestra.run.smithi053.stderr:2023-11-05T08:31:24.503+0000 7f5f8639b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-9/keyring: can't open /var/lib/ceph/osd/ceph-9/keyring: (2) No such file or directory 2023-11-05T08:31:24.508 INFO:teuthology.orchestra.run.smithi053.stderr:2023-11-05T08:31:24.503+0000 7f5f8639b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-9/keyring 2023-11-05T08:31:24.508 INFO:teuthology.orchestra.run.smithi053.stderr:2023-11-05T08:31:24.503+0000 7f5f8639b640 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2023-11-05T08:31:24.508 INFO:teuthology.orchestra.run.smithi053.stderr:2023-11-05T08:31:24.503+0000 7f5f8639b640 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2023-11-05T08:31:24.508 INFO:teuthology.orchestra.run.smithi053.stderr:2023-11-05T08:31:24.503+0000 7f5f8639b640 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2023-11-05T08:31:24.509 INFO:teuthology.orchestra.run.smithi053.stderr:2023-11-05T08:31:24.503+0000 7f5f8639b640 -1 bluestore(/var/lib/ceph/osd/ceph-9) _read_fsid unparsable uuid 2023-11-05T08:31:27.505 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-9 2023-11-05T08:31:27.563 DEBUG:teuthology.orchestra.run.smithi053:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 13 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-05T08:31:27.647 INFO:teuthology.orchestra.run.smithi053.stderr:2023-11-05T08:31:27.643+0000 7f91086a4640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-13/keyring: can't open /var/lib/ceph/osd/ceph-13/keyring: (2) No such file or directory 2023-11-05T08:31:27.648 INFO:teuthology.orchestra.run.smithi053.stderr:2023-11-05T08:31:27.643+0000 7f91086a4640 -1 created new key in keyring /var/lib/ceph/osd/ceph-13/keyring 2023-11-05T08:31:27.648 INFO:teuthology.orchestra.run.smithi053.stderr:2023-11-05T08:31:27.643+0000 7f91086a4640 -1 bluestore(/var/lib/ceph/osd/ceph-13/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-13/block: (2) No such file or directory 2023-11-05T08:31:27.648 INFO:teuthology.orchestra.run.smithi053.stderr:2023-11-05T08:31:27.643+0000 7f91086a4640 -1 bluestore(/var/lib/ceph/osd/ceph-13/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-13/block: (2) No such file or directory 2023-11-05T08:31:27.648 INFO:teuthology.orchestra.run.smithi053.stderr:2023-11-05T08:31:27.643+0000 7f91086a4640 -1 bluestore(/var/lib/ceph/osd/ceph-13/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-13/block: (2) No such file or directory 2023-11-05T08:31:27.649 INFO:teuthology.orchestra.run.smithi053.stderr:2023-11-05T08:31:27.643+0000 7f91086a4640 -1 bluestore(/var/lib/ceph/osd/ceph-13) _read_fsid unparsable uuid 2023-11-05T08:31:30.653 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-13 2023-11-05T08:31:30.711 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-11-05T08:31:30.725 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2023-11-05T08:31:30.726 INFO:tasks.ceph:role: osd.2 2023-11-05T08:31:30.726 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi069.front.sepia.ceph.com 2023-11-05T08:31:30.726 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-05T08:31:30.789 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-05T08:31:30.789 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-05T08:31:30.790 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-05T08:31:30.790 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-05T08:31:30.790 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-05T08:31:30.790 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2023-11-05T08:31:30.791 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-05T08:31:30.791 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-05T08:31:30.791 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-05T08:31:30.791 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-05T08:31:31.276 INFO:teuthology.orchestra.run.smithi069.stdout:Discarding blocks...Done. 2023-11-05T08:31:31.278 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2023-11-05T08:31:31.278 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2023-11-05T08:31:31.415 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-11-05T08:31:31.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-05T08:31:31.427 INFO:teuthology.orchestra.run.smithi069.stderr:sudo: /sbin/restorecon: command not found 2023-11-05T08:31:31.428 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-11-05T08:31:31.480 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2023-11-05T08:31:31.480 INFO:tasks.ceph:role: osd.6 2023-11-05T08:31:31.480 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi069.front.sepia.ceph.com 2023-11-05T08:31:31.481 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-05T08:31:31.542 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-05T08:31:31.542 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-05T08:31:31.543 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-05T08:31:31.543 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-05T08:31:31.543 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-05T08:31:31.543 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2023-11-05T08:31:31.543 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-05T08:31:31.544 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-05T08:31:31.544 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-05T08:31:31.544 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-05T08:31:31.972 INFO:teuthology.orchestra.run.smithi069.stdout:Discarding blocks...Done. 2023-11-05T08:31:31.975 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2023-11-05T08:31:31.975 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-6 2023-11-05T08:31:32.037 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-11-05T08:31:32.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-05T08:31:32.094 INFO:teuthology.orchestra.run.smithi069.stderr:sudo: /sbin/restorecon: command not found 2023-11-05T08:31:32.094 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/ceph-10 2023-11-05T08:31:32.148 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2023-11-05T08:31:32.148 INFO:tasks.ceph:role: osd.10 2023-11-05T08:31:32.148 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi069.front.sepia.ceph.com 2023-11-05T08:31:32.149 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-05T08:31:32.211 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-05T08:31:32.211 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-05T08:31:32.211 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-05T08:31:32.212 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-05T08:31:32.212 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-05T08:31:32.212 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2023-11-05T08:31:32.212 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-05T08:31:32.212 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-05T08:31:32.213 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-05T08:31:32.213 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-05T08:31:32.666 INFO:teuthology.orchestra.run.smithi069.stdout:Discarding blocks...Done. 2023-11-05T08:31:32.667 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2023-11-05T08:31:32.668 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-10 2023-11-05T08:31:32.759 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-10 2023-11-05T08:31:32.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-05T08:31:32.769 INFO:teuthology.orchestra.run.smithi069.stderr:sudo: /sbin/restorecon: command not found 2023-11-05T08:31:32.769 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/ceph-14 2023-11-05T08:31:32.822 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2023-11-05T08:31:32.822 INFO:tasks.ceph:role: osd.14 2023-11-05T08:31:32.822 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi069.front.sepia.ceph.com 2023-11-05T08:31:32.823 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-11-05T08:31:32.885 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-11-05T08:31:32.885 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-05T08:31:32.885 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-05T08:31:32.885 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-05T08:31:32.886 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-05T08:31:32.886 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2023-11-05T08:31:32.886 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-05T08:31:32.886 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-05T08:31:32.886 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-05T08:31:32.887 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-05T08:31:33.348 INFO:teuthology.orchestra.run.smithi069.stdout:Discarding blocks...Done. 2023-11-05T08:31:33.350 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2023-11-05T08:31:33.351 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-14 2023-11-05T08:31:33.475 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-14 2023-11-05T08:31:33.484 INFO:teuthology.orchestra.run.smithi069.stderr:sudo: /sbin/restorecon: command not found 2023-11-05T08:31:33.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-05T08:31:33.486 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-05T08:31:33.665 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-05T08:31:33.663+0000 7f217cd47640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-11-05T08:31:33.670 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-05T08:31:33.671+0000 7f217cd47640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-11-05T08:31:33.670 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-05T08:31:33.671+0000 7f217cd47640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-05T08:31:33.670 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-05T08:31:33.671+0000 7f217cd47640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-05T08:31:33.670 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-05T08:31:33.671+0000 7f217cd47640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-05T08:31:33.677 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-05T08:31:33.679+0000 7f217cd47640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-11-05T08:31:36.706 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-11-05T08:31:36.719 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-05T08:31:36.799 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-05T08:31:36.799+0000 7f81f2df8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-11-05T08:31:36.799 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-05T08:31:36.799+0000 7f81f2df8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-11-05T08:31:36.800 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-05T08:31:36.799+0000 7f81f2df8640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-11-05T08:31:36.800 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-05T08:31:36.799+0000 7f81f2df8640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-11-05T08:31:36.800 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-05T08:31:36.799+0000 7f81f2df8640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-11-05T08:31:36.800 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-05T08:31:36.803+0000 7f81f2df8640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-11-05T08:31:39.812 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-11-05T08:31:39.869 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 10 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-05T08:31:39.953 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-05T08:31:39.955+0000 7f36ee1d8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-10/keyring: can't open /var/lib/ceph/osd/ceph-10/keyring: (2) No such file or directory 2023-11-05T08:31:39.954 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-05T08:31:39.955+0000 7f36ee1d8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-10/keyring 2023-11-05T08:31:39.954 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-05T08:31:39.955+0000 7f36ee1d8640 -1 bluestore(/var/lib/ceph/osd/ceph-10/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-10/block: (2) No such file or directory 2023-11-05T08:31:39.954 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-05T08:31:39.955+0000 7f36ee1d8640 -1 bluestore(/var/lib/ceph/osd/ceph-10/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-10/block: (2) No such file or directory 2023-11-05T08:31:39.954 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-05T08:31:39.955+0000 7f36ee1d8640 -1 bluestore(/var/lib/ceph/osd/ceph-10/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-10/block: (2) No such file or directory 2023-11-05T08:31:39.955 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-05T08:31:39.955+0000 7f36ee1d8640 -1 bluestore(/var/lib/ceph/osd/ceph-10) _read_fsid unparsable uuid 2023-11-05T08:31:42.968 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-10 2023-11-05T08:31:43.024 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 14 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-05T08:31:43.107 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-05T08:31:43.107+0000 7f338eb72640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-14/keyring: can't open /var/lib/ceph/osd/ceph-14/keyring: (2) No such file or directory 2023-11-05T08:31:43.108 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-05T08:31:43.111+0000 7f338eb72640 -1 created new key in keyring /var/lib/ceph/osd/ceph-14/keyring 2023-11-05T08:31:43.109 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-05T08:31:43.111+0000 7f338eb72640 -1 bluestore(/var/lib/ceph/osd/ceph-14/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-14/block: (2) No such file or directory 2023-11-05T08:31:43.109 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-05T08:31:43.111+0000 7f338eb72640 -1 bluestore(/var/lib/ceph/osd/ceph-14/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-14/block: (2) No such file or directory 2023-11-05T08:31:43.109 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-05T08:31:43.111+0000 7f338eb72640 -1 bluestore(/var/lib/ceph/osd/ceph-14/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-14/block: (2) No such file or directory 2023-11-05T08:31:43.110 INFO:teuthology.orchestra.run.smithi069.stderr:2023-11-05T08:31:43.111+0000 7f338eb72640 -1 bluestore(/var/lib/ceph/osd/ceph-14) _read_fsid unparsable uuid 2023-11-05T08:31:46.121 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-14 2023-11-05T08:31:46.180 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-11-05T08:31:46.195 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2023-11-05T08:31:46.195 INFO:tasks.ceph:role: osd.3 2023-11-05T08:31:46.195 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi090.front.sepia.ceph.com 2023-11-05T08:31:46.196 DEBUG:teuthology.orchestra.run.smithi090:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-05T08:31:46.256 INFO:teuthology.orchestra.run.smithi090.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-05T08:31:46.256 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-05T08:31:46.256 INFO:teuthology.orchestra.run.smithi090.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-05T08:31:46.256 INFO:teuthology.orchestra.run.smithi090.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-05T08:31:46.257 INFO:teuthology.orchestra.run.smithi090.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-05T08:31:46.257 INFO:teuthology.orchestra.run.smithi090.stdout: = sunit=0 swidth=0 blks 2023-11-05T08:31:46.257 INFO:teuthology.orchestra.run.smithi090.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-05T08:31:46.257 INFO:teuthology.orchestra.run.smithi090.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-05T08:31:46.257 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-05T08:31:46.258 INFO:teuthology.orchestra.run.smithi090.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-05T08:31:46.712 INFO:teuthology.orchestra.run.smithi090.stdout:Discarding blocks...Done. 2023-11-05T08:31:46.714 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi090.front.sepia.ceph.com -o noatime 2023-11-05T08:31:46.715 DEBUG:teuthology.orchestra.run.smithi090:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2023-11-05T08:31:46.809 DEBUG:teuthology.orchestra.run.smithi090:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-11-05T08:31:46.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-05T08:31:46.820 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: /sbin/restorecon: command not found 2023-11-05T08:31:46.821 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-11-05T08:31:46.875 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2023-11-05T08:31:46.875 INFO:tasks.ceph:role: osd.7 2023-11-05T08:31:46.875 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi090.front.sepia.ceph.com 2023-11-05T08:31:46.876 DEBUG:teuthology.orchestra.run.smithi090:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-05T08:31:46.935 INFO:teuthology.orchestra.run.smithi090.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-05T08:31:46.935 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-05T08:31:46.935 INFO:teuthology.orchestra.run.smithi090.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-05T08:31:46.935 INFO:teuthology.orchestra.run.smithi090.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-05T08:31:46.936 INFO:teuthology.orchestra.run.smithi090.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-05T08:31:46.936 INFO:teuthology.orchestra.run.smithi090.stdout: = sunit=0 swidth=0 blks 2023-11-05T08:31:46.936 INFO:teuthology.orchestra.run.smithi090.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-05T08:31:46.936 INFO:teuthology.orchestra.run.smithi090.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-05T08:31:46.936 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-05T08:31:46.937 INFO:teuthology.orchestra.run.smithi090.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-05T08:31:47.401 INFO:teuthology.orchestra.run.smithi090.stdout:Discarding blocks...Done. 2023-11-05T08:31:47.403 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi090.front.sepia.ceph.com -o noatime 2023-11-05T08:31:47.403 DEBUG:teuthology.orchestra.run.smithi090:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-7 2023-11-05T08:31:47.544 DEBUG:teuthology.orchestra.run.smithi090:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-11-05T08:31:47.555 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: /sbin/restorecon: command not found 2023-11-05T08:31:47.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-05T08:31:47.557 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/osd/ceph-11 2023-11-05T08:31:47.612 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2023-11-05T08:31:47.612 INFO:tasks.ceph:role: osd.11 2023-11-05T08:31:47.612 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi090.front.sepia.ceph.com 2023-11-05T08:31:47.613 DEBUG:teuthology.orchestra.run.smithi090:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-05T08:31:47.673 INFO:teuthology.orchestra.run.smithi090.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-05T08:31:47.673 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-05T08:31:47.674 INFO:teuthology.orchestra.run.smithi090.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-05T08:31:47.674 INFO:teuthology.orchestra.run.smithi090.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-05T08:31:47.674 INFO:teuthology.orchestra.run.smithi090.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-05T08:31:47.674 INFO:teuthology.orchestra.run.smithi090.stdout: = sunit=0 swidth=0 blks 2023-11-05T08:31:47.674 INFO:teuthology.orchestra.run.smithi090.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-05T08:31:47.675 INFO:teuthology.orchestra.run.smithi090.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-05T08:31:47.675 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-05T08:31:47.675 INFO:teuthology.orchestra.run.smithi090.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-05T08:31:48.147 INFO:teuthology.orchestra.run.smithi090.stdout:Discarding blocks...Done. 2023-11-05T08:31:48.148 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi090.front.sepia.ceph.com -o noatime 2023-11-05T08:31:48.149 DEBUG:teuthology.orchestra.run.smithi090:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-11 2023-11-05T08:31:48.302 DEBUG:teuthology.orchestra.run.smithi090:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-11 2023-11-05T08:31:48.314 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: /sbin/restorecon: command not found 2023-11-05T08:31:48.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-05T08:31:48.315 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/osd/ceph-15 2023-11-05T08:31:48.368 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2023-11-05T08:31:48.369 INFO:tasks.ceph:role: osd.15 2023-11-05T08:31:48.369 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi090.front.sepia.ceph.com 2023-11-05T08:31:48.369 DEBUG:teuthology.orchestra.run.smithi090:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-11-05T08:31:48.432 INFO:teuthology.orchestra.run.smithi090.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-11-05T08:31:48.432 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-05T08:31:48.433 INFO:teuthology.orchestra.run.smithi090.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-05T08:31:48.433 INFO:teuthology.orchestra.run.smithi090.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-05T08:31:48.433 INFO:teuthology.orchestra.run.smithi090.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-05T08:31:48.433 INFO:teuthology.orchestra.run.smithi090.stdout: = sunit=0 swidth=0 blks 2023-11-05T08:31:48.434 INFO:teuthology.orchestra.run.smithi090.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-05T08:31:48.434 INFO:teuthology.orchestra.run.smithi090.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-05T08:31:48.434 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-05T08:31:48.434 INFO:teuthology.orchestra.run.smithi090.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-05T08:31:48.855 INFO:teuthology.orchestra.run.smithi090.stdout:Discarding blocks...Done. 2023-11-05T08:31:48.857 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi090.front.sepia.ceph.com -o noatime 2023-11-05T08:31:48.858 DEBUG:teuthology.orchestra.run.smithi090:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-15 2023-11-05T08:31:48.950 DEBUG:teuthology.orchestra.run.smithi090:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-15 2023-11-05T08:31:48.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-05T08:31:48.960 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: /sbin/restorecon: command not found 2023-11-05T08:31:48.960 DEBUG:teuthology.orchestra.run.smithi090:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-05T08:31:49.062 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-05T08:31:49.059+0000 7f231558b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-11-05T08:31:49.063 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-05T08:31:49.059+0000 7f231558b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-11-05T08:31:49.064 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-05T08:31:49.059+0000 7f231558b640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-05T08:31:49.064 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-05T08:31:49.059+0000 7f231558b640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-05T08:31:49.064 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-05T08:31:49.059+0000 7f231558b640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-05T08:31:49.064 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-05T08:31:49.059+0000 7f231558b640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-11-05T08:31:52.071 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-11-05T08:31:52.085 DEBUG:teuthology.orchestra.run.smithi090:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-05T08:31:52.172 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-05T08:31:52.167+0000 7f0e1700a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-11-05T08:31:52.172 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-05T08:31:52.167+0000 7f0e1700a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-11-05T08:31:52.173 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-05T08:31:52.167+0000 7f0e1700a640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-11-05T08:31:52.173 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-05T08:31:52.167+0000 7f0e1700a640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-11-05T08:31:52.173 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-05T08:31:52.167+0000 7f0e1700a640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-11-05T08:31:52.173 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-05T08:31:52.171+0000 7f0e1700a640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-11-05T08:31:55.173 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-11-05T08:31:55.230 DEBUG:teuthology.orchestra.run.smithi090:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 11 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-05T08:31:55.312 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-05T08:31:55.307+0000 7f2b0f79b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-11/keyring: can't open /var/lib/ceph/osd/ceph-11/keyring: (2) No such file or directory 2023-11-05T08:31:55.313 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-05T08:31:55.307+0000 7f2b0f79b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-11/keyring 2023-11-05T08:31:55.313 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-05T08:31:55.307+0000 7f2b0f79b640 -1 bluestore(/var/lib/ceph/osd/ceph-11/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-11/block: (2) No such file or directory 2023-11-05T08:31:55.313 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-05T08:31:55.307+0000 7f2b0f79b640 -1 bluestore(/var/lib/ceph/osd/ceph-11/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-11/block: (2) No such file or directory 2023-11-05T08:31:55.314 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-05T08:31:55.307+0000 7f2b0f79b640 -1 bluestore(/var/lib/ceph/osd/ceph-11/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-11/block: (2) No such file or directory 2023-11-05T08:31:55.314 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-05T08:31:55.311+0000 7f2b0f79b640 -1 bluestore(/var/lib/ceph/osd/ceph-11) _read_fsid unparsable uuid 2023-11-05T08:31:58.338 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-11 2023-11-05T08:31:58.397 DEBUG:teuthology.orchestra.run.smithi090:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 15 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-05T08:31:58.480 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-05T08:31:58.475+0000 7f37096b4640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-15/keyring: can't open /var/lib/ceph/osd/ceph-15/keyring: (2) No such file or directory 2023-11-05T08:31:58.481 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-05T08:31:58.479+0000 7f37096b4640 -1 created new key in keyring /var/lib/ceph/osd/ceph-15/keyring 2023-11-05T08:31:58.481 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-05T08:31:58.479+0000 7f37096b4640 -1 bluestore(/var/lib/ceph/osd/ceph-15/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-15/block: (2) No such file or directory 2023-11-05T08:31:58.481 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-05T08:31:58.479+0000 7f37096b4640 -1 bluestore(/var/lib/ceph/osd/ceph-15/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-15/block: (2) No such file or directory 2023-11-05T08:31:58.482 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-05T08:31:58.479+0000 7f37096b4640 -1 bluestore(/var/lib/ceph/osd/ceph-15/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-15/block: (2) No such file or directory 2023-11-05T08:31:58.482 INFO:teuthology.orchestra.run.smithi090.stderr:2023-11-05T08:31:58.479+0000 7f37096b4640 -1 bluestore(/var/lib/ceph/osd/ceph-15) _read_fsid unparsable uuid 2023-11-05T08:32:01.499 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-15 2023-11-05T08:32:01.553 INFO:tasks.ceph:Reading keys from all nodes... 2023-11-05T08:32:01.554 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-05T08:32:01.554 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-11-05T08:32:01.570 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-05T08:32:01.570 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-11-05T08:32:01.626 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-05T08:32:01.626 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-11-05T08:32:01.686 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-05T08:32:01.686 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-8/keyring of=/dev/stdout 2023-11-05T08:32:01.741 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-05T08:32:01.742 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-12/keyring of=/dev/stdout 2023-11-05T08:32:01.796 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-11-05T08:32:01.797 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-11-05T08:32:01.812 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-11-05T08:32:01.812 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-11-05T08:32:01.869 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-11-05T08:32:01.869 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-9/keyring of=/dev/stdout 2023-11-05T08:32:01.926 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-11-05T08:32:01.926 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-13/keyring of=/dev/stdout 2023-11-05T08:32:01.982 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-05T08:32:01.982 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-11-05T08:32:01.996 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-05T08:32:01.997 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-11-05T08:32:02.051 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-05T08:32:02.051 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-10/keyring of=/dev/stdout 2023-11-05T08:32:02.103 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-05T08:32:02.103 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-14/keyring of=/dev/stdout 2023-11-05T08:32:02.155 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-11-05T08:32:02.155 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-11-05T08:32:02.168 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-11-05T08:32:02.169 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-11-05T08:32:02.225 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-11-05T08:32:02.225 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-11-05T08:32:02.280 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-11-05T08:32:02.280 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-11/keyring of=/dev/stdout 2023-11-05T08:32:02.336 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-11-05T08:32:02.336 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-15/keyring of=/dev/stdout 2023-11-05T08:32:02.388 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-11-05T08:32:02.389 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-11-05T08:32:02.437 INFO:tasks.ceph:Adding keys to all mons... 2023-11-05T08:32:02.438 DEBUG:teuthology.orchestra.run.smithi002:> sudo tee -a /etc/ceph/ceph.keyring 2023-11-05T08:32:02.440 DEBUG:teuthology.orchestra.run.smithi053:> sudo tee -a /etc/ceph/ceph.keyring 2023-11-05T08:32:02.443 DEBUG:teuthology.orchestra.run.smithi069:> sudo tee -a /etc/ceph/ceph.keyring 2023-11-05T08:32:02.450 INFO:teuthology.orchestra.run.smithi002.stdout:[mgr.y] 2023-11-05T08:32:02.450 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDHUkdl6RLlGRAAg/D5rJ3UCQYhBiGGYHpfGg== 2023-11-05T08:32:02.451 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.0] 2023-11-05T08:32:02.451 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDIUkdlwmskKBAAHXHEJ6un5KWHLPyS64ONMA== 2023-11-05T08:32:02.451 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.4] 2023-11-05T08:32:02.451 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDLUkdlIvnVLhAA9A11EwPK855LzqZZU2oLcQ== 2023-11-05T08:32:02.451 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.8] 2023-11-05T08:32:02.452 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDOUkdlP+zTOBAAXctogkVUU9JkCtifFXpN/Q== 2023-11-05T08:32:02.452 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.12] 2023-11-05T08:32:02.452 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDSUkdloDmWBRAAB3xP9vGkvoFjU9HLWer7zQ== 2023-11-05T08:32:02.452 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.1] 2023-11-05T08:32:02.453 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDWUkdl5dA4DRAAR2jrSe4fN5iRqijZvktMDA== 2023-11-05T08:32:02.453 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.5] 2023-11-05T08:32:02.453 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDZUkdlW0sCFBAAdjPCjnYcdRo2nV38tgG7YA== 2023-11-05T08:32:02.453 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.9] 2023-11-05T08:32:02.453 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDcUkdly9JIHhAAtbVoTHcH+F4cr1eJEkL3gA== 2023-11-05T08:32:02.454 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.13] 2023-11-05T08:32:02.454 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDfUkdlgD2mJhAA5gKDC+GwPXT687UoBvHsYQ== 2023-11-05T08:32:02.454 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.2] 2023-11-05T08:32:02.454 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDlUkdlNjbRJxAADZsXptS4lDxT9ytsnQ5E4A== 2023-11-05T08:32:02.454 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.6] 2023-11-05T08:32:02.455 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDoUkdlVWjPLxAARJWJyzXWMzC6rvVJRMzgwQ== 2023-11-05T08:32:02.455 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.10] 2023-11-05T08:32:02.455 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDrUkdl4NABORAAP5KiTk54d+lHbIcoubZ63w== 2023-11-05T08:32:02.455 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.14] 2023-11-05T08:32:02.455 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDvUkdlX5CdBhAAorA6YFyQaA0jCdaE+slJyA== 2023-11-05T08:32:02.456 INFO:teuthology.orchestra.run.smithi002.stdout:[mgr.x] 2023-11-05T08:32:02.456 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDHUkdl8BKXHBAA/+9bj/CrHKI/vQbJifQl+Q== 2023-11-05T08:32:02.456 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.3] 2023-11-05T08:32:02.456 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQD1UkdlpbfgAxAA5ghsrm3bCS0DZgWUU0O/CQ== 2023-11-05T08:32:02.456 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.7] 2023-11-05T08:32:02.457 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQD4UkdloYNmChAAYrBLEcUDDJLO9ZA3vy6J3A== 2023-11-05T08:32:02.457 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.11] 2023-11-05T08:32:02.457 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQD7UkdlNlTBEhAAhhtZwaMxuJMDrlKHrZ9U5g== 2023-11-05T08:32:02.457 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.15] 2023-11-05T08:32:02.457 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQD+UkdlvknHHBAAtav74dHm+wgU3BV/dZh/rw== 2023-11-05T08:32:02.458 INFO:teuthology.orchestra.run.smithi002.stdout:[client.0] 2023-11-05T08:32:02.458 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDHUkdlhveEIxAAdhtgYT/OAdqLBmEP4+i/KA== 2023-11-05T08:32:02.460 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr.y] 2023-11-05T08:32:02.460 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQDHUkdl6RLlGRAAg/D5rJ3UCQYhBiGGYHpfGg== 2023-11-05T08:32:02.460 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.0] 2023-11-05T08:32:02.460 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQDIUkdlwmskKBAAHXHEJ6un5KWHLPyS64ONMA== 2023-11-05T08:32:02.461 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.4] 2023-11-05T08:32:02.461 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQDLUkdlIvnVLhAA9A11EwPK855LzqZZU2oLcQ== 2023-11-05T08:32:02.461 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.8] 2023-11-05T08:32:02.461 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQDOUkdlP+zTOBAAXctogkVUU9JkCtifFXpN/Q== 2023-11-05T08:32:02.461 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.12] 2023-11-05T08:32:02.461 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQDSUkdloDmWBRAAB3xP9vGkvoFjU9HLWer7zQ== 2023-11-05T08:32:02.462 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.1] 2023-11-05T08:32:02.462 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQDWUkdl5dA4DRAAR2jrSe4fN5iRqijZvktMDA== 2023-11-05T08:32:02.462 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.5] 2023-11-05T08:32:02.462 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQDZUkdlW0sCFBAAdjPCjnYcdRo2nV38tgG7YA== 2023-11-05T08:32:02.462 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.9] 2023-11-05T08:32:02.463 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQDcUkdly9JIHhAAtbVoTHcH+F4cr1eJEkL3gA== 2023-11-05T08:32:02.463 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.13] 2023-11-05T08:32:02.463 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQDfUkdlgD2mJhAA5gKDC+GwPXT687UoBvHsYQ== 2023-11-05T08:32:02.463 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.2] 2023-11-05T08:32:02.463 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQDlUkdlNjbRJxAADZsXptS4lDxT9ytsnQ5E4A== 2023-11-05T08:32:02.464 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.6] 2023-11-05T08:32:02.464 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQDoUkdlVWjPLxAARJWJyzXWMzC6rvVJRMzgwQ== 2023-11-05T08:32:02.464 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.10] 2023-11-05T08:32:02.464 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQDrUkdl4NABORAAP5KiTk54d+lHbIcoubZ63w== 2023-11-05T08:32:02.464 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.14] 2023-11-05T08:32:02.465 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQDvUkdlX5CdBhAAorA6YFyQaA0jCdaE+slJyA== 2023-11-05T08:32:02.465 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr.x] 2023-11-05T08:32:02.465 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQDHUkdl8BKXHBAA/+9bj/CrHKI/vQbJifQl+Q== 2023-11-05T08:32:02.465 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.3] 2023-11-05T08:32:02.465 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQD1UkdlpbfgAxAA5ghsrm3bCS0DZgWUU0O/CQ== 2023-11-05T08:32:02.466 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.7] 2023-11-05T08:32:02.466 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQD4UkdloYNmChAAYrBLEcUDDJLO9ZA3vy6J3A== 2023-11-05T08:32:02.466 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.11] 2023-11-05T08:32:02.466 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQD7UkdlNlTBEhAAhhtZwaMxuJMDrlKHrZ9U5g== 2023-11-05T08:32:02.466 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.15] 2023-11-05T08:32:02.466 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQD+UkdlvknHHBAAtav74dHm+wgU3BV/dZh/rw== 2023-11-05T08:32:02.467 INFO:teuthology.orchestra.run.smithi069.stdout:[client.0] 2023-11-05T08:32:02.467 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQDHUkdlhveEIxAAdhtgYT/OAdqLBmEP4+i/KA== 2023-11-05T08:32:02.467 INFO:teuthology.orchestra.run.smithi053.stdout:[mgr.y] 2023-11-05T08:32:02.467 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDHUkdl6RLlGRAAg/D5rJ3UCQYhBiGGYHpfGg== 2023-11-05T08:32:02.467 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.0] 2023-11-05T08:32:02.468 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDIUkdlwmskKBAAHXHEJ6un5KWHLPyS64ONMA== 2023-11-05T08:32:02.468 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.4] 2023-11-05T08:32:02.468 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDLUkdlIvnVLhAA9A11EwPK855LzqZZU2oLcQ== 2023-11-05T08:32:02.468 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.8] 2023-11-05T08:32:02.468 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDOUkdlP+zTOBAAXctogkVUU9JkCtifFXpN/Q== 2023-11-05T08:32:02.469 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.12] 2023-11-05T08:32:02.469 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDSUkdloDmWBRAAB3xP9vGkvoFjU9HLWer7zQ== 2023-11-05T08:32:02.469 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.1] 2023-11-05T08:32:02.469 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDWUkdl5dA4DRAAR2jrSe4fN5iRqijZvktMDA== 2023-11-05T08:32:02.470 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.5] 2023-11-05T08:32:02.470 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDZUkdlW0sCFBAAdjPCjnYcdRo2nV38tgG7YA== 2023-11-05T08:32:02.470 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.9] 2023-11-05T08:32:02.470 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDcUkdly9JIHhAAtbVoTHcH+F4cr1eJEkL3gA== 2023-11-05T08:32:02.470 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.13] 2023-11-05T08:32:02.470 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDfUkdlgD2mJhAA5gKDC+GwPXT687UoBvHsYQ== 2023-11-05T08:32:02.471 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.2] 2023-11-05T08:32:02.471 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDlUkdlNjbRJxAADZsXptS4lDxT9ytsnQ5E4A== 2023-11-05T08:32:02.471 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.6] 2023-11-05T08:32:02.471 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDoUkdlVWjPLxAARJWJyzXWMzC6rvVJRMzgwQ== 2023-11-05T08:32:02.471 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.10] 2023-11-05T08:32:02.472 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDrUkdl4NABORAAP5KiTk54d+lHbIcoubZ63w== 2023-11-05T08:32:02.472 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.14] 2023-11-05T08:32:02.472 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDvUkdlX5CdBhAAorA6YFyQaA0jCdaE+slJyA== 2023-11-05T08:32:02.472 INFO:teuthology.orchestra.run.smithi053.stdout:[mgr.x] 2023-11-05T08:32:02.473 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDHUkdl8BKXHBAA/+9bj/CrHKI/vQbJifQl+Q== 2023-11-05T08:32:02.473 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.3] 2023-11-05T08:32:02.473 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQD1UkdlpbfgAxAA5ghsrm3bCS0DZgWUU0O/CQ== 2023-11-05T08:32:02.473 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.7] 2023-11-05T08:32:02.473 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQD4UkdloYNmChAAYrBLEcUDDJLO9ZA3vy6J3A== 2023-11-05T08:32:02.474 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.11] 2023-11-05T08:32:02.474 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQD7UkdlNlTBEhAAhhtZwaMxuJMDrlKHrZ9U5g== 2023-11-05T08:32:02.474 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.15] 2023-11-05T08:32:02.474 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQD+UkdlvknHHBAAtav74dHm+wgU3BV/dZh/rw== 2023-11-05T08:32:02.475 INFO:teuthology.orchestra.run.smithi053.stdout:[client.0] 2023-11-05T08:32:02.475 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDHUkdlhveEIxAAdhtgYT/OAdqLBmEP4+i/KA== 2023-11-05T08:32:02.475 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-05T08:32:02.502 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-05T08:32:02.505 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-05T08:32:02.550 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:02.554 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:02.557 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:02.632 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:02.638 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:02.643 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:02.715 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:02.722 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:02.726 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:02.799 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:02.806 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:02.814 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:02.878 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:02.894 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:02.897 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:02.959 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:02.978 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:02.982 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:03.043 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:03.062 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:03.070 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:03.123 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:03.150 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:03.159 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:03.206 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:03.234 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:03.247 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:03.287 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:03.318 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:03.330 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:03.371 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:03.402 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:03.414 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:03.455 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:03.486 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:03.498 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:03.541 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-05T08:32:03.571 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-05T08:32:03.587 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-05T08:32:03.630 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:03.654 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:03.674 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:03.715 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:03.717 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:03.762 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:03.803 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:03.806 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:03.846 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:03.890 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:03.893 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:03.934 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-05T08:32:03.978 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-05T08:32:03.980 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-05T08:32:04.026 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-05T08:32:04.075 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-11-05T08:32:04.075 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-11-05T08:32:04.089 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-05T08:32:04.189 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-11-05T08:32:04.201 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-11-05T08:32:04.216 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-05T08:32:04.314 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-11-05T08:32:04.326 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-11-05T08:32:04.340 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-05T08:32:04.441 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-11-05T08:32:04.452 DEBUG:teuthology.orchestra.run.smithi002:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-11-05T08:32:04.454 DEBUG:teuthology.orchestra.run.smithi053:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-11-05T08:32:04.457 DEBUG:teuthology.orchestra.run.smithi069:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-11-05T08:32:04.500 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-11-05T08:32:04.500 INFO:tasks.ceph.mon.a:Restarting daemon 2023-11-05T08:32:04.501 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-11-05T08:32:04.503 INFO:tasks.ceph.mon.a:Started 2023-11-05T08:32:04.504 INFO:tasks.ceph.mon.b:Restarting daemon 2023-11-05T08:32:04.504 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-11-05T08:32:04.507 INFO:tasks.ceph.mon.b:Started 2023-11-05T08:32:04.507 INFO:tasks.ceph.mon.c:Restarting daemon 2023-11-05T08:32:04.507 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-11-05T08:32:04.545 INFO:tasks.ceph.mon.c:Started 2023-11-05T08:32:04.545 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-11-05T08:32:04.545 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-11-05T08:32:04.546 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-11-05T08:32:04.548 INFO:tasks.ceph.mgr.y:Started 2023-11-05T08:32:04.549 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-11-05T08:32:04.549 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-11-05T08:32:04.551 INFO:tasks.ceph.mgr.x:Started 2023-11-05T08:32:04.552 INFO:tasks.ceph:Setting crush tunables to default 2023-11-05T08:32:04.552 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph osd crush tunables default 2023-11-05T08:32:16.079 INFO:teuthology.orchestra.run.smithi002.stderr:adjusted tunables profile to default 2023-11-05T08:32:16.099 INFO:tasks.ceph:check_enable_crimson: False 2023-11-05T08:32:16.100 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-11-05T08:32:16.100 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-05T08:32:16.100 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-11-05T08:32:16.111 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-05T08:32:16.111 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-11-05T08:32:16.167 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-05T08:32:16.167 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-8/fsid of=/dev/stdout 2023-11-05T08:32:16.168 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-05T08:32:16.163+0000 7fd2d4590180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-05T08:32:16.198 INFO:tasks.ceph.mgr.y.smithi002.stderr:2023-11-05T08:32:16.197+0000 7fddc612b180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-05T08:32:16.210 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-05T08:32:16.211 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-12/fsid of=/dev/stdout 2023-11-05T08:32:16.268 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-11-05T08:32:16.268 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-11-05T08:32:16.284 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-11-05T08:32:16.284 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-11-05T08:32:16.338 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-11-05T08:32:16.338 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-9/fsid of=/dev/stdout 2023-11-05T08:32:16.393 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-11-05T08:32:16.394 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-13/fsid of=/dev/stdout 2023-11-05T08:32:16.449 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-05T08:32:16.449 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-11-05T08:32:16.464 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-05T08:32:16.464 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-11-05T08:32:16.520 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-05T08:32:16.520 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-10/fsid of=/dev/stdout 2023-11-05T08:32:16.575 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-11-05T08:32:16.575 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-14/fsid of=/dev/stdout 2023-11-05T08:32:16.627 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-11-05T08:32:16.627 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-11-05T08:32:16.640 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-11-05T08:32:16.640 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-11-05T08:32:16.692 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-11-05T08:32:16.692 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-11/fsid of=/dev/stdout 2023-11-05T08:32:16.700 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-05T08:32:16.695+0000 7fd2d4590180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-05T08:32:16.710 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-11-05T08:32:16.711 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-15/fsid of=/dev/stdout 2023-11-05T08:32:16.763 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd new 3626fa17-5bde-458d-96fb-f5e70ac6edd6 0 2023-11-05T08:32:16.772 INFO:tasks.ceph.mgr.y.smithi002.stderr:2023-11-05T08:32:16.769+0000 7fddc612b180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-05T08:32:16.894 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-05T08:32:16.891+0000 7fd2d4590180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-05T08:32:16.968 INFO:tasks.ceph.mgr.y.smithi002.stderr:2023-11-05T08:32:16.965+0000 7fddc612b180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-05T08:32:17.001 INFO:teuthology.orchestra.run.smithi090.stdout:0 2023-11-05T08:32:17.017 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd new 8f524504-91f5-44f3-afbd-69dfc19f03d3 1 2023-11-05T08:32:17.228 INFO:teuthology.orchestra.run.smithi090.stdout:1 2023-11-05T08:32:17.244 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd new 696f71d0-a98b-46b5-bdc0-0da3d6065469 2 2023-11-05T08:32:17.444 INFO:teuthology.orchestra.run.smithi090.stdout:2 2023-11-05T08:32:17.460 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd new fc979d30-659b-47f4-b06b-97a4d41c2e90 3 2023-11-05T08:32:17.482 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-05T08:32:17.479+0000 7fd2d4590180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-05T08:32:17.559 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-05T08:32:17.555+0000 7fd2d4590180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-05T08:32:17.579 INFO:tasks.ceph.mgr.y.smithi002.stderr:2023-11-05T08:32:17.577+0000 7fddc612b180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-05T08:32:17.615 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-05T08:32:17.611+0000 7fd2d4590180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-05T08:32:17.636 INFO:tasks.ceph.mgr.y.smithi002.stderr:2023-11-05T08:32:17.633+0000 7fddc612b180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-05T08:32:17.657 INFO:teuthology.orchestra.run.smithi090.stdout:3 2023-11-05T08:32:17.674 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd new 2ed87cb9-9ced-46fd-8346-3f47709bb69e 4 2023-11-05T08:32:17.691 INFO:tasks.ceph.mgr.y.smithi002.stderr:2023-11-05T08:32:17.689+0000 7fddc612b180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-05T08:32:17.693 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-05T08:32:17.691+0000 7fd2d4590180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-05T08:32:17.746 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-05T08:32:17.743+0000 7fd2d4590180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-05T08:32:17.772 INFO:tasks.ceph.mgr.y.smithi002.stderr:2023-11-05T08:32:17.769+0000 7fddc612b180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-05T08:32:17.825 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-05T08:32:17.823+0000 7fd2d4590180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-05T08:32:17.827 INFO:tasks.ceph.mgr.y.smithi002.stderr:2023-11-05T08:32:17.825+0000 7fddc612b180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-05T08:32:17.877 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-05T08:32:17.875+0000 7fd2d4590180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-05T08:32:17.888 INFO:teuthology.orchestra.run.smithi090.stdout:4 2023-11-05T08:32:17.904 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd new 5bb59d8d-9b93-48b2-a5ee-13646b19ce50 5 2023-11-05T08:32:17.909 INFO:tasks.ceph.mgr.y.smithi002.stderr:2023-11-05T08:32:17.905+0000 7fddc612b180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-05T08:32:17.963 INFO:tasks.ceph.mgr.y.smithi002.stderr:2023-11-05T08:32:17.961+0000 7fddc612b180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-05T08:32:18.020 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-05T08:32:18.015+0000 7fd2d4590180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-05T08:32:18.080 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-05T08:32:18.079+0000 7fd2d4590180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-05T08:32:18.110 INFO:tasks.ceph.mgr.y.smithi002.stderr:2023-11-05T08:32:18.109+0000 7fddc612b180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-05T08:32:18.112 INFO:teuthology.orchestra.run.smithi090.stdout:5 2023-11-05T08:32:18.129 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd new 46cd8f39-841d-4b7b-a4ef-6ec6493eea5c 6 2023-11-05T08:32:18.173 INFO:tasks.ceph.mgr.y.smithi002.stderr:2023-11-05T08:32:18.173+0000 7fddc612b180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-05T08:32:18.329 INFO:teuthology.orchestra.run.smithi090.stdout:6 2023-11-05T08:32:18.345 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd new c98d93d9-c1e5-4ac0-8f71-9fc1d2767963 7 2023-11-05T08:32:18.579 INFO:teuthology.orchestra.run.smithi090.stdout:7 2023-11-05T08:32:18.597 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd new fcb20ed4-d041-4ef8-bf17-724a43758e50 8 2023-11-05T08:32:18.749 INFO:tasks.ceph.mgr.y.smithi002.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-11-05T08:32:18.749 INFO:tasks.ceph.mgr.y.smithi002.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-11-05T08:32:18.750 INFO:tasks.ceph.mgr.y.smithi002.stderr: from numpy import show_config as show_numpy_config 2023-11-05T08:32:18.762 INFO:tasks.ceph.mgr.x.smithi090.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-11-05T08:32:18.763 INFO:tasks.ceph.mgr.x.smithi090.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-11-05T08:32:18.763 INFO:tasks.ceph.mgr.x.smithi090.stderr: from numpy import show_config as show_numpy_config 2023-11-05T08:32:18.791 INFO:teuthology.orchestra.run.smithi090.stdout:8 2023-11-05T08:32:18.793 INFO:tasks.ceph.mgr.y.smithi002.stderr:2023-11-05T08:32:18.789+0000 7fddc612b180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-05T08:32:18.807 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd new f85ae771-21d9-4974-a203-7ff48225c8a4 9 2023-11-05T08:32:18.819 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-05T08:32:18.815+0000 7fd2d4590180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-05T08:32:18.921 INFO:tasks.ceph.mgr.y.smithi002.stderr:2023-11-05T08:32:18.921+0000 7fddc612b180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-05T08:32:18.962 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-05T08:32:18.959+0000 7fd2d4590180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-05T08:32:18.994 INFO:teuthology.orchestra.run.smithi090.stdout:9 2023-11-05T08:32:19.011 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd new 496442a8-9bc1-4b66-a8bc-294488f6f089 10 2023-11-05T08:32:19.080 INFO:tasks.ceph.mgr.y.smithi002.stderr:2023-11-05T08:32:19.077+0000 7fddc612b180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-05T08:32:19.141 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-05T08:32:19.139+0000 7fd2d4590180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-05T08:32:19.145 INFO:tasks.ceph.mgr.y.smithi002.stderr:2023-11-05T08:32:19.145+0000 7fddc612b180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-05T08:32:19.213 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-05T08:32:19.211+0000 7fd2d4590180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-05T08:32:19.246 INFO:teuthology.orchestra.run.smithi090.stdout:10 2023-11-05T08:32:19.262 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd new 5680ea76-e74b-43e8-8933-39204c641f00 11 2023-11-05T08:32:19.276 INFO:tasks.ceph.mgr.y.smithi002.stderr:2023-11-05T08:32:19.273+0000 7fddc612b180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-05T08:32:19.340 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-05T08:32:19.335+0000 7fd2d4590180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-05T08:32:19.450 INFO:teuthology.orchestra.run.smithi090.stdout:11 2023-11-05T08:32:19.466 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd new 2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa 12 2023-11-05T08:32:19.651 INFO:teuthology.orchestra.run.smithi090.stdout:12 2023-11-05T08:32:19.669 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd new 356e7e3c-675d-4e78-ba4a-1d993093dc20 13 2023-11-05T08:32:19.867 INFO:teuthology.orchestra.run.smithi090.stdout:13 2023-11-05T08:32:19.884 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd new 1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f 14 2023-11-05T08:32:19.914 INFO:tasks.ceph.mgr.y.smithi002.stderr:2023-11-05T08:32:19.913+0000 7fddc612b180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-05T08:32:19.970 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-05T08:32:19.971+0000 7fd2d4590180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-05T08:32:19.993 INFO:tasks.ceph.mgr.y.smithi002.stderr:2023-11-05T08:32:19.989+0000 7fddc612b180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-05T08:32:20.051 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-05T08:32:20.051+0000 7fd2d4590180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-05T08:32:20.073 INFO:teuthology.orchestra.run.smithi090.stdout:14 2023-11-05T08:32:20.089 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd new 4a37e23e-e1ec-4444-b87b-a9897a448f2a 15 2023-11-05T08:32:20.172 INFO:tasks.ceph.mgr.y.smithi002.stderr:2023-11-05T08:32:20.169+0000 7fddc612b180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-05T08:32:20.226 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-05T08:32:20.227+0000 7fd2d4590180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-05T08:32:20.249 INFO:tasks.ceph.mgr.y.smithi002.stderr:2023-11-05T08:32:20.249+0000 7fddc612b180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-05T08:32:20.284 INFO:teuthology.orchestra.run.smithi090.stdout:15 2023-11-05T08:32:20.300 INFO:tasks.ceph.osd.0:Restarting daemon 2023-11-05T08:32:20.300 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-11-05T08:32:20.301 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-05T08:32:20.303+0000 7fd2d4590180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-05T08:32:20.303 INFO:tasks.ceph.osd.0:Started 2023-11-05T08:32:20.303 INFO:tasks.ceph.osd.4:Restarting daemon 2023-11-05T08:32:20.304 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-11-05T08:32:20.306 INFO:tasks.ceph.osd.4:Started 2023-11-05T08:32:20.308 INFO:tasks.ceph.osd.8:Restarting daemon 2023-11-05T08:32:20.308 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 8 2023-11-05T08:32:20.311 INFO:tasks.ceph.osd.8:Started 2023-11-05T08:32:20.311 INFO:tasks.ceph.osd.12:Restarting daemon 2023-11-05T08:32:20.312 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 12 2023-11-05T08:32:20.313 INFO:tasks.ceph.osd.12:Started 2023-11-05T08:32:20.314 INFO:tasks.ceph.osd.1:Restarting daemon 2023-11-05T08:32:20.314 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-11-05T08:32:20.318 INFO:tasks.ceph.osd.1:Started 2023-11-05T08:32:20.318 INFO:tasks.ceph.osd.5:Restarting daemon 2023-11-05T08:32:20.319 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-11-05T08:32:20.321 INFO:tasks.ceph.osd.5:Started 2023-11-05T08:32:20.322 INFO:tasks.ceph.osd.9:Restarting daemon 2023-11-05T08:32:20.322 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 9 2023-11-05T08:32:20.325 INFO:tasks.ceph.osd.9:Started 2023-11-05T08:32:20.325 INFO:tasks.ceph.osd.13:Restarting daemon 2023-11-05T08:32:20.325 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 13 2023-11-05T08:32:20.328 INFO:tasks.ceph.osd.13:Started 2023-11-05T08:32:20.328 INFO:tasks.ceph.osd.2:Restarting daemon 2023-11-05T08:32:20.329 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-11-05T08:32:20.332 INFO:tasks.ceph.osd.2:Started 2023-11-05T08:32:20.332 INFO:tasks.ceph.osd.6:Restarting daemon 2023-11-05T08:32:20.332 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-11-05T08:32:20.335 INFO:tasks.ceph.osd.6:Started 2023-11-05T08:32:20.335 INFO:tasks.ceph.osd.10:Restarting daemon 2023-11-05T08:32:20.336 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 10 2023-11-05T08:32:20.338 INFO:tasks.ceph.osd.10:Started 2023-11-05T08:32:20.339 INFO:tasks.ceph.osd.14:Restarting daemon 2023-11-05T08:32:20.339 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 14 2023-11-05T08:32:20.341 INFO:tasks.ceph.osd.14:Started 2023-11-05T08:32:20.342 INFO:tasks.ceph.osd.3:Restarting daemon 2023-11-05T08:32:20.342 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-11-05T08:32:20.345 INFO:tasks.ceph.osd.3:Started 2023-11-05T08:32:20.345 INFO:tasks.ceph.osd.7:Restarting daemon 2023-11-05T08:32:20.345 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-11-05T08:32:20.348 INFO:tasks.ceph.osd.7:Started 2023-11-05T08:32:20.348 INFO:tasks.ceph.osd.11:Restarting daemon 2023-11-05T08:32:20.348 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 11 2023-11-05T08:32:20.351 INFO:tasks.ceph.osd.11:Started 2023-11-05T08:32:20.351 INFO:tasks.ceph.osd.15:Restarting daemon 2023-11-05T08:32:20.351 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 15 2023-11-05T08:32:20.353 INFO:tasks.ceph.osd.15:Started 2023-11-05T08:32:20.354 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:32:20.549 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:32:20.550 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":18,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:32:20.284340+0000","last_up_change":"0.000000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:32:20.565 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-11-05T08:32:20.566 INFO:tasks.ceph:Waiting for OSDs to come up 2023-11-05T08:32:20.578 INFO:tasks.ceph.mgr.y.smithi002.stderr:2023-11-05T08:32:20.577+0000 7fddc612b180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-05T08:32:20.611 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-05T08:32:20.611+0000 7fd2d4590180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-05T08:32:20.641 INFO:tasks.ceph.mgr.y.smithi002.stderr:2023-11-05T08:32:20.637+0000 7fddc612b180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-05T08:32:20.669 INFO:tasks.ceph.mgr.x.smithi090.stderr:2023-11-05T08:32:20.671+0000 7fd2d4590180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-05T08:32:20.723 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:32:20.719+0000 7f1c70e27640 -1 Falling back to public interface 2023-11-05T08:32:20.915 INFO:tasks.ceph.osd.0.smithi002.stderr:2023-11-05T08:32:20.913+0000 7f89ee037640 -1 Falling back to public interface 2023-11-05T08:32:20.950 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-11-05T08:32:20.951+0000 7f96691e0640 -1 Falling back to public interface 2023-11-05T08:32:20.951 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:32:20.947+0000 7f429f98a640 -1 Falling back to public interface 2023-11-05T08:32:20.957 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:32:20.959+0000 7f0097115640 -1 Falling back to public interface 2023-11-05T08:32:20.958 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:32:20.955+0000 7f55c8690640 -1 Falling back to public interface 2023-11-05T08:32:20.959 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:32:20.955+0000 7fcd98856640 -1 Falling back to public interface 2023-11-05T08:32:20.961 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:32:20.962+0000 7f168d3f9640 -1 Falling back to public interface 2023-11-05T08:32:20.962 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:32:20.963+0000 7ffbd0385640 -1 Falling back to public interface 2023-11-05T08:32:20.973 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:32:20.975+0000 7fc111c7c640 -1 Falling back to public interface 2023-11-05T08:32:20.974 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:32:20.973+0000 7f8cbf4bf640 -1 Falling back to public interface 2023-11-05T08:32:20.977 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:32:20.978+0000 7f22653c1640 -1 Falling back to public interface 2023-11-05T08:32:20.977 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:32:20.978+0000 7ff0d82aa640 -1 Falling back to public interface 2023-11-05T08:32:20.978 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:32:20.978+0000 7fa109b10640 -1 Falling back to public interface 2023-11-05T08:32:20.994 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:32:20.993+0000 7fdc5fd9b640 -1 Falling back to public interface 2023-11-05T08:32:20.995 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:32:20.993+0000 7fda70047640 -1 Falling back to public interface 2023-11-05T08:32:21.689 INFO:tasks.ceph.mgr.y.smithi002.stderr:2023-11-05T08:32:21.685+0000 7fdd6d8a1640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-11-05T08:32:21.690 INFO:tasks.ceph.mgr.y.smithi002.stderr:2023-11-05T08:32:21.689+0000 7fdd6d8a1640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-11-05T08:32:22.180 DEBUG:teuthology.orchestra.run.smithi002:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-05T08:32:22.477 INFO:teuthology.misc.health.smithi002.stdout: 2023-11-05T08:32:22.477 INFO:teuthology.misc.health.smithi002.stdout:{"epoch":18,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:32:20.284340+0000","last_up_change":"0.000000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:32:22.494 DEBUG:teuthology.misc:0 of 16 OSDs are up 2023-11-05T08:32:24.508 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:32:24.502+0000 7f1c70e27640 -1 osd.9 0 log_to_monitors true 2023-11-05T08:32:24.513 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:32:24.514+0000 7ffbd0385640 -1 osd.11 0 log_to_monitors true 2023-11-05T08:32:24.519 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-11-05T08:32:24.518+0000 7f96691e0640 -1 osd.3 0 log_to_monitors true 2023-11-05T08:32:24.623 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:32:24.622+0000 7fc111c7c640 -1 osd.15 0 log_to_monitors true 2023-11-05T08:32:24.626 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:32:24.626+0000 7fa109b10640 -1 osd.6 0 log_to_monitors true 2023-11-05T08:32:24.640 INFO:tasks.ceph.osd.0.smithi002.stderr:2023-11-05T08:32:24.637+0000 7f89ee037640 -1 osd.0 0 log_to_monitors true 2023-11-05T08:32:24.716 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:32:24.710+0000 7f429f98a640 -1 osd.5 0 log_to_monitors true 2023-11-05T08:32:24.721 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:32:24.722+0000 7f168d3f9640 -1 osd.2 0 log_to_monitors true 2023-11-05T08:32:24.722 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:32:24.718+0000 7fcd98856640 -1 osd.13 0 log_to_monitors true 2023-11-05T08:32:24.726 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:32:24.726+0000 7f0097115640 -1 osd.7 0 log_to_monitors true 2023-11-05T08:32:24.799 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:32:24.797+0000 7f8cbf4bf640 -1 osd.4 0 log_to_monitors true 2023-11-05T08:32:24.815 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:32:24.813+0000 7fdc5fd9b640 -1 osd.8 0 log_to_monitors true 2023-11-05T08:32:24.815 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:32:24.814+0000 7ff0d82aa640 -1 osd.14 0 log_to_monitors true 2023-11-05T08:32:24.820 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:32:24.814+0000 7f55c8690640 -1 osd.1 0 log_to_monitors true 2023-11-05T08:32:24.828 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:32:24.825+0000 7fda70047640 -1 osd.12 0 log_to_monitors true 2023-11-05T08:32:24.913 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:32:24.914+0000 7f22653c1640 -1 osd.10 0 log_to_monitors true 2023-11-05T08:32:25.698 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:32:25.694+0000 7f1c6cdcf640 -1 osd.9 0 waiting for initial osdmap 2023-11-05T08:32:25.707 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:32:25.706+0000 7ffbcc32d640 -1 osd.11 0 waiting for initial osdmap 2023-11-05T08:32:25.708 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:32:25.706+0000 7fa105ab8640 -1 osd.6 0 waiting for initial osdmap 2023-11-05T08:32:25.709 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:32:25.706+0000 7fc10dc24640 -1 osd.15 0 waiting for initial osdmap 2023-11-05T08:32:25.711 INFO:tasks.ceph.osd.0.smithi002.stderr:2023-11-05T08:32:25.709+0000 7f89e9fdf640 -1 osd.0 0 waiting for initial osdmap 2023-11-05T08:32:25.716 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-11-05T08:32:25.714+0000 7f9665188640 -1 osd.3 0 waiting for initial osdmap 2023-11-05T08:32:25.741 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:32:25.742+0000 7fa1000f2640 -1 osd.6 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-05T08:32:25.742 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:32:25.738+0000 7f1c67c0a640 -1 osd.9 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-05T08:32:25.746 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:32:25.746+0000 7ffbc7168640 -1 osd.11 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-05T08:32:25.746 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:32:25.746+0000 7fc108a5f640 -1 osd.15 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-05T08:32:25.762 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-11-05T08:32:25.762+0000 7f965ffc3640 -1 osd.3 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-05T08:32:25.762 INFO:tasks.ceph.osd.0.smithi002.stderr:2023-11-05T08:32:25.761+0000 7f89e4e1a640 -1 osd.0 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-05T08:32:26.703 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:32:26.701+0000 7fda6bfef640 -1 osd.12 0 waiting for initial osdmap 2023-11-05T08:32:26.703 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:32:26.701+0000 7fdc5bd43640 -1 osd.8 0 waiting for initial osdmap 2023-11-05T08:32:26.704 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:32:26.698+0000 7f55c4638640 -1 osd.1 0 waiting for initial osdmap 2023-11-05T08:32:26.705 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:32:26.705+0000 7f16893a1640 -1 osd.2 0 waiting for initial osdmap 2023-11-05T08:32:26.705 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:32:26.698+0000 7fcd947fe640 -1 osd.13 0 waiting for initial osdmap 2023-11-05T08:32:26.707 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:32:26.705+0000 7f8cbb467640 -1 osd.4 0 waiting for initial osdmap 2023-11-05T08:32:26.737 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:32:26.737+0000 7f16841dc640 -1 osd.2 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-05T08:32:26.746 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:32:26.745+0000 7f8cb5aa1640 -1 osd.4 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-05T08:32:26.747 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:32:26.745+0000 7fdc5637d640 -1 osd.8 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-05T08:32:26.747 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:32:26.745+0000 7fda66629640 -1 osd.12 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-05T08:32:26.750 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:32:26.746+0000 7f55bec72640 -1 osd.1 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-05T08:32:26.750 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:32:26.746+0000 7fcd8ee38640 -1 osd.13 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-05T08:32:26.752 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:32:26.746+0000 7f429b932640 -1 osd.5 0 waiting for initial osdmap 2023-11-05T08:32:26.753 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:32:26.749+0000 7ff0d4252640 -1 osd.14 0 waiting for initial osdmap 2023-11-05T08:32:26.753 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:32:26.749+0000 7f2261369640 -1 osd.10 0 waiting for initial osdmap 2023-11-05T08:32:26.753 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:32:26.750+0000 7f00930bd640 -1 osd.7 0 waiting for initial osdmap 2023-11-05T08:32:26.782 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:32:26.778+0000 7f429676d640 -1 osd.5 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-05T08:32:26.789 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:32:26.789+0000 7f225b9a3640 -1 osd.10 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-05T08:32:26.793 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:32:26.793+0000 7ff0ce88c640 -1 osd.14 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-05T08:32:26.798 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:32:26.798+0000 7f008d6f7640 -1 osd.7 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-05T08:32:30.110 DEBUG:teuthology.orchestra.run.smithi002:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-05T08:32:30.419 INFO:teuthology.misc.health.smithi002.stdout: 2023-11-05T08:32:30.420 INFO:teuthology.misc.health.smithi002.stdout:{"epoch":24,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:32:29.755550+0000","last_up_change":"2023-11-05T08:32:27.693099+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists","up"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":3243652226},{"type":"v1","addr":"172.21.15.90:6801","nonce":3243652226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":3243652226},{"type":"v1","addr":"172.21.15.90:6803","nonce":3243652226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":3243652226},{"type":"v1","addr":"172.21.15.90:6807","nonce":3243652226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":3243652226},{"type":"v1","addr":"172.21.15.90:6805","nonce":3243652226}]},"public_addr":"172.21.15.90:6801/3243652226","cluster_addr":"172.21.15.90:6803/3243652226","heartbeat_back_addr":"172.21.15.90:6807/3243652226","heartbeat_front_addr":"172.21.15.90:6805/3243652226","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.814871+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.716676+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.473986+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.759750+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.709948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.654567+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.680121+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.823011+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.506760+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.887003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.501879+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.829603+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.688663+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.836642+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.664931+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:32:30.439 DEBUG:teuthology.misc:16 of 16 OSDs are up 2023-11-05T08:32:30.439 INFO:tasks.ceph:Creating RBD pool 2023-11-05T08:32:30.440 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph osd pool create rbd 8 2023-11-05T08:32:30.922 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'rbd' created 2023-11-05T08:32:30.965 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-11-05T08:32:31.925 INFO:teuthology.orchestra.run.smithi002.stderr:enabled application 'rbd' on pool 'rbd' 2023-11-05T08:32:31.954 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-11-05T08:32:31.955 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-11-05T08:32:31.955 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-11-05T08:32:31.956 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-11-05T08:32:31.956 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-11-05T08:32:32.308 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:32:32.327 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":5,"active_gid":4107,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":1912842100},{"type":"v1","addr":"172.21.15.2:6801","nonce":1912842100}]},"active_addr":"172.21.15.2:6801/1912842100","active_change":"2023-11-05T08:32:20.665875+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4101,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":310812524}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":2732825876}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":931228882}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":3972885323}]}]} 2023-11-05T08:32:32.332 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-11-05T08:32:32.332 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-11-05T08:32:32.333 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:32:32.630 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:32:32.631 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":26,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:32:31.923824+0000","last_up_change":"2023-11-05T08:32:27.693099+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists","up"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":3243652226},{"type":"v1","addr":"172.21.15.90:6801","nonce":3243652226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":3243652226},{"type":"v1","addr":"172.21.15.90:6803","nonce":3243652226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":3243652226},{"type":"v1","addr":"172.21.15.90:6807","nonce":3243652226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":3243652226},{"type":"v1","addr":"172.21.15.90:6805","nonce":3243652226}]},"public_addr":"172.21.15.90:6801/3243652226","cluster_addr":"172.21.15.90:6803/3243652226","heartbeat_back_addr":"172.21.15.90:6807/3243652226","heartbeat_front_addr":"172.21.15.90:6805/3243652226","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.814871+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.716676+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.473986+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.759750+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.709948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.654567+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.680121+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.823011+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.506760+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.887003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.501879+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.829603+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.688663+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.836642+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.664931+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:32:32.649 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-11-05T08:32:32.650 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:32:32.950 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:32:32.951 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":27,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:32:32.927566+0000","last_up_change":"2023-11-05T08:32:27.693099+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists","up"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":3243652226},{"type":"v1","addr":"172.21.15.90:6801","nonce":3243652226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":3243652226},{"type":"v1","addr":"172.21.15.90:6803","nonce":3243652226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":3243652226},{"type":"v1","addr":"172.21.15.90:6807","nonce":3243652226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":3243652226},{"type":"v1","addr":"172.21.15.90:6805","nonce":3243652226}]},"public_addr":"172.21.15.90:6801/3243652226","cluster_addr":"172.21.15.90:6803/3243652226","heartbeat_back_addr":"172.21.15.90:6807/3243652226","heartbeat_front_addr":"172.21.15.90:6805/3243652226","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.814871+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.716676+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.473986+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.759750+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.709948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.654567+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.680121+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.823011+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.506760+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.887003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.501879+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.829603+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.688663+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.836642+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.664931+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:32:32.970 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-11-05T08:32:32.971 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-11-05T08:32:32.973 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-11-05T08:32:32.974 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-11-05T08:32:32.974 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-11-05T08:32:32.975 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-11-05T08:32:32.975 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-11-05T08:32:32.976 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-11-05T08:32:32.976 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2023-11-05T08:32:32.977 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2023-11-05T08:32:32.978 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2023-11-05T08:32:32.978 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2023-11-05T08:32:32.979 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 flush_pg_stats 2023-11-05T08:32:32.979 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 flush_pg_stats 2023-11-05T08:32:32.980 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 flush_pg_stats 2023-11-05T08:32:32.981 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 flush_pg_stats 2023-11-05T08:32:33.374 INFO:teuthology.orchestra.run.smithi002.stdout:94489280515 2023-11-05T08:32:33.375 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-11-05T08:32:33.448 INFO:teuthology.orchestra.run.smithi002.stdout:90194313219 2023-11-05T08:32:33.448 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-11-05T08:32:33.470 INFO:teuthology.orchestra.run.smithi002.stdout:90194313220 2023-11-05T08:32:33.471 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2023-11-05T08:32:33.473 INFO:teuthology.orchestra.run.smithi002.stdout:94489280515 2023-11-05T08:32:33.474 INFO:teuthology.orchestra.run.smithi002.stdout:94489280515 2023-11-05T08:32:33.474 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-11-05T08:32:33.475 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2023-11-05T08:32:33.482 INFO:teuthology.orchestra.run.smithi002.stdout:90194313220 2023-11-05T08:32:33.482 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-05T08:32:33.499 INFO:teuthology.orchestra.run.smithi002.stdout:90194313220 2023-11-05T08:32:33.499 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-11-05T08:32:33.512 INFO:teuthology.orchestra.run.smithi002.stdout:94489280515 2023-11-05T08:32:33.512 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2023-11-05T08:32:33.514 INFO:teuthology.orchestra.run.smithi002.stdout:90194313220 2023-11-05T08:32:33.514 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2023-11-05T08:32:33.528 INFO:teuthology.orchestra.run.smithi002.stdout:94489280515 2023-11-05T08:32:33.529 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2023-11-05T08:32:33.534 INFO:teuthology.orchestra.run.smithi002.stdout:94489280515 2023-11-05T08:32:33.535 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2023-11-05T08:32:33.543 INFO:teuthology.orchestra.run.smithi002.stdout:94489280515 2023-11-05T08:32:33.544 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2023-11-05T08:32:33.549 INFO:teuthology.orchestra.run.smithi002.stdout:94489280516 2023-11-05T08:32:33.550 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-11-05T08:32:33.605 INFO:teuthology.orchestra.run.smithi002.stdout:90194313219 2023-11-05T08:32:33.605 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2023-11-05T08:32:33.613 INFO:teuthology.orchestra.run.smithi002.stdout:94489280515 2023-11-05T08:32:33.614 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-11-05T08:32:33.617 INFO:teuthology.orchestra.run.smithi002.stdout:94489280515 2023-11-05T08:32:33.617 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-11-05T08:32:34.203 INFO:teuthology.orchestra.run.smithi002.stdout:94489280513 2023-11-05T08:32:34.241 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280513 for osd.1 2023-11-05T08:32:34.309 INFO:teuthology.orchestra.run.smithi002.stdout:94489280513 2023-11-05T08:32:34.318 INFO:teuthology.orchestra.run.smithi002.stdout:94489280514 2023-11-05T08:32:34.350 INFO:teuthology.orchestra.run.smithi002.stdout:90194313219 2023-11-05T08:32:34.362 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280513 for osd.2 2023-11-05T08:32:34.366 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280514 for osd.13 2023-11-05T08:32:34.405 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313219 for osd.0 2023-11-05T08:32:34.411 INFO:teuthology.orchestra.run.smithi002.stdout:90194313218 2023-11-05T08:32:34.465 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313218 for osd.15 2023-11-05T08:32:34.502 INFO:teuthology.orchestra.run.smithi002.stdout:90194313218 2023-11-05T08:32:34.521 INFO:teuthology.orchestra.run.smithi002.stdout:94489280513 2023-11-05T08:32:34.550 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313219 got 90194313218 for osd.6 2023-11-05T08:32:34.555 INFO:teuthology.orchestra.run.smithi002.stdout:94489280513 2023-11-05T08:32:34.564 INFO:teuthology.orchestra.run.smithi002.stdout:94489280514 2023-11-05T08:32:34.575 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280513 for osd.12 2023-11-05T08:32:34.575 INFO:teuthology.orchestra.run.smithi002.stdout:90194313219 2023-11-05T08:32:34.587 INFO:teuthology.orchestra.run.smithi002.stdout:94489280513 2023-11-05T08:32:34.589 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280513 for osd.10 2023-11-05T08:32:34.605 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313219 for osd.3 2023-11-05T08:32:34.607 INFO:teuthology.orchestra.run.smithi002.stdout:90194313217 2023-11-05T08:32:34.612 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280514 for osd.7 2023-11-05T08:32:34.613 INFO:teuthology.orchestra.run.smithi002.stdout:94489280513 2023-11-05T08:32:34.631 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313219 got 90194313217 for osd.9 2023-11-05T08:32:34.631 INFO:teuthology.orchestra.run.smithi002.stdout:94489280513 2023-11-05T08:32:34.632 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280513 for osd.14 2023-11-05T08:32:34.635 INFO:teuthology.orchestra.run.smithi002.stdout:90194313218 2023-11-05T08:32:34.645 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280513 for osd.8 2023-11-05T08:32:34.649 INFO:teuthology.orchestra.run.smithi002.stdout:94489280513 2023-11-05T08:32:34.652 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280513 for osd.5 2023-11-05T08:32:34.656 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313218 for osd.11 2023-11-05T08:32:34.666 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280513 for osd.4 2023-11-05T08:32:35.242 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-11-05T08:32:35.363 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-11-05T08:32:35.368 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2023-11-05T08:32:35.406 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-05T08:32:35.466 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2023-11-05T08:32:35.551 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-11-05T08:32:35.567 INFO:teuthology.orchestra.run.smithi002.stdout:94489280513 2023-11-05T08:32:35.576 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2023-11-05T08:32:35.589 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2023-11-05T08:32:35.590 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280513 for osd.1 2023-11-05T08:32:35.606 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-11-05T08:32:35.614 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-11-05T08:32:35.632 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2023-11-05T08:32:35.633 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2023-11-05T08:32:35.646 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2023-11-05T08:32:35.652 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-11-05T08:32:35.657 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2023-11-05T08:32:35.667 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-11-05T08:32:35.752 INFO:teuthology.orchestra.run.smithi002.stdout:94489280515 2023-11-05T08:32:35.820 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.2 2023-11-05T08:32:35.820 DEBUG:teuthology.parallel:result is None 2023-11-05T08:32:36.043 INFO:teuthology.orchestra.run.smithi002.stdout:94489280516 2023-11-05T08:32:36.053 INFO:teuthology.orchestra.run.smithi002.stdout:90194313220 2023-11-05T08:32:36.107 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313220 for osd.0 2023-11-05T08:32:36.108 DEBUG:teuthology.parallel:result is None 2023-11-05T08:32:36.110 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280516 for osd.13 2023-11-05T08:32:36.110 DEBUG:teuthology.parallel:result is None 2023-11-05T08:32:36.207 INFO:teuthology.orchestra.run.smithi002.stdout:90194313220 2023-11-05T08:32:36.249 INFO:teuthology.orchestra.run.smithi002.stdout:94489280515 2023-11-05T08:32:36.269 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313220 for osd.15 2023-11-05T08:32:36.269 DEBUG:teuthology.parallel:result is None 2023-11-05T08:32:36.308 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.12 2023-11-05T08:32:36.308 DEBUG:teuthology.parallel:result is None 2023-11-05T08:32:36.346 INFO:teuthology.orchestra.run.smithi002.stdout:94489280516 2023-11-05T08:32:36.363 INFO:teuthology.orchestra.run.smithi002.stdout:90194313219 2023-11-05T08:32:36.384 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.7 2023-11-05T08:32:36.385 DEBUG:teuthology.parallel:result is None 2023-11-05T08:32:36.406 INFO:teuthology.orchestra.run.smithi002.stdout:90194313219 2023-11-05T08:32:36.408 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313219 got 90194313219 for osd.9 2023-11-05T08:32:36.408 INFO:teuthology.orchestra.run.smithi002.stdout:90194313220 2023-11-05T08:32:36.409 DEBUG:teuthology.parallel:result is None 2023-11-05T08:32:36.435 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313219 got 90194313219 for osd.6 2023-11-05T08:32:36.435 DEBUG:teuthology.parallel:result is None 2023-11-05T08:32:36.440 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313220 for osd.3 2023-11-05T08:32:36.440 DEBUG:teuthology.parallel:result is None 2023-11-05T08:32:36.443 INFO:teuthology.orchestra.run.smithi002.stdout:94489280515 2023-11-05T08:32:36.448 INFO:teuthology.orchestra.run.smithi002.stdout:94489280515 2023-11-05T08:32:36.451 INFO:teuthology.orchestra.run.smithi002.stdout:94489280515 2023-11-05T08:32:36.469 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.10 2023-11-05T08:32:36.470 DEBUG:teuthology.parallel:result is None 2023-11-05T08:32:36.472 INFO:teuthology.orchestra.run.smithi002.stdout:94489280515 2023-11-05T08:32:36.473 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.5 2023-11-05T08:32:36.474 DEBUG:teuthology.parallel:result is None 2023-11-05T08:32:36.480 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.4 2023-11-05T08:32:36.480 DEBUG:teuthology.parallel:result is None 2023-11-05T08:32:36.490 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.8 2023-11-05T08:32:36.490 DEBUG:teuthology.parallel:result is None 2023-11-05T08:32:36.507 INFO:teuthology.orchestra.run.smithi002.stdout:94489280515 2023-11-05T08:32:36.524 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.14 2023-11-05T08:32:36.524 DEBUG:teuthology.parallel:result is None 2023-11-05T08:32:36.533 INFO:teuthology.orchestra.run.smithi002.stdout:90194313220 2023-11-05T08:32:36.549 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313220 for osd.11 2023-11-05T08:32:36.549 DEBUG:teuthology.parallel:result is None 2023-11-05T08:32:36.591 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-11-05T08:32:36.868 INFO:teuthology.orchestra.run.smithi002.stdout:94489280515 2023-11-05T08:32:36.885 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.1 2023-11-05T08:32:36.886 DEBUG:teuthology.parallel:result is None 2023-11-05T08:32:36.886 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-05T08:32:36.886 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-05T08:32:37.164 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:32:37.165 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-11-05T08:32:37.182 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2023-11-05T08:32:36.674865+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":436440,"kb_used_data":3988,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509513000,"statfs":{"total":1546188226560,"available":1545741312000,"internally_reserved":0,"allocated":4083712,"data_stored":2709326,"data_compressed":149306,"data_compressed_allocated":393216,"data_compressed_original":1679360,"omap_allocated":25505,"internal_metadata":442473567},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000444"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-05T08:32:31.943951+0000","last_change":"2023-11-05T08:32:31.942607+0000","last_active":"2023-11-05T08:32:31.943951+0000","last_peered":"2023-11-05T08:32:31.943951+0000","last_clean":"2023-11-05T08:32:31.943951+0000","last_became_active":"2023-11-05T08:32:31.941858+0000","last_became_peered":"2023-11-05T08:32:31.941858+0000","last_unstale":"2023-11-05T08:32:31.943951+0000","last_undegraded":"2023-11-05T08:32:31.943951+0000","last_fullsized":"2023-11-05T08:32:31.943951+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:57.380113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-05T08:32:31.942539+0000","last_change":"2023-11-05T08:32:31.941613+0000","last_active":"2023-11-05T08:32:31.942539+0000","last_peered":"2023-11-05T08:32:31.942539+0000","last_clean":"2023-11-05T08:32:31.942539+0000","last_became_active":"2023-11-05T08:32:31.941207+0000","last_became_peered":"2023-11-05T08:32:31.941207+0000","last_unstale":"2023-11-05T08:32:31.942539+0000","last_undegraded":"2023-11-05T08:32:31.942539+0000","last_fullsized":"2023-11-05T08:32:31.942539+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:56.890103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-05T08:32:31.940904+0000","last_change":"2023-11-05T08:32:31.940904+0000","last_active":"2023-11-05T08:32:31.940904+0000","last_peered":"2023-11-05T08:32:31.940904+0000","last_clean":"2023-11-05T08:32:31.940904+0000","last_became_active":"2023-11-05T08:32:31.940312+0000","last_became_peered":"2023-11-05T08:32:31.940312+0000","last_unstale":"2023-11-05T08:32:31.940904+0000","last_undegraded":"2023-11-05T08:32:31.940904+0000","last_fullsized":"2023-11-05T08:32:31.940904+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:49.777374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-05T08:32:32.255335+0000","last_change":"2023-11-05T08:32:32.253735+0000","last_active":"2023-11-05T08:32:32.255335+0000","last_peered":"2023-11-05T08:32:32.255335+0000","last_clean":"2023-11-05T08:32:32.255335+0000","last_became_active":"2023-11-05T08:32:32.252836+0000","last_became_peered":"2023-11-05T08:32:32.252836+0000","last_unstale":"2023-11-05T08:32:32.255335+0000","last_undegraded":"2023-11-05T08:32:32.255335+0000","last_fullsized":"2023-11-05T08:32:32.255335+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:34.415367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-05T08:32:32.939559+0000","last_change":"2023-11-05T08:32:32.238846+0000","last_active":"2023-11-05T08:32:32.939559+0000","last_peered":"2023-11-05T08:32:32.939559+0000","last_clean":"2023-11-05T08:32:32.939559+0000","last_became_active":"2023-11-05T08:32:32.238097+0000","last_became_peered":"2023-11-05T08:32:32.238097+0000","last_unstale":"2023-11-05T08:32:32.939559+0000","last_undegraded":"2023-11-05T08:32:32.939559+0000","last_fullsized":"2023-11-05T08:32:32.939559+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:34.588917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-05T08:32:32.940322+0000","last_change":"2023-11-05T08:32:31.941357+0000","last_active":"2023-11-05T08:32:32.940322+0000","last_peered":"2023-11-05T08:32:32.940322+0000","last_clean":"2023-11-05T08:32:32.940322+0000","last_became_active":"2023-11-05T08:32:31.940564+0000","last_became_peered":"2023-11-05T08:32:31.940564+0000","last_unstale":"2023-11-05T08:32:32.940322+0000","last_undegraded":"2023-11-05T08:32:32.940322+0000","last_fullsized":"2023-11-05T08:32:32.940322+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:58.168161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-05T08:32:32.939591+0000","last_change":"2023-11-05T08:32:31.941142+0000","last_active":"2023-11-05T08:32:32.939591+0000","last_peered":"2023-11-05T08:32:32.939591+0000","last_clean":"2023-11-05T08:32:32.939591+0000","last_became_active":"2023-11-05T08:32:31.940432+0000","last_became_peered":"2023-11-05T08:32:31.940432+0000","last_unstale":"2023-11-05T08:32:32.939591+0000","last_undegraded":"2023-11-05T08:32:32.939591+0000","last_fullsized":"2023-11-05T08:32:32.939591+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:53.518597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-05T08:32:32.940490+0000","last_change":"2023-11-05T08:32:31.940561+0000","last_active":"2023-11-05T08:32:32.940490+0000","last_peered":"2023-11-05T08:32:32.940490+0000","last_clean":"2023-11-05T08:32:32.940490+0000","last_became_active":"2023-11-05T08:32:31.939705+0000","last_became_peered":"2023-11-05T08:32:31.939705+0000","last_unstale":"2023-11-05T08:32:32.940490+0000","last_undegraded":"2023-11-05T08:32:32.940490+0000","last_fullsized":"2023-11-05T08:32:32.940490+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:54.088838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"25'157","reported_seq":210,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-05T08:32:31.935616+0000","last_change":"2023-11-05T08:32:28.761556+0000","last_active":"2023-11-05T08:32:31.935616+0000","last_peered":"2023-11-05T08:32:31.935616+0000","last_clean":"2023-11-05T08:32:31.935616+0000","last_became_active":"2023-11-05T08:32:28.758834+0000","last_became_peered":"2023-11-05T08:32:28.758834+0000","last_unstale":"2023-11-05T08:32:31.935616+0000","last_undegraded":"2023-11-05T08:32:31.935616+0000","last_fullsized":"2023-11-05T08:32:31.935616+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:27.693099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:27.693099+0000","last_clean_scrub_stamp":"2023-11-05T08:32:27.693099+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:40.584365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":13898,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":303104,"data_stored":632328,"data_compressed":15412,"data_compressed_allocated":53248,"data_compressed_original":552960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.876}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.419}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93200000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2070000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1080000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.244}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.589}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.52}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7270000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.645}]}]},{"osd":15,"up_from":21,"seq":90194313220,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":245760,"data_stored":102814,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5920000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8540000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0030000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3049999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4239999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.508}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.589}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0289999999999999}]}]},{"osd":14,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":103400,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78500000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91700000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0569999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95499999999999996}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2490000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.292}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.212}]}]},{"osd":0,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":245760,"data_stored":102814,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,9,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.859}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4330000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2599999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3260000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.97}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.109}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8129999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.0939999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.766}]}]},{"osd":3,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":103400,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,2,4,5,6,7,8,9,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.365}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.52}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.901}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.583}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9900000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9279999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.3050000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.0299999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8919999999999999}]}]},{"osd":6,"up_from":21,"seq":90194313219,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":245760,"data_stored":102814,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[3,4,5,7,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78000000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.117}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0669999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96399999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1830000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5249999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.278}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.379}]}]},{"osd":1,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27264,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":249856,"data_stored":103400,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1279999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84399999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97099999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6110000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4580000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.708}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5630000000000002}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2320000000000002}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7250000000000001}]}]},{"osd":2,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27332,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":303104,"data_stored":632328,"data_compressed":15412,"data_compressed_allocated":53248,"data_compressed_original":552960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3759999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2210000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9019999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4259999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0880000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5960000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7969999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.976}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9009999999999998}]}]},{"osd":4,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":102814,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,3,5,6,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.127}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.196}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2789999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.667}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4870000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6739999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5209999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.98}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.897}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.427}]}]},{"osd":5,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":103400,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[4,6,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.597}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.52}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.657}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6970000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.885}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.488}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.714}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.806}]}]},{"osd":7,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":103400,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[4,5,6,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78200000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1879999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.139}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.389}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.931}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.282}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.653}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6829999999999998}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3570000000000002}]}]},{"osd":8,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":103400,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[5,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.046}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.167}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4399999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4650000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3160000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2749999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5269999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.226}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.946}]}]},{"osd":9,"up_from":21,"seq":90194313219,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":103400,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,3,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.052}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6240000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3700000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8010000000000002}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.171}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3939999999999999}]}]},{"osd":10,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":103400,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.135}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0369999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.768}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3859999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.95}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4450000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1190000000000002}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.274}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1800000000000002}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0550000000000002}]}]},{"osd":11,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":103400,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,3,5,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1830000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0109999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.115}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3360000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.228}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.389}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.599}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7589999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6479999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.796}]}]},{"osd":12,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":245760,"data_stored":102814,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,6,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5760000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1930000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6830000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5269999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2190000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84599999999999997}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.71}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.167}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6949,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6949,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-05T08:32:37.185 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-05T08:32:37.462 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:32:37.462 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-11-05T08:32:37.478 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2023-11-05T08:32:36.674865+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":436440,"kb_used_data":3988,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509513000,"statfs":{"total":1546188226560,"available":1545741312000,"internally_reserved":0,"allocated":4083712,"data_stored":2709326,"data_compressed":149306,"data_compressed_allocated":393216,"data_compressed_original":1679360,"omap_allocated":25505,"internal_metadata":442473567},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000444"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-05T08:32:31.943951+0000","last_change":"2023-11-05T08:32:31.942607+0000","last_active":"2023-11-05T08:32:31.943951+0000","last_peered":"2023-11-05T08:32:31.943951+0000","last_clean":"2023-11-05T08:32:31.943951+0000","last_became_active":"2023-11-05T08:32:31.941858+0000","last_became_peered":"2023-11-05T08:32:31.941858+0000","last_unstale":"2023-11-05T08:32:31.943951+0000","last_undegraded":"2023-11-05T08:32:31.943951+0000","last_fullsized":"2023-11-05T08:32:31.943951+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:57.380113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-05T08:32:31.942539+0000","last_change":"2023-11-05T08:32:31.941613+0000","last_active":"2023-11-05T08:32:31.942539+0000","last_peered":"2023-11-05T08:32:31.942539+0000","last_clean":"2023-11-05T08:32:31.942539+0000","last_became_active":"2023-11-05T08:32:31.941207+0000","last_became_peered":"2023-11-05T08:32:31.941207+0000","last_unstale":"2023-11-05T08:32:31.942539+0000","last_undegraded":"2023-11-05T08:32:31.942539+0000","last_fullsized":"2023-11-05T08:32:31.942539+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:56.890103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-05T08:32:31.940904+0000","last_change":"2023-11-05T08:32:31.940904+0000","last_active":"2023-11-05T08:32:31.940904+0000","last_peered":"2023-11-05T08:32:31.940904+0000","last_clean":"2023-11-05T08:32:31.940904+0000","last_became_active":"2023-11-05T08:32:31.940312+0000","last_became_peered":"2023-11-05T08:32:31.940312+0000","last_unstale":"2023-11-05T08:32:31.940904+0000","last_undegraded":"2023-11-05T08:32:31.940904+0000","last_fullsized":"2023-11-05T08:32:31.940904+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:49.777374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-05T08:32:32.255335+0000","last_change":"2023-11-05T08:32:32.253735+0000","last_active":"2023-11-05T08:32:32.255335+0000","last_peered":"2023-11-05T08:32:32.255335+0000","last_clean":"2023-11-05T08:32:32.255335+0000","last_became_active":"2023-11-05T08:32:32.252836+0000","last_became_peered":"2023-11-05T08:32:32.252836+0000","last_unstale":"2023-11-05T08:32:32.255335+0000","last_undegraded":"2023-11-05T08:32:32.255335+0000","last_fullsized":"2023-11-05T08:32:32.255335+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:34.415367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-05T08:32:32.939559+0000","last_change":"2023-11-05T08:32:32.238846+0000","last_active":"2023-11-05T08:32:32.939559+0000","last_peered":"2023-11-05T08:32:32.939559+0000","last_clean":"2023-11-05T08:32:32.939559+0000","last_became_active":"2023-11-05T08:32:32.238097+0000","last_became_peered":"2023-11-05T08:32:32.238097+0000","last_unstale":"2023-11-05T08:32:32.939559+0000","last_undegraded":"2023-11-05T08:32:32.939559+0000","last_fullsized":"2023-11-05T08:32:32.939559+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:34.588917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-05T08:32:32.940322+0000","last_change":"2023-11-05T08:32:31.941357+0000","last_active":"2023-11-05T08:32:32.940322+0000","last_peered":"2023-11-05T08:32:32.940322+0000","last_clean":"2023-11-05T08:32:32.940322+0000","last_became_active":"2023-11-05T08:32:31.940564+0000","last_became_peered":"2023-11-05T08:32:31.940564+0000","last_unstale":"2023-11-05T08:32:32.940322+0000","last_undegraded":"2023-11-05T08:32:32.940322+0000","last_fullsized":"2023-11-05T08:32:32.940322+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:58.168161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-05T08:32:32.939591+0000","last_change":"2023-11-05T08:32:31.941142+0000","last_active":"2023-11-05T08:32:32.939591+0000","last_peered":"2023-11-05T08:32:32.939591+0000","last_clean":"2023-11-05T08:32:32.939591+0000","last_became_active":"2023-11-05T08:32:31.940432+0000","last_became_peered":"2023-11-05T08:32:31.940432+0000","last_unstale":"2023-11-05T08:32:32.939591+0000","last_undegraded":"2023-11-05T08:32:32.939591+0000","last_fullsized":"2023-11-05T08:32:32.939591+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:53.518597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-05T08:32:32.940490+0000","last_change":"2023-11-05T08:32:31.940561+0000","last_active":"2023-11-05T08:32:32.940490+0000","last_peered":"2023-11-05T08:32:32.940490+0000","last_clean":"2023-11-05T08:32:32.940490+0000","last_became_active":"2023-11-05T08:32:31.939705+0000","last_became_peered":"2023-11-05T08:32:31.939705+0000","last_unstale":"2023-11-05T08:32:32.940490+0000","last_undegraded":"2023-11-05T08:32:32.940490+0000","last_fullsized":"2023-11-05T08:32:32.940490+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:54.088838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"25'157","reported_seq":210,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-05T08:32:31.935616+0000","last_change":"2023-11-05T08:32:28.761556+0000","last_active":"2023-11-05T08:32:31.935616+0000","last_peered":"2023-11-05T08:32:31.935616+0000","last_clean":"2023-11-05T08:32:31.935616+0000","last_became_active":"2023-11-05T08:32:28.758834+0000","last_became_peered":"2023-11-05T08:32:28.758834+0000","last_unstale":"2023-11-05T08:32:31.935616+0000","last_undegraded":"2023-11-05T08:32:31.935616+0000","last_fullsized":"2023-11-05T08:32:31.935616+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:27.693099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:27.693099+0000","last_clean_scrub_stamp":"2023-11-05T08:32:27.693099+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:40.584365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":13898,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":303104,"data_stored":632328,"data_compressed":15412,"data_compressed_allocated":53248,"data_compressed_original":552960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.876}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.419}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93200000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2070000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1080000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.244}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.589}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.52}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7270000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.645}]}]},{"osd":15,"up_from":21,"seq":90194313220,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":245760,"data_stored":102814,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5920000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8540000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0030000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3049999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4239999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.508}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.589}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0289999999999999}]}]},{"osd":14,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":103400,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78500000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91700000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0569999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95499999999999996}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2490000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.292}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.212}]}]},{"osd":0,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":245760,"data_stored":102814,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,9,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.859}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4330000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2599999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3260000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.97}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.109}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8129999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.0939999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.766}]}]},{"osd":3,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":103400,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,2,4,5,6,7,8,9,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.365}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.52}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.901}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.583}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9900000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9279999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.3050000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.0299999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8919999999999999}]}]},{"osd":6,"up_from":21,"seq":90194313219,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":245760,"data_stored":102814,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[3,4,5,7,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78000000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.117}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0669999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96399999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1830000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5249999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.278}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.379}]}]},{"osd":1,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27264,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":249856,"data_stored":103400,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1279999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84399999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97099999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6110000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4580000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.708}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5630000000000002}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2320000000000002}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7250000000000001}]}]},{"osd":2,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27332,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":303104,"data_stored":632328,"data_compressed":15412,"data_compressed_allocated":53248,"data_compressed_original":552960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3759999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2210000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9019999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4259999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0880000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5960000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7969999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.976}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9009999999999998}]}]},{"osd":4,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":102814,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,3,5,6,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.127}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.196}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2789999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.667}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4870000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6739999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5209999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.98}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.897}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.427}]}]},{"osd":5,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":103400,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[4,6,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.597}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.52}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.657}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6970000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.885}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.488}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.714}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.806}]}]},{"osd":7,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":103400,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[4,5,6,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78200000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1879999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.139}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.389}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.931}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.282}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.653}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6829999999999998}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3570000000000002}]}]},{"osd":8,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":103400,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[5,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.046}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.167}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4399999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4650000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3160000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2749999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5269999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.226}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.946}]}]},{"osd":9,"up_from":21,"seq":90194313219,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":103400,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,3,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.052}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6240000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3700000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8010000000000002}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.171}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3939999999999999}]}]},{"osd":10,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":103400,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.135}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0369999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.768}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3859999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.95}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4450000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1190000000000002}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.274}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1800000000000002}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0550000000000002}]}]},{"osd":11,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":103400,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,3,5,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1830000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0109999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.115}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3360000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.228}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.389}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.599}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7589999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6479999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.796}]}]},{"osd":12,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":245760,"data_stored":102814,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,6,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5760000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1930000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6830000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5269999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2190000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84599999999999997}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.71}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.167}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6949,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6949,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-05T08:32:37.481 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-11-05T08:32:37.481 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-11-05T08:32:37.481 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-11-05T08:32:37.482 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-05T08:32:37.791 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:32:37.792 INFO:teuthology.orchestra.run.smithi002.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-11-05T08:32:37.809 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-11-05T08:32:37.809 INFO:teuthology.run_tasks:Running task thrashosds... 2023-11-05T08:32:37.820 INFO:tasks.thrashosds:config is {'bluestore_new_sharding': 'random', 'chance_bluestore_reshard': 1, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2023-11-05T08:32:37.820 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'DBG_maintain_prefix_itr_2'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 5000, 'osd_async_recovery_min_cost': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'sha1': '420cd129397301b1521ceb74e1bdbd9b855d2189'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '420cd129397301b1521ceb74e1bdbd9b855d2189'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'DBG_maintain_prefix_itr_2', 'sha1': '420cd129397301b1521ceb74e1bdbd9b855d2189'}} 2023-11-05T08:32:37.821 INFO:tasks.thrashosds:config is {'bluestore_new_sharding': 'random', 'chance_bluestore_reshard': 1, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2023-11-05T08:32:37.821 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2023-11-05T08:32:38.133 INFO:tasks.thrashosds:Beginning thrashosds... 2023-11-05T08:32:38.133 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-05T08:32:38.133 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-05T08:32:38.417 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:32:38.417 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-11-05T08:32:38.433 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2023-11-05T08:32:36.674865+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":436440,"kb_used_data":3988,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509513000,"statfs":{"total":1546188226560,"available":1545741312000,"internally_reserved":0,"allocated":4083712,"data_stored":2709326,"data_compressed":149306,"data_compressed_allocated":393216,"data_compressed_original":1679360,"omap_allocated":25505,"internal_metadata":442473567},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000444"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-05T08:32:31.943951+0000","last_change":"2023-11-05T08:32:31.942607+0000","last_active":"2023-11-05T08:32:31.943951+0000","last_peered":"2023-11-05T08:32:31.943951+0000","last_clean":"2023-11-05T08:32:31.943951+0000","last_became_active":"2023-11-05T08:32:31.941858+0000","last_became_peered":"2023-11-05T08:32:31.941858+0000","last_unstale":"2023-11-05T08:32:31.943951+0000","last_undegraded":"2023-11-05T08:32:31.943951+0000","last_fullsized":"2023-11-05T08:32:31.943951+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:57.380113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-05T08:32:31.942539+0000","last_change":"2023-11-05T08:32:31.941613+0000","last_active":"2023-11-05T08:32:31.942539+0000","last_peered":"2023-11-05T08:32:31.942539+0000","last_clean":"2023-11-05T08:32:31.942539+0000","last_became_active":"2023-11-05T08:32:31.941207+0000","last_became_peered":"2023-11-05T08:32:31.941207+0000","last_unstale":"2023-11-05T08:32:31.942539+0000","last_undegraded":"2023-11-05T08:32:31.942539+0000","last_fullsized":"2023-11-05T08:32:31.942539+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:56.890103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-05T08:32:31.940904+0000","last_change":"2023-11-05T08:32:31.940904+0000","last_active":"2023-11-05T08:32:31.940904+0000","last_peered":"2023-11-05T08:32:31.940904+0000","last_clean":"2023-11-05T08:32:31.940904+0000","last_became_active":"2023-11-05T08:32:31.940312+0000","last_became_peered":"2023-11-05T08:32:31.940312+0000","last_unstale":"2023-11-05T08:32:31.940904+0000","last_undegraded":"2023-11-05T08:32:31.940904+0000","last_fullsized":"2023-11-05T08:32:31.940904+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:49.777374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-05T08:32:32.255335+0000","last_change":"2023-11-05T08:32:32.253735+0000","last_active":"2023-11-05T08:32:32.255335+0000","last_peered":"2023-11-05T08:32:32.255335+0000","last_clean":"2023-11-05T08:32:32.255335+0000","last_became_active":"2023-11-05T08:32:32.252836+0000","last_became_peered":"2023-11-05T08:32:32.252836+0000","last_unstale":"2023-11-05T08:32:32.255335+0000","last_undegraded":"2023-11-05T08:32:32.255335+0000","last_fullsized":"2023-11-05T08:32:32.255335+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:34.415367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-05T08:32:32.939559+0000","last_change":"2023-11-05T08:32:32.238846+0000","last_active":"2023-11-05T08:32:32.939559+0000","last_peered":"2023-11-05T08:32:32.939559+0000","last_clean":"2023-11-05T08:32:32.939559+0000","last_became_active":"2023-11-05T08:32:32.238097+0000","last_became_peered":"2023-11-05T08:32:32.238097+0000","last_unstale":"2023-11-05T08:32:32.939559+0000","last_undegraded":"2023-11-05T08:32:32.939559+0000","last_fullsized":"2023-11-05T08:32:32.939559+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:34.588917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-05T08:32:32.940322+0000","last_change":"2023-11-05T08:32:31.941357+0000","last_active":"2023-11-05T08:32:32.940322+0000","last_peered":"2023-11-05T08:32:32.940322+0000","last_clean":"2023-11-05T08:32:32.940322+0000","last_became_active":"2023-11-05T08:32:31.940564+0000","last_became_peered":"2023-11-05T08:32:31.940564+0000","last_unstale":"2023-11-05T08:32:32.940322+0000","last_undegraded":"2023-11-05T08:32:32.940322+0000","last_fullsized":"2023-11-05T08:32:32.940322+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:58.168161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-05T08:32:32.939591+0000","last_change":"2023-11-05T08:32:31.941142+0000","last_active":"2023-11-05T08:32:32.939591+0000","last_peered":"2023-11-05T08:32:32.939591+0000","last_clean":"2023-11-05T08:32:32.939591+0000","last_became_active":"2023-11-05T08:32:31.940432+0000","last_became_peered":"2023-11-05T08:32:31.940432+0000","last_unstale":"2023-11-05T08:32:32.939591+0000","last_undegraded":"2023-11-05T08:32:32.939591+0000","last_fullsized":"2023-11-05T08:32:32.939591+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:53.518597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-05T08:32:32.940490+0000","last_change":"2023-11-05T08:32:31.940561+0000","last_active":"2023-11-05T08:32:32.940490+0000","last_peered":"2023-11-05T08:32:32.940490+0000","last_clean":"2023-11-05T08:32:32.940490+0000","last_became_active":"2023-11-05T08:32:31.939705+0000","last_became_peered":"2023-11-05T08:32:31.939705+0000","last_unstale":"2023-11-05T08:32:32.940490+0000","last_undegraded":"2023-11-05T08:32:32.940490+0000","last_fullsized":"2023-11-05T08:32:32.940490+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:54.088838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"25'157","reported_seq":210,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-05T08:32:31.935616+0000","last_change":"2023-11-05T08:32:28.761556+0000","last_active":"2023-11-05T08:32:31.935616+0000","last_peered":"2023-11-05T08:32:31.935616+0000","last_clean":"2023-11-05T08:32:31.935616+0000","last_became_active":"2023-11-05T08:32:28.758834+0000","last_became_peered":"2023-11-05T08:32:28.758834+0000","last_unstale":"2023-11-05T08:32:31.935616+0000","last_undegraded":"2023-11-05T08:32:31.935616+0000","last_fullsized":"2023-11-05T08:32:31.935616+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:27.693099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:27.693099+0000","last_clean_scrub_stamp":"2023-11-05T08:32:27.693099+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:40.584365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":13898,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":303104,"data_stored":632328,"data_compressed":15412,"data_compressed_allocated":53248,"data_compressed_original":552960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.876}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.419}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93200000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2070000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1080000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.244}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.589}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.52}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7270000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.645}]}]},{"osd":15,"up_from":21,"seq":90194313220,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":245760,"data_stored":102814,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5920000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8540000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0030000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3049999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4239999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.508}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.589}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0289999999999999}]}]},{"osd":14,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":103400,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78500000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91700000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0569999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95499999999999996}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2490000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.292}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.212}]}]},{"osd":0,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":245760,"data_stored":102814,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,9,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.859}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4330000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2599999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3260000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.97}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.109}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8129999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.0939999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.766}]}]},{"osd":3,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":103400,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,2,4,5,6,7,8,9,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.365}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.52}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.901}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.583}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9900000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9279999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.3050000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.0299999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8919999999999999}]}]},{"osd":6,"up_from":21,"seq":90194313219,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":245760,"data_stored":102814,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[3,4,5,7,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78000000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.117}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0669999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96399999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1830000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5249999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.278}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.379}]}]},{"osd":1,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27264,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":249856,"data_stored":103400,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1279999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84399999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97099999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6110000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4580000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.708}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5630000000000002}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2320000000000002}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7250000000000001}]}]},{"osd":2,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27332,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":303104,"data_stored":632328,"data_compressed":15412,"data_compressed_allocated":53248,"data_compressed_original":552960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3759999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2210000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9019999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4259999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0880000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5960000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7969999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.976}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9009999999999998}]}]},{"osd":4,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":102814,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,3,5,6,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.127}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.196}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2789999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.667}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4870000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6739999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5209999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.98}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.897}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.427}]}]},{"osd":5,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":103400,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[4,6,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.597}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.52}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.657}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6970000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.885}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.488}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.714}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.806}]}]},{"osd":7,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":103400,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[4,5,6,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78200000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1879999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.139}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.389}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.931}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.282}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.653}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6829999999999998}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3570000000000002}]}]},{"osd":8,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":103400,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[5,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.046}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.167}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4399999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4650000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3160000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2749999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5269999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.226}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.946}]}]},{"osd":9,"up_from":21,"seq":90194313219,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":103400,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,3,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.052}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6240000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3700000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8010000000000002}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.171}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3939999999999999}]}]},{"osd":10,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":103400,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.135}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0369999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.768}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3859999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.95}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4450000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1190000000000002}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.274}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1800000000000002}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0550000000000002}]}]},{"osd":11,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":103400,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,3,5,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1830000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0109999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.115}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3360000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.228}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.389}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.599}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7589999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6479999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.796}]}]},{"osd":12,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":245760,"data_stored":102814,"data_compressed":8463,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,6,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5760000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1930000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6830000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5269999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2190000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84599999999999997}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.71}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.167}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6949,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6949,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-05T08:32:38.436 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-05T08:32:38.727 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:32:38.728 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-11-05T08:32:38.744 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-11-05T08:32:38.675290+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":436556,"kb_used_data":4168,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509512884,"statfs":{"total":1546188226560,"available":1545741193216,"internally_reserved":0,"allocated":4268032,"data_stored":2851256,"data_compressed":175151,"data_compressed_allocated":454656,"data_compressed_original":1802240,"omap_allocated":25505,"internal_metadata":442473567},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000869"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-05T08:32:33.832942+0000","last_change":"2023-11-05T08:32:31.942607+0000","last_active":"2023-11-05T08:32:33.832942+0000","last_peered":"2023-11-05T08:32:33.832942+0000","last_clean":"2023-11-05T08:32:33.832942+0000","last_became_active":"2023-11-05T08:32:31.941858+0000","last_became_peered":"2023-11-05T08:32:31.941858+0000","last_unstale":"2023-11-05T08:32:33.832942+0000","last_undegraded":"2023-11-05T08:32:33.832942+0000","last_fullsized":"2023-11-05T08:32:33.832942+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:48.710632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-05T08:32:34.625249+0000","last_change":"2023-11-05T08:32:31.941613+0000","last_active":"2023-11-05T08:32:34.625249+0000","last_peered":"2023-11-05T08:32:34.625249+0000","last_clean":"2023-11-05T08:32:34.625249+0000","last_became_active":"2023-11-05T08:32:31.941207+0000","last_became_peered":"2023-11-05T08:32:31.941207+0000","last_unstale":"2023-11-05T08:32:34.625249+0000","last_undegraded":"2023-11-05T08:32:34.625249+0000","last_fullsized":"2023-11-05T08:32:34.625249+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:44.932076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-05T08:32:34.626285+0000","last_change":"2023-11-05T08:32:31.940904+0000","last_active":"2023-11-05T08:32:34.626285+0000","last_peered":"2023-11-05T08:32:34.626285+0000","last_clean":"2023-11-05T08:32:34.626285+0000","last_became_active":"2023-11-05T08:32:31.940312+0000","last_became_peered":"2023-11-05T08:32:31.940312+0000","last_unstale":"2023-11-05T08:32:34.626285+0000","last_undegraded":"2023-11-05T08:32:34.626285+0000","last_fullsized":"2023-11-05T08:32:34.626285+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:43.223063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-05T08:32:34.508527+0000","last_change":"2023-11-05T08:32:32.253735+0000","last_active":"2023-11-05T08:32:34.508527+0000","last_peered":"2023-11-05T08:32:34.508527+0000","last_clean":"2023-11-05T08:32:34.508527+0000","last_became_active":"2023-11-05T08:32:32.252836+0000","last_became_peered":"2023-11-05T08:32:32.252836+0000","last_unstale":"2023-11-05T08:32:34.508527+0000","last_undegraded":"2023-11-05T08:32:34.508527+0000","last_fullsized":"2023-11-05T08:32:34.508527+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:57.123923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-05T08:32:32.939559+0000","last_change":"2023-11-05T08:32:32.238846+0000","last_active":"2023-11-05T08:32:32.939559+0000","last_peered":"2023-11-05T08:32:32.939559+0000","last_clean":"2023-11-05T08:32:32.939559+0000","last_became_active":"2023-11-05T08:32:32.238097+0000","last_became_peered":"2023-11-05T08:32:32.238097+0000","last_unstale":"2023-11-05T08:32:32.939559+0000","last_undegraded":"2023-11-05T08:32:32.939559+0000","last_fullsized":"2023-11-05T08:32:32.939559+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:34.588917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-05T08:32:32.940322+0000","last_change":"2023-11-05T08:32:31.941357+0000","last_active":"2023-11-05T08:32:32.940322+0000","last_peered":"2023-11-05T08:32:32.940322+0000","last_clean":"2023-11-05T08:32:32.940322+0000","last_became_active":"2023-11-05T08:32:31.940564+0000","last_became_peered":"2023-11-05T08:32:31.940564+0000","last_unstale":"2023-11-05T08:32:32.940322+0000","last_undegraded":"2023-11-05T08:32:32.940322+0000","last_fullsized":"2023-11-05T08:32:32.940322+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:58.168161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-05T08:32:32.939591+0000","last_change":"2023-11-05T08:32:31.941142+0000","last_active":"2023-11-05T08:32:32.939591+0000","last_peered":"2023-11-05T08:32:32.939591+0000","last_clean":"2023-11-05T08:32:32.939591+0000","last_became_active":"2023-11-05T08:32:31.940432+0000","last_became_peered":"2023-11-05T08:32:31.940432+0000","last_unstale":"2023-11-05T08:32:32.939591+0000","last_undegraded":"2023-11-05T08:32:32.939591+0000","last_fullsized":"2023-11-05T08:32:32.939591+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:53.518597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-05T08:32:32.940490+0000","last_change":"2023-11-05T08:32:31.940561+0000","last_active":"2023-11-05T08:32:32.940490+0000","last_peered":"2023-11-05T08:32:32.940490+0000","last_clean":"2023-11-05T08:32:32.940490+0000","last_became_active":"2023-11-05T08:32:31.939705+0000","last_became_peered":"2023-11-05T08:32:31.939705+0000","last_unstale":"2023-11-05T08:32:32.940490+0000","last_undegraded":"2023-11-05T08:32:32.940490+0000","last_fullsized":"2023-11-05T08:32:32.940490+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:54.088838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"25'157","reported_seq":210,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-05T08:32:31.935616+0000","last_change":"2023-11-05T08:32:28.761556+0000","last_active":"2023-11-05T08:32:31.935616+0000","last_peered":"2023-11-05T08:32:31.935616+0000","last_clean":"2023-11-05T08:32:31.935616+0000","last_became_active":"2023-11-05T08:32:28.758834+0000","last_became_peered":"2023-11-05T08:32:28.758834+0000","last_unstale":"2023-11-05T08:32:31.935616+0000","last_undegraded":"2023-11-05T08:32:31.935616+0000","last_fullsized":"2023-11-05T08:32:31.935616+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:27.693099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:27.693099+0000","last_clean_scrub_stamp":"2023-11-05T08:32:27.693099+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:40.584365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":13898,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":303104,"data_stored":632328,"data_compressed":15412,"data_compressed_allocated":53248,"data_compressed_original":552960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.876}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.419}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93200000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2070000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1080000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.244}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.589}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.52}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7270000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.645}]}]},{"osd":15,"up_from":21,"seq":90194313221,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":258048,"data_stored":112276,"data_compressed":10186,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.103}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93799999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.228}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3300000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.407}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.504}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5720000000000001}]}]},{"osd":14,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":112862,"data_compressed":10186,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92600000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.157}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.968}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.109}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.544}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2050000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313221,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":258048,"data_stored":112276,"data_compressed":10186,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,9,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5029999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5660000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96199999999999997}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]}]},{"osd":3,"up_from":21,"seq":90194313221,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":262144,"data_stored":112862,"data_compressed":10186,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,2,4,5,6,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1579999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.319}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98999999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9900000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4330000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.498}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5229999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5529999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3839999999999999}]}]},{"osd":6,"up_from":21,"seq":90194313220,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":258048,"data_stored":112276,"data_compressed":10186,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[3,4,5,7,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78000000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.117}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0669999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96399999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1830000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5249999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.278}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.379}]}]},{"osd":1,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":112862,"data_compressed":10186,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.032}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6799999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.304}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.401}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.556}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.732}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0270000000000001}]}]},{"osd":2,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":315392,"data_stored":641790,"data_compressed":17135,"data_compressed_allocated":57344,"data_compressed_original":561152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5529999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1850000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5029999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4590000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.591}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6180000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7709999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.823}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8620000000000001}]}]},{"osd":4,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":258048,"data_stored":112276,"data_compressed":10186,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,3,5,6,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.988}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0640000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95499999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.667}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69099999999999995}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1339999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1839999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.147}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0939999999999999}]}]},{"osd":5,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":112862,"data_compressed":10186,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[4,6,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.645}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5049999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.286}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.78}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.343}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9199999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.823}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.869}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.097}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8999999999999999}]}]},{"osd":7,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":112862,"data_compressed":10186,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[4,5,6,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.034}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.266}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5629999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.083}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6830000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.49}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.641}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7949999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.831}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8660000000000001}]}]},{"osd":8,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":112862,"data_compressed":10186,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[5,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0429999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72899999999999998}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2130000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1000000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0209999999999999}]}]},{"osd":9,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":112862,"data_compressed":10186,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,3,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1950000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81999999999999995}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.085}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.276}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.135}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3300000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4730000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5600000000000001}]}]},{"osd":10,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":112862,"data_compressed":10186,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.085}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90600000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.169}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.208}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6259999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.141}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4670000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.583}]}]},{"osd":11,"up_from":21,"seq":90194313221,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":112862,"data_compressed":10186,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,3,5,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1830000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0109999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.115}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3360000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.228}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.389}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.599}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7589999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6479999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.796}]}]},{"osd":12,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":258048,"data_stored":112276,"data_compressed":10186,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,6,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72599999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88300000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81899999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6949,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6949,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-05T08:32:38.747 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-11-05T08:32:38.747 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2023-11-05T08:32:39.048 INFO:teuthology.orchestra.run.smithi002.stdout:epoch 27 2023-11-05T08:32:39.049 INFO:teuthology.orchestra.run.smithi002.stdout:fsid 8f1fa9b5-06de-4da7-985f-30d4ad103e2d 2023-11-05T08:32:39.049 INFO:teuthology.orchestra.run.smithi002.stdout:created 2023-11-05T08:32:15.953286+0000 2023-11-05T08:32:39.049 INFO:teuthology.orchestra.run.smithi002.stdout:modified 2023-11-05T08:32:32.927566+0000 2023-11-05T08:32:39.050 INFO:teuthology.orchestra.run.smithi002.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2023-11-05T08:32:39.050 INFO:teuthology.orchestra.run.smithi002.stdout:crush_version 5 2023-11-05T08:32:39.050 INFO:teuthology.orchestra.run.smithi002.stdout:full_ratio 0.95 2023-11-05T08:32:39.050 INFO:teuthology.orchestra.run.smithi002.stdout:backfillfull_ratio 0.9 2023-11-05T08:32:39.050 INFO:teuthology.orchestra.run.smithi002.stdout:nearfull_ratio 0.85 2023-11-05T08:32:39.051 INFO:teuthology.orchestra.run.smithi002.stdout:require_min_compat_client luminous 2023-11-05T08:32:39.051 INFO:teuthology.orchestra.run.smithi002.stdout:min_compat_client jewel 2023-11-05T08:32:39.051 INFO:teuthology.orchestra.run.smithi002.stdout:require_osd_release reef 2023-11-05T08:32:39.051 INFO:teuthology.orchestra.run.smithi002.stdout:stretch_mode_enabled false 2023-11-05T08:32:39.052 INFO:teuthology.orchestra.run.smithi002.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 24 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 15.38 2023-11-05T08:32:39.052 INFO:teuthology.orchestra.run.smithi002.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 27 flags hashpspool stripe_width 0 application rbd read_balance_score 4.02 2023-11-05T08:32:39.052 INFO:teuthology.orchestra.run.smithi002.stdout:max_osd 16 2023-11-05T08:32:39.052 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.2:6802/1948090427,v1:172.21.15.2:6803/1948090427] [v2:172.21.15.2:6804/1948090427,v1:172.21.15.2:6805/1948090427] exists,up 3626fa17-5bde-458d-96fb-f5e70ac6edd6 2023-11-05T08:32:39.052 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.53:6816/121537835,v1:172.21.15.53:6818/121537835] [v2:172.21.15.53:6820/121537835,v1:172.21.15.53:6822/121537835] exists,up 8f524504-91f5-44f3-afbd-69dfc19f03d3 2023-11-05T08:32:39.053 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.69:6800/1463424062,v1:172.21.15.69:6801/1463424062] [v2:172.21.15.69:6802/1463424062,v1:172.21.15.69:6803/1463424062] exists,up 696f71d0-a98b-46b5-bdc0-0da3d6065469 2023-11-05T08:32:39.053 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 up in weight 1 up_from 21 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.90:6800/3243652226,v1:172.21.15.90:6801/3243652226] [v2:172.21.15.90:6802/3243652226,v1:172.21.15.90:6803/3243652226] exists,up fc979d30-659b-47f4-b06b-97a4d41c2e90 2023-11-05T08:32:39.053 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.2:6810/2916800887,v1:172.21.15.2:6811/2916800887] [v2:172.21.15.2:6812/2916800887,v1:172.21.15.2:6813/2916800887] exists,up 2ed87cb9-9ced-46fd-8346-3f47709bb69e 2023-11-05T08:32:39.053 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.53:6808/588137330,v1:172.21.15.53:6809/588137330] [v2:172.21.15.53:6810/588137330,v1:172.21.15.53:6811/588137330] exists,up 5bb59d8d-9b93-48b2-a5ee-13646b19ce50 2023-11-05T08:32:39.054 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 up in weight 1 up_from 21 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.69:6815/3720624375,v1:172.21.15.69:6818/3720624375] [v2:172.21.15.69:6821/3720624375,v1:172.21.15.69:6823/3720624375] exists,up 46cd8f39-841d-4b7b-a4ef-6ec6493eea5c 2023-11-05T08:32:39.054 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.90:6808/2028724524,v1:172.21.15.90:6809/2028724524] [v2:172.21.15.90:6810/2028724524,v1:172.21.15.90:6811/2028724524] exists,up c98d93d9-c1e5-4ac0-8f71-9fc1d2767963 2023-11-05T08:32:39.054 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.2:6818/1795116288,v1:172.21.15.2:6820/1795116288] [v2:172.21.15.2:6822/1795116288,v1:172.21.15.2:6823/1795116288] exists,up fcb20ed4-d041-4ef8-bf17-724a43758e50 2023-11-05T08:32:39.054 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.53:6800/1366183694,v1:172.21.15.53:6801/1366183694] [v2:172.21.15.53:6802/1366183694,v1:172.21.15.53:6803/1366183694] exists,up f85ae771-21d9-4974-a203-7ff48225c8a4 2023-11-05T08:32:39.054 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.69:6810/2763568647,v1:172.21.15.69:6813/2763568647] [v2:172.21.15.69:6819/2763568647,v1:172.21.15.69:6822/2763568647] exists,up 496442a8-9bc1-4b66-a8bc-294488f6f089 2023-11-05T08:32:39.055 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.90:6816/2584956675,v1:172.21.15.90:6817/2584956675] [v2:172.21.15.90:6818/2584956675,v1:172.21.15.90:6819/2584956675] exists,up 5680ea76-e74b-43e8-8933-39204c641f00 2023-11-05T08:32:39.055 INFO:teuthology.orchestra.run.smithi002.stdout:osd.12 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.2:6819/205377094,v1:172.21.15.2:6821/205377094] [v2:172.21.15.2:6824/205377094,v1:172.21.15.2:6826/205377094] exists,up 2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa 2023-11-05T08:32:39.055 INFO:teuthology.orchestra.run.smithi002.stdout:osd.13 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.53:6817/714840017,v1:172.21.15.53:6819/714840017] [v2:172.21.15.53:6821/714840017,v1:172.21.15.53:6823/714840017] exists,up 356e7e3c-675d-4e78-ba4a-1d993093dc20 2023-11-05T08:32:39.055 INFO:teuthology.orchestra.run.smithi002.stdout:osd.14 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.69:6808/567567051,v1:172.21.15.69:6809/567567051] [v2:172.21.15.69:6811/567567051,v1:172.21.15.69:6812/567567051] exists,up 1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f 2023-11-05T08:32:39.055 INFO:teuthology.orchestra.run.smithi002.stdout:osd.15 up in weight 1 up_from 21 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.90:6824/739151808,v1:172.21.15.90:6825/739151808] [v2:172.21.15.90:6826/739151808,v1:172.21.15.90:6827/739151808] exists,up 4a37e23e-e1ec-4444-b87b-a9897a448f2a 2023-11-05T08:32:39.067 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.2:6802/1948090427,v1:172.21.15.2:6803/1948090427] [v2:172.21.15.2:6804/1948090427,v1:172.21.15.2:6805/1948090427] exists,up 3626fa17-5bde-458d-96fb-f5e70ac6edd6', 'osd.1 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.53:6816/121537835,v1:172.21.15.53:6818/121537835] [v2:172.21.15.53:6820/121537835,v1:172.21.15.53:6822/121537835] exists,up 8f524504-91f5-44f3-afbd-69dfc19f03d3', 'osd.2 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.69:6800/1463424062,v1:172.21.15.69:6801/1463424062] [v2:172.21.15.69:6802/1463424062,v1:172.21.15.69:6803/1463424062] exists,up 696f71d0-a98b-46b5-bdc0-0da3d6065469', 'osd.3 up in weight 1 up_from 21 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.90:6800/3243652226,v1:172.21.15.90:6801/3243652226] [v2:172.21.15.90:6802/3243652226,v1:172.21.15.90:6803/3243652226] exists,up fc979d30-659b-47f4-b06b-97a4d41c2e90', 'osd.4 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.2:6810/2916800887,v1:172.21.15.2:6811/2916800887] [v2:172.21.15.2:6812/2916800887,v1:172.21.15.2:6813/2916800887] exists,up 2ed87cb9-9ced-46fd-8346-3f47709bb69e', 'osd.5 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.53:6808/588137330,v1:172.21.15.53:6809/588137330] [v2:172.21.15.53:6810/588137330,v1:172.21.15.53:6811/588137330] exists,up 5bb59d8d-9b93-48b2-a5ee-13646b19ce50', 'osd.6 up in weight 1 up_from 21 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.69:6815/3720624375,v1:172.21.15.69:6818/3720624375] [v2:172.21.15.69:6821/3720624375,v1:172.21.15.69:6823/3720624375] exists,up 46cd8f39-841d-4b7b-a4ef-6ec6493eea5c', 'osd.7 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.90:6808/2028724524,v1:172.21.15.90:6809/2028724524] [v2:172.21.15.90:6810/2028724524,v1:172.21.15.90:6811/2028724524] exists,up c98d93d9-c1e5-4ac0-8f71-9fc1d2767963', 'osd.8 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.2:6818/1795116288,v1:172.21.15.2:6820/1795116288] [v2:172.21.15.2:6822/1795116288,v1:172.21.15.2:6823/1795116288] exists,up fcb20ed4-d041-4ef8-bf17-724a43758e50', 'osd.9 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.53:6800/1366183694,v1:172.21.15.53:6801/1366183694] [v2:172.21.15.53:6802/1366183694,v1:172.21.15.53:6803/1366183694] exists,up f85ae771-21d9-4974-a203-7ff48225c8a4', 'osd.10 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.69:6810/2763568647,v1:172.21.15.69:6813/2763568647] [v2:172.21.15.69:6819/2763568647,v1:172.21.15.69:6822/2763568647] exists,up 496442a8-9bc1-4b66-a8bc-294488f6f089', 'osd.11 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.90:6816/2584956675,v1:172.21.15.90:6817/2584956675] [v2:172.21.15.90:6818/2584956675,v1:172.21.15.90:6819/2584956675] exists,up 5680ea76-e74b-43e8-8933-39204c641f00', 'osd.12 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.2:6819/205377094,v1:172.21.15.2:6821/205377094] [v2:172.21.15.2:6824/205377094,v1:172.21.15.2:6826/205377094] exists,up 2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa', 'osd.13 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.53:6817/714840017,v1:172.21.15.53:6819/714840017] [v2:172.21.15.53:6821/714840017,v1:172.21.15.53:6823/714840017] exists,up 356e7e3c-675d-4e78-ba4a-1d993093dc20', 'osd.14 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.69:6808/567567051,v1:172.21.15.69:6809/567567051] [v2:172.21.15.69:6811/567567051,v1:172.21.15.69:6812/567567051] exists,up 1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f', 'osd.15 up in weight 1 up_from 21 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.90:6824/739151808,v1:172.21.15.90:6825/739151808] [v2:172.21.15.90:6826/739151808,v1:172.21.15.90:6827/739151808] exists,up 4a37e23e-e1ec-4444-b87b-a9897a448f2a'] 2023-11-05T08:32:39.068 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2023-11-05T08:32:39.181 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:32:39.181 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mon.a", 2023-11-05T08:32:39.181 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster": "ceph", 2023-11-05T08:32:39.181 INFO:teuthology.orchestra.run.smithi002.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2023-11-05T08:32:39.182 INFO:teuthology.orchestra.run.smithi002.stdout: "admin_socket_mode": "", 2023-11-05T08:32:39.182 INFO:teuthology.orchestra.run.smithi002.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2023-11-05T08:32:39.182 INFO:teuthology.orchestra.run.smithi002.stdout: "auth_client_required": "cephx, none", 2023-11-05T08:32:39.182 INFO:teuthology.orchestra.run.smithi002.stdout: "auth_cluster_required": "cephx", 2023-11-05T08:32:39.182 INFO:teuthology.orchestra.run.smithi002.stdout: "auth_debug": "true", 2023-11-05T08:32:39.183 INFO:teuthology.orchestra.run.smithi002.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2023-11-05T08:32:39.183 INFO:teuthology.orchestra.run.smithi002.stdout: "auth_mon_ticket_ttl": "660.000000", 2023-11-05T08:32:39.183 INFO:teuthology.orchestra.run.smithi002.stdout: "auth_service_required": "cephx", 2023-11-05T08:32:39.183 INFO:teuthology.orchestra.run.smithi002.stdout: "auth_service_ticket_ttl": "240.000000", 2023-11-05T08:32:39.183 INFO:teuthology.orchestra.run.smithi002.stdout: "auth_supported": "cephx", 2023-11-05T08:32:39.184 INFO:teuthology.orchestra.run.smithi002.stdout: "bdev_aio": "true", 2023-11-05T08:32:39.184 INFO:teuthology.orchestra.run.smithi002.stdout: "bdev_aio_max_queue_depth": "1024", 2023-11-05T08:32:39.184 INFO:teuthology.orchestra.run.smithi002.stdout: "bdev_aio_poll_ms": "250", 2023-11-05T08:32:39.184 INFO:teuthology.orchestra.run.smithi002.stdout: "bdev_aio_reap_max": "16", 2023-11-05T08:32:39.184 INFO:teuthology.orchestra.run.smithi002.stdout: "bdev_async_discard": "false", 2023-11-05T08:32:39.185 INFO:teuthology.orchestra.run.smithi002.stdout: "bdev_block_size": "4096", 2023-11-05T08:32:39.185 INFO:teuthology.orchestra.run.smithi002.stdout: "bdev_debug_aio": "false", 2023-11-05T08:32:39.185 INFO:teuthology.orchestra.run.smithi002.stdout: "bdev_debug_aio_log_age": "5.000000", 2023-11-05T08:32:39.185 INFO:teuthology.orchestra.run.smithi002.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2023-11-05T08:32:39.185 INFO:teuthology.orchestra.run.smithi002.stdout: "bdev_debug_inflight_ios": "false", 2023-11-05T08:32:39.186 INFO:teuthology.orchestra.run.smithi002.stdout: "bdev_enable_discard": "false", 2023-11-05T08:32:39.186 INFO:teuthology.orchestra.run.smithi002.stdout: "bdev_flock_retry": "3", 2023-11-05T08:32:39.186 INFO:teuthology.orchestra.run.smithi002.stdout: "bdev_flock_retry_interval": "0.100000", 2023-11-05T08:32:39.186 INFO:teuthology.orchestra.run.smithi002.stdout: "bdev_inject_crash": "0", 2023-11-05T08:32:39.186 INFO:teuthology.orchestra.run.smithi002.stdout: "bdev_inject_crash_flush_delay": "2", 2023-11-05T08:32:39.186 INFO:teuthology.orchestra.run.smithi002.stdout: "bdev_ioring": "false", 2023-11-05T08:32:39.187 INFO:teuthology.orchestra.run.smithi002.stdout: "bdev_ioring_hipri": "false", 2023-11-05T08:32:39.187 INFO:teuthology.orchestra.run.smithi002.stdout: "bdev_ioring_sqthread_poll": "false", 2023-11-05T08:32:39.187 INFO:teuthology.orchestra.run.smithi002.stdout: "bdev_nvme_unbind_from_kernel": "false", 2023-11-05T08:32:39.187 INFO:teuthology.orchestra.run.smithi002.stdout: "bdev_read_buffer_alignment": "4096", 2023-11-05T08:32:39.187 INFO:teuthology.orchestra.run.smithi002.stdout: "bdev_read_preallocated_huge_buffers": "", 2023-11-05T08:32:39.188 INFO:teuthology.orchestra.run.smithi002.stdout: "bdev_type": "", 2023-11-05T08:32:39.188 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_alloc_size": "1048576", 2023-11-05T08:32:39.188 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_allocator": "hybrid", 2023-11-05T08:32:39.188 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_buffered_io": "true", 2023-11-05T08:32:39.188 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_check_for_zeros": "false", 2023-11-05T08:32:39.189 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_check_volume_selector_often": "false", 2023-11-05T08:32:39.189 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_check_volume_selector_on_umount": "false", 2023-11-05T08:32:39.189 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_compact_log_sync": "false", 2023-11-05T08:32:39.189 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2023-11-05T08:32:39.189 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2023-11-05T08:32:39.190 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_log_compact_min_size": "16777216", 2023-11-05T08:32:39.190 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_log_replay_check_allocations": "true", 2023-11-05T08:32:39.190 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_max_log_runway": "4194304", 2023-11-05T08:32:39.190 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_max_prefetch": "1048576", 2023-11-05T08:32:39.190 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_min_flush_size": "524288", 2023-11-05T08:32:39.191 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_min_log_runway": "1048576", 2023-11-05T08:32:39.191 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_replay_recovery": "false", 2023-11-05T08:32:39.191 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_replay_recovery_disable_compact": "false", 2023-11-05T08:32:39.191 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_shared_alloc_size": "65536", 2023-11-05T08:32:39.191 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_sync_write": "false", 2023-11-05T08:32:39.191 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2023-11-05T08:32:39.192 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2023-11-05T08:32:39.192 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2023-11-05T08:32:39.192 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_allocation_from_file": "true", 2023-11-05T08:32:39.192 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_allocator": "hybrid", 2023-11-05T08:32:39.192 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2023-11-05T08:32:39.192 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2023-11-05T08:32:39.193 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2023-11-05T08:32:39.193 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2023-11-05T08:32:39.193 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2023-11-05T08:32:39.193 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_bitmapallocator_span_size": "1024", 2023-11-05T08:32:39.193 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_blobid_prealloc": "10240", 2023-11-05T08:32:39.194 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_block_create": "true", 2023-11-05T08:32:39.194 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_block_db_create": "false", 2023-11-05T08:32:39.194 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_block_db_path": "", 2023-11-05T08:32:39.194 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_block_db_size": "0", 2023-11-05T08:32:39.194 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_block_path": "", 2023-11-05T08:32:39.195 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_block_preallocate_file": "false", 2023-11-05T08:32:39.195 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_block_size": "107374182400", 2023-11-05T08:32:39.195 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_block_wal_create": "false", 2023-11-05T08:32:39.195 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_block_wal_path": "", 2023-11-05T08:32:39.195 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_block_wal_size": "100663296", 2023-11-05T08:32:39.196 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_bluefs": "true", 2023-11-05T08:32:39.196 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2023-11-05T08:32:39.196 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_bluefs_env_mirror": "false", 2023-11-05T08:32:39.196 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_bluefs_max_free": "10737418240", 2023-11-05T08:32:39.196 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2023-11-05T08:32:39.197 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2023-11-05T08:32:39.197 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2023-11-05T08:32:39.197 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2023-11-05T08:32:39.197 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2023-11-05T08:32:39.197 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_autotune": "true", 2023-11-05T08:32:39.198 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_autotune_interval": "5.000000", 2023-11-05T08:32:39.198 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2023-11-05T08:32:39.198 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_kv_ratio": "0.450000", 2023-11-05T08:32:39.198 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_meta_ratio": "0.450000", 2023-11-05T08:32:39.198 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_size": "0", 2023-11-05T08:32:39.199 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_size_hdd": "1073741824", 2023-11-05T08:32:39.199 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_size_ssd": "3221225472", 2023-11-05T08:32:39.199 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_trim_interval": "0.050000", 2023-11-05T08:32:39.199 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2023-11-05T08:32:39.199 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_type": "2q", 2023-11-05T08:32:39.200 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2023-11-05T08:32:39.200 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_clone_cow": "true", 2023-11-05T08:32:39.200 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_compression_algorithm": "snappy", 2023-11-05T08:32:39.200 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_compression_max_blob_size": "0", 2023-11-05T08:32:39.200 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2023-11-05T08:32:39.201 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2023-11-05T08:32:39.201 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_compression_min_blob_size": "0", 2023-11-05T08:32:39.201 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2023-11-05T08:32:39.201 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2023-11-05T08:32:39.201 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_compression_mode": "none", 2023-11-05T08:32:39.202 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_compression_required_ratio": "0.875000", 2023-11-05T08:32:39.202 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_csum_type": "crc32c", 2023-11-05T08:32:39.202 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_debug_enforce_settings": "default", 2023-11-05T08:32:39.202 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_debug_freelist": "false", 2023-11-05T08:32:39.203 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_debug_fsck_abort": "false", 2023-11-05T08:32:39.203 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2023-11-05T08:32:39.203 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2023-11-05T08:32:39.203 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_debug_inject_read_err": "false", 2023-11-05T08:32:39.203 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_debug_legacy_omap": "false", 2023-11-05T08:32:39.204 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_debug_no_reuse_blocks": "false", 2023-11-05T08:32:39.204 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_debug_omit_block_device_write": "false", 2023-11-05T08:32:39.204 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_debug_omit_kv_commit": "false", 2023-11-05T08:32:39.204 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2023-11-05T08:32:39.204 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_debug_prefill": "0.000000", 2023-11-05T08:32:39.205 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_debug_prefragment_max": "1048576", 2023-11-05T08:32:39.205 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_debug_random_read_err": "0.000000", 2023-11-05T08:32:39.205 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2023-11-05T08:32:39.205 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_debug_small_allocations": "0", 2023-11-05T08:32:39.205 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2023-11-05T08:32:39.206 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_default_buffered_read": "true", 2023-11-05T08:32:39.206 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_default_buffered_write": "false", 2023-11-05T08:32:39.206 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_deferred_batch_ops": "0", 2023-11-05T08:32:39.206 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2023-11-05T08:32:39.206 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2023-11-05T08:32:39.207 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_elastic_shared_blobs": "true", 2023-11-05T08:32:39.207 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2023-11-05T08:32:39.207 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_extent_map_shard_max_size": "1200", 2023-11-05T08:32:39.207 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_extent_map_shard_min_size": "150", 2023-11-05T08:32:39.207 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_extent_map_shard_target_size": "500", 2023-11-05T08:32:39.207 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2023-11-05T08:32:39.208 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fail_eio": "false", 2023-11-05T08:32:39.208 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_freelist_blocks_per_key": "128", 2023-11-05T08:32:39.208 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2023-11-05T08:32:39.208 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2023-11-05T08:32:39.208 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2023-11-05T08:32:39.209 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_on_mkfs": "true", 2023-11-05T08:32:39.209 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2023-11-05T08:32:39.209 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_on_mount": "false", 2023-11-05T08:32:39.209 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_on_mount_deep": "false", 2023-11-05T08:32:39.209 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_on_umount": "false", 2023-11-05T08:32:39.210 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_on_umount_deep": "false", 2023-11-05T08:32:39.210 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2023-11-05T08:32:39.210 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_quick_fix_threads": "2", 2023-11-05T08:32:39.210 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2023-11-05T08:32:39.210 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2023-11-05T08:32:39.211 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_gc_enable_blob_threshold": "0", 2023-11-05T08:32:39.211 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_gc_enable_total_threshold": "0", 2023-11-05T08:32:39.211 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2023-11-05T08:32:39.211 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_ignore_data_csum": "false", 2023-11-05T08:32:39.211 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2023-11-05T08:32:39.211 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_kvbackend": "rocksdb", 2023-11-05T08:32:39.212 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_log_collection_list_age": "60.000000", 2023-11-05T08:32:39.212 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2023-11-05T08:32:39.212 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_log_op_age": "5.000000", 2023-11-05T08:32:39.212 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_max_alloc_size": "0", 2023-11-05T08:32:39.212 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_max_blob_size": "0", 2023-11-05T08:32:39.213 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_max_blob_size_hdd": "65536", 2023-11-05T08:32:39.213 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_max_blob_size_ssd": "65536", 2023-11-05T08:32:39.213 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_max_defer_interval": "3.000000", 2023-11-05T08:32:39.213 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_max_deferred_txc": "32", 2023-11-05T08:32:39.213 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_min_alloc_size": "0", 2023-11-05T08:32:39.214 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_min_alloc_size_hdd": "4096", 2023-11-05T08:32:39.214 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_min_alloc_size_ssd": "4096", 2023-11-05T08:32:39.214 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_nid_prealloc": "1024", 2023-11-05T08:32:39.214 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_prefer_deferred_size": "0", 2023-11-05T08:32:39.214 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2023-11-05T08:32:39.215 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2023-11-05T08:32:39.215 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_qfsck_on_mount": "true", 2023-11-05T08:32:39.215 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_retry_disk_reads": "3", 2023-11-05T08:32:39.215 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_rocksdb_cf": "true", 2023-11-05T08:32:39.215 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2023-11-05T08:32:39.216 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2023-11-05T08:32:39.216 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_rocksdb_options_annex": "", 2023-11-05T08:32:39.216 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_spdk_coremask": "0x1", 2023-11-05T08:32:39.216 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_spdk_io_sleep": "5", 2023-11-05T08:32:39.216 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_spdk_max_io_completion": "0", 2023-11-05T08:32:39.217 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_spdk_mem": "512", 2023-11-05T08:32:39.217 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_sync_submit_transaction": "false", 2023-11-05T08:32:39.217 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_throttle_bytes": "67108864", 2023-11-05T08:32:39.217 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_throttle_cost_per_io": "0", 2023-11-05T08:32:39.217 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2023-11-05T08:32:39.218 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2023-11-05T08:32:39.218 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2023-11-05T08:32:39.218 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_throttle_trace_rate": "0.000000", 2023-11-05T08:32:39.218 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_tracing": "false", 2023-11-05T08:32:39.219 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2023-11-05T08:32:39.219 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2023-11-05T08:32:39.219 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_volume_selection_reserved": "0", 2023-11-05T08:32:39.219 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2023-11-05T08:32:39.219 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2023-11-05T08:32:39.220 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_warn_on_legacy_statfs": "true", 2023-11-05T08:32:39.220 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2023-11-05T08:32:39.220 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2023-11-05T08:32:39.220 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2023-11-05T08:32:39.220 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_zero_block_detection": "false", 2023-11-05T08:32:39.220 INFO:teuthology.orchestra.run.smithi002.stdout: "cephadm_path": "/usr/sbin/cephadm", 2023-11-05T08:32:39.221 INFO:teuthology.orchestra.run.smithi002.stdout: "cephfs_mirror_action_update_interval": "2", 2023-11-05T08:32:39.221 INFO:teuthology.orchestra.run.smithi002.stdout: "cephfs_mirror_directory_scan_interval": "10", 2023-11-05T08:32:39.221 INFO:teuthology.orchestra.run.smithi002.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2023-11-05T08:32:39.221 INFO:teuthology.orchestra.run.smithi002.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2023-11-05T08:32:39.221 INFO:teuthology.orchestra.run.smithi002.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2023-11-05T08:32:39.222 INFO:teuthology.orchestra.run.smithi002.stdout: "cephfs_mirror_mount_timeout": "10", 2023-11-05T08:32:39.222 INFO:teuthology.orchestra.run.smithi002.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2023-11-05T08:32:39.222 INFO:teuthology.orchestra.run.smithi002.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2023-11-05T08:32:39.222 INFO:teuthology.orchestra.run.smithi002.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2023-11-05T08:32:39.222 INFO:teuthology.orchestra.run.smithi002.stdout: "cephsqlite_blocklist_dead_locker": "true", 2023-11-05T08:32:39.223 INFO:teuthology.orchestra.run.smithi002.stdout: "cephsqlite_lock_renewal_interval": "2000", 2023-11-05T08:32:39.223 INFO:teuthology.orchestra.run.smithi002.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2023-11-05T08:32:39.223 INFO:teuthology.orchestra.run.smithi002.stdout: "cephx_cluster_require_signatures": "false", 2023-11-05T08:32:39.223 INFO:teuthology.orchestra.run.smithi002.stdout: "cephx_cluster_require_version": "2", 2023-11-05T08:32:39.224 INFO:teuthology.orchestra.run.smithi002.stdout: "cephx_require_signatures": "false", 2023-11-05T08:32:39.224 INFO:teuthology.orchestra.run.smithi002.stdout: "cephx_require_version": "2", 2023-11-05T08:32:39.224 INFO:teuthology.orchestra.run.smithi002.stdout: "cephx_service_require_signatures": "false", 2023-11-05T08:32:39.224 INFO:teuthology.orchestra.run.smithi002.stdout: "cephx_service_require_version": "2", 2023-11-05T08:32:39.224 INFO:teuthology.orchestra.run.smithi002.stdout: "cephx_sign_messages": "true", 2023-11-05T08:32:39.224 INFO:teuthology.orchestra.run.smithi002.stdout: "chdir": "", 2023-11-05T08:32:39.225 INFO:teuthology.orchestra.run.smithi002.stdout: "client_acl_type": "", 2023-11-05T08:32:39.225 INFO:teuthology.orchestra.run.smithi002.stdout: "client_asio_thread_count": "2", 2023-11-05T08:32:39.225 INFO:teuthology.orchestra.run.smithi002.stdout: "client_cache_mid": "0.750000", 2023-11-05T08:32:39.225 INFO:teuthology.orchestra.run.smithi002.stdout: "client_cache_size": "16384", 2023-11-05T08:32:39.226 INFO:teuthology.orchestra.run.smithi002.stdout: "client_caps_release_delay": "5", 2023-11-05T08:32:39.226 INFO:teuthology.orchestra.run.smithi002.stdout: "client_check_pool_perm": "true", 2023-11-05T08:32:39.226 INFO:teuthology.orchestra.run.smithi002.stdout: "client_collect_and_send_global_metrics": "false", 2023-11-05T08:32:39.226 INFO:teuthology.orchestra.run.smithi002.stdout: "client_debug_force_sync_read": "false", 2023-11-05T08:32:39.226 INFO:teuthology.orchestra.run.smithi002.stdout: "client_debug_getattr_caps": "false", 2023-11-05T08:32:39.226 INFO:teuthology.orchestra.run.smithi002.stdout: "client_debug_inject_tick_delay": "0", 2023-11-05T08:32:39.227 INFO:teuthology.orchestra.run.smithi002.stdout: "client_die_on_failed_dentry_invalidate": "true", 2023-11-05T08:32:39.227 INFO:teuthology.orchestra.run.smithi002.stdout: "client_die_on_failed_remount": "false", 2023-11-05T08:32:39.227 INFO:teuthology.orchestra.run.smithi002.stdout: "client_dirsize_rbytes": "true", 2023-11-05T08:32:39.227 INFO:teuthology.orchestra.run.smithi002.stdout: "client_force_lazyio": "false", 2023-11-05T08:32:39.228 INFO:teuthology.orchestra.run.smithi002.stdout: "client_fs": "", 2023-11-05T08:32:39.228 INFO:teuthology.orchestra.run.smithi002.stdout: "client_inject_fixed_oldest_tid": "false", 2023-11-05T08:32:39.228 INFO:teuthology.orchestra.run.smithi002.stdout: "client_inject_release_failure": "false", 2023-11-05T08:32:39.228 INFO:teuthology.orchestra.run.smithi002.stdout: "client_max_inline_size": "4096", 2023-11-05T08:32:39.228 INFO:teuthology.orchestra.run.smithi002.stdout: "client_max_retries_on_remount_failure": "5", 2023-11-05T08:32:39.228 INFO:teuthology.orchestra.run.smithi002.stdout: "client_mds_namespace": "", 2023-11-05T08:32:39.229 INFO:teuthology.orchestra.run.smithi002.stdout: "client_metadata": "", 2023-11-05T08:32:39.229 INFO:teuthology.orchestra.run.smithi002.stdout: "client_mount_gid": "-1", 2023-11-05T08:32:39.229 INFO:teuthology.orchestra.run.smithi002.stdout: "client_mount_timeout": "300", 2023-11-05T08:32:39.229 INFO:teuthology.orchestra.run.smithi002.stdout: "client_mount_uid": "-1", 2023-11-05T08:32:39.229 INFO:teuthology.orchestra.run.smithi002.stdout: "client_mountpoint": "/", 2023-11-05T08:32:39.230 INFO:teuthology.orchestra.run.smithi002.stdout: "client_notify_timeout": "10", 2023-11-05T08:32:39.230 INFO:teuthology.orchestra.run.smithi002.stdout: "client_oc": "true", 2023-11-05T08:32:39.230 INFO:teuthology.orchestra.run.smithi002.stdout: "client_oc_max_dirty": "104857600", 2023-11-05T08:32:39.230 INFO:teuthology.orchestra.run.smithi002.stdout: "client_oc_max_dirty_age": "5.000000", 2023-11-05T08:32:39.230 INFO:teuthology.orchestra.run.smithi002.stdout: "client_oc_max_objects": "1000", 2023-11-05T08:32:39.231 INFO:teuthology.orchestra.run.smithi002.stdout: "client_oc_size": "209715200", 2023-11-05T08:32:39.231 INFO:teuthology.orchestra.run.smithi002.stdout: "client_oc_target_dirty": "8388608", 2023-11-05T08:32:39.231 INFO:teuthology.orchestra.run.smithi002.stdout: "client_permissions": "true", 2023-11-05T08:32:39.231 INFO:teuthology.orchestra.run.smithi002.stdout: "client_quota": "true", 2023-11-05T08:32:39.231 INFO:teuthology.orchestra.run.smithi002.stdout: "client_quota_df": "true", 2023-11-05T08:32:39.232 INFO:teuthology.orchestra.run.smithi002.stdout: "client_readahead_max_bytes": "0", 2023-11-05T08:32:39.232 INFO:teuthology.orchestra.run.smithi002.stdout: "client_readahead_max_periods": "4", 2023-11-05T08:32:39.232 INFO:teuthology.orchestra.run.smithi002.stdout: "client_readahead_min": "131072", 2023-11-05T08:32:39.232 INFO:teuthology.orchestra.run.smithi002.stdout: "client_reconnect_stale": "false", 2023-11-05T08:32:39.232 INFO:teuthology.orchestra.run.smithi002.stdout: "client_shutdown_timeout": "30", 2023-11-05T08:32:39.232 INFO:teuthology.orchestra.run.smithi002.stdout: "client_snapdir": ".snap", 2023-11-05T08:32:39.233 INFO:teuthology.orchestra.run.smithi002.stdout: "client_tick_interval": "1", 2023-11-05T08:32:39.233 INFO:teuthology.orchestra.run.smithi002.stdout: "client_trace": "", 2023-11-05T08:32:39.233 INFO:teuthology.orchestra.run.smithi002.stdout: "client_try_dentry_invalidate": "false", 2023-11-05T08:32:39.233 INFO:teuthology.orchestra.run.smithi002.stdout: "client_use_faked_inos": "false", 2023-11-05T08:32:39.233 INFO:teuthology.orchestra.run.smithi002.stdout: "client_use_random_mds": "false", 2023-11-05T08:32:39.233 INFO:teuthology.orchestra.run.smithi002.stdout: "clog_to_graylog": "false", 2023-11-05T08:32:39.234 INFO:teuthology.orchestra.run.smithi002.stdout: "clog_to_graylog_host": "127.0.0.1", 2023-11-05T08:32:39.234 INFO:teuthology.orchestra.run.smithi002.stdout: "clog_to_graylog_port": "12201", 2023-11-05T08:32:39.234 INFO:teuthology.orchestra.run.smithi002.stdout: "clog_to_monitors": "default=true", 2023-11-05T08:32:39.234 INFO:teuthology.orchestra.run.smithi002.stdout: "clog_to_syslog": "false", 2023-11-05T08:32:39.234 INFO:teuthology.orchestra.run.smithi002.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2023-11-05T08:32:39.235 INFO:teuthology.orchestra.run.smithi002.stdout: "clog_to_syslog_level": "info", 2023-11-05T08:32:39.235 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_addr": "-", 2023-11-05T08:32:39.235 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_network": "", 2023-11-05T08:32:39.235 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_network_interface": "", 2023-11-05T08:32:39.235 INFO:teuthology.orchestra.run.smithi002.stdout: "compressor_zlib_isal": "false", 2023-11-05T08:32:39.235 INFO:teuthology.orchestra.run.smithi002.stdout: "compressor_zlib_level": "5", 2023-11-05T08:32:39.236 INFO:teuthology.orchestra.run.smithi002.stdout: "compressor_zlib_winsize": "-15", 2023-11-05T08:32:39.236 INFO:teuthology.orchestra.run.smithi002.stdout: "compressor_zstd_level": "1", 2023-11-05T08:32:39.236 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2023-11-05T08:32:39.236 INFO:teuthology.orchestra.run.smithi002.stdout: "crash_dir": "/var/lib/ceph/crash", 2023-11-05T08:32:39.236 INFO:teuthology.orchestra.run.smithi002.stdout: "crimson_alien_op_num_threads": "6", 2023-11-05T08:32:39.237 INFO:teuthology.orchestra.run.smithi002.stdout: "crimson_alien_thread_cpu_cores": "", 2023-11-05T08:32:39.237 INFO:teuthology.orchestra.run.smithi002.stdout: "crimson_osd_obc_lru_size": "10", 2023-11-05T08:32:39.237 INFO:teuthology.orchestra.run.smithi002.stdout: "crimson_osd_scheduler_concurrency": "0", 2023-11-05T08:32:39.237 INFO:teuthology.orchestra.run.smithi002.stdout: "crimson_seastar_smp": "1", 2023-11-05T08:32:39.237 INFO:teuthology.orchestra.run.smithi002.stdout: "crush_location": "", 2023-11-05T08:32:39.238 INFO:teuthology.orchestra.run.smithi002.stdout: "crush_location_hook": "", 2023-11-05T08:32:39.238 INFO:teuthology.orchestra.run.smithi002.stdout: "crush_location_hook_timeout": "10", 2023-11-05T08:32:39.238 INFO:teuthology.orchestra.run.smithi002.stdout: "daemonize": "false", 2023-11-05T08:32:39.238 INFO:teuthology.orchestra.run.smithi002.stdout: "daos_pool": "tank", 2023-11-05T08:32:39.238 INFO:teuthology.orchestra.run.smithi002.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2023-11-05T08:32:39.239 INFO:teuthology.orchestra.run.smithi002.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2023-11-05T08:32:39.239 INFO:teuthology.orchestra.run.smithi002.stdout: "dbstore_db_name_prefix": "dbstore", 2023-11-05T08:32:39.239 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_alienstore": "0/5", 2023-11-05T08:32:39.239 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_allow_any_pool_priority": "false", 2023-11-05T08:32:39.239 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_asok": "1/5", 2023-11-05T08:32:39.240 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_asok_assert_abort": "false", 2023-11-05T08:32:39.240 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_asserts_on_shutdown": "true", 2023-11-05T08:32:39.240 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_auth": "20/20", 2023-11-05T08:32:39.240 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_bdev": "1/3", 2023-11-05T08:32:39.241 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_bluefs": "1/5", 2023-11-05T08:32:39.241 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_bluestore": "1/5", 2023-11-05T08:32:39.241 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_buffer": "0/1", 2023-11-05T08:32:39.241 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_ceph_exporter": "1/5", 2023-11-05T08:32:39.241 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_cephfs_mirror": "0/5", 2023-11-05T08:32:39.242 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_cephsqlite": "0/5", 2023-11-05T08:32:39.242 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_client": "0/5", 2023-11-05T08:32:39.242 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_compressor": "1/5", 2023-11-05T08:32:39.242 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_context": "0/1", 2023-11-05T08:32:39.243 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_crush": "1/1", 2023-11-05T08:32:39.243 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_crypto": "1/5", 2023-11-05T08:32:39.243 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_cyanstore": "0/5", 2023-11-05T08:32:39.243 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_deliberately_leak_memory": "false", 2023-11-05T08:32:39.243 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_disable_randomized_ping": "false", 2023-11-05T08:32:39.244 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_dpdk": "1/5", 2023-11-05T08:32:39.244 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_eventtrace": "1/5", 2023-11-05T08:32:39.244 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_filer": "0/1", 2023-11-05T08:32:39.244 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_filestore": "1/3", 2023-11-05T08:32:39.244 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_finisher": "1/1", 2023-11-05T08:32:39.245 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_fuse": "1/5", 2023-11-05T08:32:39.245 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_heartbeat_testing_span": "0", 2023-11-05T08:32:39.245 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_heartbeatmap": "1/5", 2023-11-05T08:32:39.245 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_immutable_obj_cache": "0/5", 2023-11-05T08:32:39.246 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_javaclient": "1/5", 2023-11-05T08:32:39.246 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_journal": "1/3", 2023-11-05T08:32:39.246 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_journaler": "0/5", 2023-11-05T08:32:39.246 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_kstore": "1/5", 2023-11-05T08:32:39.246 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_lockdep": "0/1", 2023-11-05T08:32:39.247 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_mclock": "1/5", 2023-11-05T08:32:39.247 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_mds": "1/5", 2023-11-05T08:32:39.247 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_mds_balancer": "1/5", 2023-11-05T08:32:39.247 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_mds_locker": "1/5", 2023-11-05T08:32:39.247 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_mds_log": "1/5", 2023-11-05T08:32:39.248 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_mds_log_expire": "1/5", 2023-11-05T08:32:39.248 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_mds_migrator": "1/5", 2023-11-05T08:32:39.248 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_memstore": "1/5", 2023-11-05T08:32:39.248 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_mgr": "2/5", 2023-11-05T08:32:39.248 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_mgrc": "1/5", 2023-11-05T08:32:39.249 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_mon": "20/20", 2023-11-05T08:32:39.249 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_monc": "0/10", 2023-11-05T08:32:39.249 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_ms": "1/1", 2023-11-05T08:32:39.249 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_none": "0/5", 2023-11-05T08:32:39.249 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_objclass": "0/5", 2023-11-05T08:32:39.249 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_objectcacher": "0/5", 2023-11-05T08:32:39.250 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_objecter": "0/1", 2023-11-05T08:32:39.250 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_optracker": "0/5", 2023-11-05T08:32:39.250 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_osd": "1/5", 2023-11-05T08:32:39.250 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_paxos": "20/20", 2023-11-05T08:32:39.250 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_perfcounter": "1/5", 2023-11-05T08:32:39.251 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_prioritycache": "1/5", 2023-11-05T08:32:39.251 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_rados": "0/5", 2023-11-05T08:32:39.251 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_rbd": "0/5", 2023-11-05T08:32:39.251 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_rbd_mirror": "0/5", 2023-11-05T08:32:39.251 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_rbd_pwl": "0/5", 2023-11-05T08:32:39.252 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_rbd_replay": "0/5", 2023-11-05T08:32:39.252 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_refs": "0/0", 2023-11-05T08:32:39.252 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_reserver": "1/1", 2023-11-05T08:32:39.252 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_rgw": "1/5", 2023-11-05T08:32:39.252 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_rgw_access": "1/5", 2023-11-05T08:32:39.253 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_rgw_datacache": "1/5", 2023-11-05T08:32:39.253 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_rgw_dbstore": "1/5", 2023-11-05T08:32:39.253 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_rgw_flight": "1/5", 2023-11-05T08:32:39.253 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_rgw_sync": "1/5", 2023-11-05T08:32:39.253 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_rocksdb": "4/5", 2023-11-05T08:32:39.254 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_seastore": "0/5", 2023-11-05T08:32:39.254 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_seastore_backref": "0/5", 2023-11-05T08:32:39.254 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_seastore_cache": "0/5", 2023-11-05T08:32:39.254 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_seastore_cleaner": "0/5", 2023-11-05T08:32:39.255 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_seastore_device": "0/5", 2023-11-05T08:32:39.255 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_seastore_epm": "0/5", 2023-11-05T08:32:39.255 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_seastore_fixedkv_tree": "0/5", 2023-11-05T08:32:39.255 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_seastore_journal": "0/5", 2023-11-05T08:32:39.255 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_seastore_lba": "0/5", 2023-11-05T08:32:39.256 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_seastore_odata": "0/5", 2023-11-05T08:32:39.256 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_seastore_omap": "0/5", 2023-11-05T08:32:39.256 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_seastore_onode": "0/5", 2023-11-05T08:32:39.256 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_seastore_t": "0/5", 2023-11-05T08:32:39.256 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_seastore_tm": "0/5", 2023-11-05T08:32:39.257 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_striper": "0/1", 2023-11-05T08:32:39.257 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_test": "0/5", 2023-11-05T08:32:39.257 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_throttle": "1/1", 2023-11-05T08:32:39.257 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_timer": "0/1", 2023-11-05T08:32:39.258 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_tp": "0/5", 2023-11-05T08:32:39.258 INFO:teuthology.orchestra.run.smithi002.stdout: "debug_trace": "1/5", 2023-11-05T08:32:39.258 INFO:teuthology.orchestra.run.smithi002.stdout: "defer_client_eviction_on_laggy_osds": "true", 2023-11-05T08:32:39.258 INFO:teuthology.orchestra.run.smithi002.stdout: "device_failure_prediction_mode": "none", 2023-11-05T08:32:39.258 INFO:teuthology.orchestra.run.smithi002.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2023-11-05T08:32:39.259 INFO:teuthology.orchestra.run.smithi002.stdout: "erasure_code_dir": "/usr/lib/ceph/erasure-code", 2023-11-05T08:32:39.259 INFO:teuthology.orchestra.run.smithi002.stdout: "err_to_graylog": "false", 2023-11-05T08:32:39.259 INFO:teuthology.orchestra.run.smithi002.stdout: "err_to_journald": "false", 2023-11-05T08:32:39.259 INFO:teuthology.orchestra.run.smithi002.stdout: "err_to_stderr": "true", 2023-11-05T08:32:39.259 INFO:teuthology.orchestra.run.smithi002.stdout: "err_to_syslog": "false", 2023-11-05T08:32:39.260 INFO:teuthology.orchestra.run.smithi002.stdout: "event_tracing": "false", 2023-11-05T08:32:39.260 INFO:teuthology.orchestra.run.smithi002.stdout: "exporter_addr": "0.0.0.0", 2023-11-05T08:32:39.260 INFO:teuthology.orchestra.run.smithi002.stdout: "exporter_http_port": "9926", 2023-11-05T08:32:39.260 INFO:teuthology.orchestra.run.smithi002.stdout: "exporter_prio_limit": "5", 2023-11-05T08:32:39.260 INFO:teuthology.orchestra.run.smithi002.stdout: "exporter_sock_dir": "/var/run/ceph/", 2023-11-05T08:32:39.261 INFO:teuthology.orchestra.run.smithi002.stdout: "exporter_sort_metrics": "true", 2023-11-05T08:32:39.261 INFO:teuthology.orchestra.run.smithi002.stdout: "exporter_stats_period": "5", 2023-11-05T08:32:39.261 INFO:teuthology.orchestra.run.smithi002.stdout: "fake_statfs_for_testing": "0", 2023-11-05T08:32:39.261 INFO:teuthology.orchestra.run.smithi002.stdout: "fatal_signal_handlers": "true", 2023-11-05T08:32:39.261 INFO:teuthology.orchestra.run.smithi002.stdout: "filer_max_purge_ops": "10", 2023-11-05T08:32:39.261 INFO:teuthology.orchestra.run.smithi002.stdout: "filer_max_truncate_ops": "128", 2023-11-05T08:32:39.262 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_apply_finisher_threads": "1", 2023-11-05T08:32:39.262 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_blackhole": "false", 2023-11-05T08:32:39.262 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_btrfs_clone_range": "true", 2023-11-05T08:32:39.262 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_btrfs_snap": "true", 2023-11-05T08:32:39.262 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_caller_concurrency": "10", 2023-11-05T08:32:39.263 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_collect_device_partition_information": "true", 2023-11-05T08:32:39.263 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_commit_timeout": "600.000000", 2023-11-05T08:32:39.263 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_debug_inject_read_err": "false", 2023-11-05T08:32:39.263 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_debug_omap_check": "false", 2023-11-05T08:32:39.264 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_debug_random_read_err": "0.000000", 2023-11-05T08:32:39.264 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_debug_verify_split": "false", 2023-11-05T08:32:39.264 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_dump_file": "", 2023-11-05T08:32:39.264 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2023-11-05T08:32:39.264 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_expected_throughput_ops": "200.000000", 2023-11-05T08:32:39.265 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_fadvise": "true", 2023-11-05T08:32:39.265 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_fail_eio": "true", 2023-11-05T08:32:39.265 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_fd_cache_shards": "16", 2023-11-05T08:32:39.265 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_fd_cache_size": "128", 2023-11-05T08:32:39.265 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_fiemap": "false", 2023-11-05T08:32:39.266 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_fiemap_threshold": "4096", 2023-11-05T08:32:39.266 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_fsync_flushes_journal_data": "false", 2023-11-05T08:32:39.266 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_index_retry_probability": "0.000000", 2023-11-05T08:32:39.266 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_inject_stall": "0", 2023-11-05T08:32:39.266 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_journal_parallel": "false", 2023-11-05T08:32:39.267 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_journal_trailing": "false", 2023-11-05T08:32:39.267 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_journal_writeahead": "false", 2023-11-05T08:32:39.267 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_kill_at": "0", 2023-11-05T08:32:39.267 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_max_alloc_hint_size": "1048576", 2023-11-05T08:32:39.267 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_max_inline_xattr_size": "0", 2023-11-05T08:32:39.268 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2023-11-05T08:32:39.268 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_max_inline_xattr_size_other": "512", 2023-11-05T08:32:39.268 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2023-11-05T08:32:39.268 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_max_inline_xattrs": "0", 2023-11-05T08:32:39.268 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2023-11-05T08:32:39.269 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_max_inline_xattrs_other": "2", 2023-11-05T08:32:39.269 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_max_inline_xattrs_xfs": "10", 2023-11-05T08:32:39.269 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_max_sync_interval": "5.000000", 2023-11-05T08:32:39.269 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_max_xattr_value_size": "0", 2023-11-05T08:32:39.269 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2023-11-05T08:32:39.270 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_max_xattr_value_size_other": "1024", 2023-11-05T08:32:39.270 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2023-11-05T08:32:39.270 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_merge_threshold": "-10", 2023-11-05T08:32:39.270 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_min_sync_interval": "0.010000", 2023-11-05T08:32:39.270 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_odsync_write": "false", 2023-11-05T08:32:39.271 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_omap_backend": "rocksdb", 2023-11-05T08:32:39.271 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_omap_backend_path": "", 2023-11-05T08:32:39.271 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_omap_header_cache_size": "1024", 2023-11-05T08:32:39.271 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_ondisk_finisher_threads": "1", 2023-11-05T08:32:39.271 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_op_thread_suicide_timeout": "180", 2023-11-05T08:32:39.272 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_op_thread_timeout": "60", 2023-11-05T08:32:39.272 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_op_threads": "2", 2023-11-05T08:32:39.272 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_punch_hole": "false", 2023-11-05T08:32:39.272 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2023-11-05T08:32:39.272 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2023-11-05T08:32:39.273 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2023-11-05T08:32:39.273 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_queue_high_threshhold": "0.900000", 2023-11-05T08:32:39.273 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_queue_low_threshhold": "0.300000", 2023-11-05T08:32:39.273 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_queue_max_bytes": "104857600", 2023-11-05T08:32:39.273 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2023-11-05T08:32:39.274 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2023-11-05T08:32:39.274 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2023-11-05T08:32:39.274 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_queue_max_ops": "50", 2023-11-05T08:32:39.274 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2023-11-05T08:32:39.274 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_seek_data_hole": "false", 2023-11-05T08:32:39.275 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_sloppy_crc": "false", 2023-11-05T08:32:39.275 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_sloppy_crc_block_size": "65536", 2023-11-05T08:32:39.275 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_splice": "false", 2023-11-05T08:32:39.275 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_split_multiple": "2", 2023-11-05T08:32:39.275 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_split_rand_factor": "20", 2023-11-05T08:32:39.276 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_update_to": "1000", 2023-11-05T08:32:39.276 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2023-11-05T08:32:39.276 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2023-11-05T08:32:39.276 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2023-11-05T08:32:39.276 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2023-11-05T08:32:39.276 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2023-11-05T08:32:39.277 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2023-11-05T08:32:39.277 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_wbthrottle_enable": "true", 2023-11-05T08:32:39.277 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2023-11-05T08:32:39.277 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2023-11-05T08:32:39.277 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2023-11-05T08:32:39.278 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2023-11-05T08:32:39.278 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2023-11-05T08:32:39.278 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2023-11-05T08:32:39.278 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_xfs_extsize": "false", 2023-11-05T08:32:39.278 INFO:teuthology.orchestra.run.smithi002.stdout: "filestore_zfs_snap": "false", 2023-11-05T08:32:39.279 INFO:teuthology.orchestra.run.smithi002.stdout: "fio_dir": "/tmp/fio", 2023-11-05T08:32:39.279 INFO:teuthology.orchestra.run.smithi002.stdout: "fsid": "8f1fa9b5-06de-4da7-985f-30d4ad103e2d", 2023-11-05T08:32:39.279 INFO:teuthology.orchestra.run.smithi002.stdout: "fuse_allow_other": "true", 2023-11-05T08:32:39.279 INFO:teuthology.orchestra.run.smithi002.stdout: "fuse_atomic_o_trunc": "true", 2023-11-05T08:32:39.279 INFO:teuthology.orchestra.run.smithi002.stdout: "fuse_big_writes": "true", 2023-11-05T08:32:39.279 INFO:teuthology.orchestra.run.smithi002.stdout: "fuse_debug": "false", 2023-11-05T08:32:39.280 INFO:teuthology.orchestra.run.smithi002.stdout: "fuse_default_permissions": "false", 2023-11-05T08:32:39.280 INFO:teuthology.orchestra.run.smithi002.stdout: "fuse_disable_pagecache": "false", 2023-11-05T08:32:39.280 INFO:teuthology.orchestra.run.smithi002.stdout: "fuse_max_write": "0", 2023-11-05T08:32:39.280 INFO:teuthology.orchestra.run.smithi002.stdout: "fuse_multithreaded": "true", 2023-11-05T08:32:39.280 INFO:teuthology.orchestra.run.smithi002.stdout: "fuse_require_active_mds": "true", 2023-11-05T08:32:39.280 INFO:teuthology.orchestra.run.smithi002.stdout: "fuse_set_user_groups": "true", 2023-11-05T08:32:39.281 INFO:teuthology.orchestra.run.smithi002.stdout: "fuse_splice_move": "true", 2023-11-05T08:32:39.281 INFO:teuthology.orchestra.run.smithi002.stdout: "fuse_splice_read": "true", 2023-11-05T08:32:39.281 INFO:teuthology.orchestra.run.smithi002.stdout: "fuse_splice_write": "true", 2023-11-05T08:32:39.281 INFO:teuthology.orchestra.run.smithi002.stdout: "fuse_syncfs_on_mksnap": "true", 2023-11-05T08:32:39.281 INFO:teuthology.orchestra.run.smithi002.stdout: "fuse_use_invalidate_cb": "true", 2023-11-05T08:32:39.281 INFO:teuthology.orchestra.run.smithi002.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2023-11-05T08:32:39.282 INFO:teuthology.orchestra.run.smithi002.stdout: "gss_target_name": "ceph", 2023-11-05T08:32:39.282 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_file": "", 2023-11-05T08:32:39.282 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_inject_failure": "0", 2023-11-05T08:32:39.282 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_interval": "5", 2023-11-05T08:32:39.282 INFO:teuthology.orchestra.run.smithi002.stdout: "host": "smithi002", 2023-11-05T08:32:39.283 INFO:teuthology.orchestra.run.smithi002.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2023-11-05T08:32:39.283 INFO:teuthology.orchestra.run.smithi002.stdout: "immutable_object_cache_max_inflight_ops": "128", 2023-11-05T08:32:39.283 INFO:teuthology.orchestra.run.smithi002.stdout: "immutable_object_cache_max_size": "1073741824", 2023-11-05T08:32:39.283 INFO:teuthology.orchestra.run.smithi002.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2023-11-05T08:32:39.283 INFO:teuthology.orchestra.run.smithi002.stdout: "immutable_object_cache_qos_bps_burst": "0", 2023-11-05T08:32:39.283 INFO:teuthology.orchestra.run.smithi002.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2023-11-05T08:32:39.284 INFO:teuthology.orchestra.run.smithi002.stdout: "immutable_object_cache_qos_bps_limit": "0", 2023-11-05T08:32:39.284 INFO:teuthology.orchestra.run.smithi002.stdout: "immutable_object_cache_qos_iops_burst": "0", 2023-11-05T08:32:39.284 INFO:teuthology.orchestra.run.smithi002.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2023-11-05T08:32:39.284 INFO:teuthology.orchestra.run.smithi002.stdout: "immutable_object_cache_qos_iops_limit": "0", 2023-11-05T08:32:39.284 INFO:teuthology.orchestra.run.smithi002.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2023-11-05T08:32:39.285 INFO:teuthology.orchestra.run.smithi002.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2023-11-05T08:32:39.285 INFO:teuthology.orchestra.run.smithi002.stdout: "immutable_object_cache_watermark": "0.900000", 2023-11-05T08:32:39.285 INFO:teuthology.orchestra.run.smithi002.stdout: "inject_early_sigterm": "false", 2023-11-05T08:32:39.285 INFO:teuthology.orchestra.run.smithi002.stdout: "jaeger_agent_port": "6799", 2023-11-05T08:32:39.285 INFO:teuthology.orchestra.run.smithi002.stdout: "jaeger_tracing_enable": "false", 2023-11-05T08:32:39.285 INFO:teuthology.orchestra.run.smithi002.stdout: "journal_aio": "true", 2023-11-05T08:32:39.286 INFO:teuthology.orchestra.run.smithi002.stdout: "journal_align_min_size": "65536", 2023-11-05T08:32:39.286 INFO:teuthology.orchestra.run.smithi002.stdout: "journal_block_align": "true", 2023-11-05T08:32:39.286 INFO:teuthology.orchestra.run.smithi002.stdout: "journal_block_size": "4096", 2023-11-05T08:32:39.286 INFO:teuthology.orchestra.run.smithi002.stdout: "journal_dio": "true", 2023-11-05T08:32:39.286 INFO:teuthology.orchestra.run.smithi002.stdout: "journal_discard": "false", 2023-11-05T08:32:39.286 INFO:teuthology.orchestra.run.smithi002.stdout: "journal_force_aio": "false", 2023-11-05T08:32:39.287 INFO:teuthology.orchestra.run.smithi002.stdout: "journal_ignore_corruption": "false", 2023-11-05T08:32:39.287 INFO:teuthology.orchestra.run.smithi002.stdout: "journal_max_write_bytes": "10485760", 2023-11-05T08:32:39.287 INFO:teuthology.orchestra.run.smithi002.stdout: "journal_max_write_entries": "100", 2023-11-05T08:32:39.287 INFO:teuthology.orchestra.run.smithi002.stdout: "journal_replay_from": "0", 2023-11-05T08:32:39.287 INFO:teuthology.orchestra.run.smithi002.stdout: "journal_throttle_high_multiple": "0.000000", 2023-11-05T08:32:39.288 INFO:teuthology.orchestra.run.smithi002.stdout: "journal_throttle_high_threshhold": "0.900000", 2023-11-05T08:32:39.288 INFO:teuthology.orchestra.run.smithi002.stdout: "journal_throttle_low_threshhold": "0.600000", 2023-11-05T08:32:39.288 INFO:teuthology.orchestra.run.smithi002.stdout: "journal_throttle_max_multiple": "0.000000", 2023-11-05T08:32:39.288 INFO:teuthology.orchestra.run.smithi002.stdout: "journal_write_header_frequency": "0", 2023-11-05T08:32:39.288 INFO:teuthology.orchestra.run.smithi002.stdout: "journal_zero_on_create": "false", 2023-11-05T08:32:39.289 INFO:teuthology.orchestra.run.smithi002.stdout: "journaler_prefetch_periods": "10", 2023-11-05T08:32:39.289 INFO:teuthology.orchestra.run.smithi002.stdout: "journaler_prezero_periods": "5", 2023-11-05T08:32:39.289 INFO:teuthology.orchestra.run.smithi002.stdout: "journaler_write_head_interval": "15", 2023-11-05T08:32:39.289 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "", 2023-11-05T08:32:39.290 INFO:teuthology.orchestra.run.smithi002.stdout: "keyfile": "", 2023-11-05T08:32:39.290 INFO:teuthology.orchestra.run.smithi002.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2023-11-05T08:32:39.290 INFO:teuthology.orchestra.run.smithi002.stdout: "kstore_backend": "rocksdb", 2023-11-05T08:32:39.290 INFO:teuthology.orchestra.run.smithi002.stdout: "kstore_default_stripe_size": "65536", 2023-11-05T08:32:39.290 INFO:teuthology.orchestra.run.smithi002.stdout: "kstore_fsck_on_mount": "false", 2023-11-05T08:32:39.291 INFO:teuthology.orchestra.run.smithi002.stdout: "kstore_fsck_on_mount_deep": "true", 2023-11-05T08:32:39.291 INFO:teuthology.orchestra.run.smithi002.stdout: "kstore_max_bytes": "67108864", 2023-11-05T08:32:39.291 INFO:teuthology.orchestra.run.smithi002.stdout: "kstore_max_ops": "512", 2023-11-05T08:32:39.291 INFO:teuthology.orchestra.run.smithi002.stdout: "kstore_nid_prealloc": "1024", 2023-11-05T08:32:39.291 INFO:teuthology.orchestra.run.smithi002.stdout: "kstore_onode_map_size": "1024", 2023-11-05T08:32:39.291 INFO:teuthology.orchestra.run.smithi002.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2023-11-05T08:32:39.292 INFO:teuthology.orchestra.run.smithi002.stdout: "kstore_sync_submit_transaction": "false", 2023-11-05T08:32:39.292 INFO:teuthology.orchestra.run.smithi002.stdout: "kstore_sync_transaction": "false", 2023-11-05T08:32:39.292 INFO:teuthology.orchestra.run.smithi002.stdout: "librados_thread_count": "2", 2023-11-05T08:32:39.292 INFO:teuthology.orchestra.run.smithi002.stdout: "lockdep": "false", 2023-11-05T08:32:39.292 INFO:teuthology.orchestra.run.smithi002.stdout: "lockdep_force_backtrace": "false", 2023-11-05T08:32:39.293 INFO:teuthology.orchestra.run.smithi002.stdout: "log_coarse_timestamps": "true", 2023-11-05T08:32:39.293 INFO:teuthology.orchestra.run.smithi002.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2023-11-05T08:32:39.293 INFO:teuthology.orchestra.run.smithi002.stdout: "log_flush_on_exit": "false", 2023-11-05T08:32:39.293 INFO:teuthology.orchestra.run.smithi002.stdout: "log_graylog_host": "127.0.0.1", 2023-11-05T08:32:39.293 INFO:teuthology.orchestra.run.smithi002.stdout: "log_graylog_port": "12201", 2023-11-05T08:32:39.294 INFO:teuthology.orchestra.run.smithi002.stdout: "log_max_new": "1000", 2023-11-05T08:32:39.294 INFO:teuthology.orchestra.run.smithi002.stdout: "log_max_recent": "10000", 2023-11-05T08:32:39.294 INFO:teuthology.orchestra.run.smithi002.stdout: "log_stderr_prefix": "", 2023-11-05T08:32:39.294 INFO:teuthology.orchestra.run.smithi002.stdout: "log_stop_at_utilization": "0.970000", 2023-11-05T08:32:39.294 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": "true", 2023-11-05T08:32:39.295 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_graylog": "false", 2023-11-05T08:32:39.295 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_journald": "false", 2023-11-05T08:32:39.295 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_stderr": "false", 2023-11-05T08:32:39.295 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_syslog": "false", 2023-11-05T08:32:39.295 INFO:teuthology.orchestra.run.smithi002.stdout: "max_rotating_auth_attempts": "10", 2023-11-05T08:32:39.295 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2023-11-05T08:32:39.296 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_action_on_write_error": "1", 2023-11-05T08:32:39.296 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_alternate_name_max": "8192", 2023-11-05T08:32:39.296 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_asio_thread_count": "2", 2023-11-05T08:32:39.296 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_bal_export_pin": "true", 2023-11-05T08:32:39.296 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_bal_fragment_dirs": "true", 2023-11-05T08:32:39.297 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2023-11-05T08:32:39.297 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_bal_fragment_interval": "5", 2023-11-05T08:32:39.297 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_bal_fragment_size_max": "100000", 2023-11-05T08:32:39.297 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_bal_idle_threshold": "0.000000", 2023-11-05T08:32:39.297 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_bal_interval": "10", 2023-11-05T08:32:39.297 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_bal_max": "-1", 2023-11-05T08:32:39.298 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_bal_max_until": "-1", 2023-11-05T08:32:39.298 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_bal_merge_size": "50", 2023-11-05T08:32:39.298 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_bal_midchunk": "0.300000", 2023-11-05T08:32:39.298 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_bal_min_rebalance": "0.100000", 2023-11-05T08:32:39.298 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_bal_min_start": "0.200000", 2023-11-05T08:32:39.299 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_bal_minchunk": "0.001000", 2023-11-05T08:32:39.299 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_bal_mode": "0", 2023-11-05T08:32:39.299 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_bal_need_max": "1.200000", 2023-11-05T08:32:39.299 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_bal_need_min": "0.800000", 2023-11-05T08:32:39.299 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_bal_overload_epochs": "2", 2023-11-05T08:32:39.300 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_bal_replicate_threshold": "8000.000000", 2023-11-05T08:32:39.300 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_bal_sample_interval": "3.000000", 2023-11-05T08:32:39.300 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_bal_split_bits": "3", 2023-11-05T08:32:39.300 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_bal_split_rd": "25000.000000", 2023-11-05T08:32:39.300 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_bal_split_size": "10000", 2023-11-05T08:32:39.300 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_bal_split_wr": "10000.000000", 2023-11-05T08:32:39.301 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_bal_target_decay": "10.000000", 2023-11-05T08:32:39.301 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2023-11-05T08:32:39.301 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_beacon_grace": "15.000000", 2023-11-05T08:32:39.301 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_beacon_interval": "4.000000", 2023-11-05T08:32:39.301 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_beacon_mon_down_grace": "60", 2023-11-05T08:32:39.302 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_cache_memory_limit": "4294967296", 2023-11-05T08:32:39.302 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_cache_mid": "0.700000", 2023-11-05T08:32:39.302 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_cache_release_free_interval": "10", 2023-11-05T08:32:39.302 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_cache_reservation": "0.050000", 2023-11-05T08:32:39.302 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_cache_trim_decay_rate": "1.000000", 2023-11-05T08:32:39.303 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_cache_trim_interval": "1", 2023-11-05T08:32:39.303 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_cache_trim_threshold": "262144", 2023-11-05T08:32:39.303 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2023-11-05T08:32:39.303 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2023-11-05T08:32:39.303 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_client_delegate_inos_pct": "50", 2023-11-05T08:32:39.304 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_client_prealloc_inos": "1000", 2023-11-05T08:32:39.304 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2023-11-05T08:32:39.304 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_connect_bootstrapping": "false", 2023-11-05T08:32:39.304 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_damage_table_max_entries": "10000", 2023-11-05T08:32:39.304 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2023-11-05T08:32:39.305 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_debug_auth_pins": "false", 2023-11-05T08:32:39.305 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_debug_frag": "false", 2023-11-05T08:32:39.305 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_debug_scatterstat": "false", 2023-11-05T08:32:39.305 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_debug_subtrees": "false", 2023-11-05T08:32:39.305 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_decay_halflife": "5.000000", 2023-11-05T08:32:39.306 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_default_dir_hash": "2", 2023-11-05T08:32:39.306 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_defer_session_stale": "true", 2023-11-05T08:32:39.306 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_deny_all_reconnect": "false", 2023-11-05T08:32:39.306 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_dir_keys_per_op": "16384", 2023-11-05T08:32:39.306 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_dir_max_commit_size": "10", 2023-11-05T08:32:39.307 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_dir_max_entries": "0", 2023-11-05T08:32:39.307 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_dir_prefetch": "true", 2023-11-05T08:32:39.307 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_dirstat_min_interval": "1.000000", 2023-11-05T08:32:39.307 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_dump_cache_after_rejoin": "false", 2023-11-05T08:32:39.307 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_dump_cache_on_map": "false", 2023-11-05T08:32:39.308 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_dump_cache_threshold_file": "0", 2023-11-05T08:32:39.308 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2023-11-05T08:32:39.308 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_early_reply": "true", 2023-11-05T08:32:39.308 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_enable_op_tracker": "true", 2023-11-05T08:32:39.308 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_enforce_unique_name": "true", 2023-11-05T08:32:39.308 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_export_ephemeral_distributed": "true", 2023-11-05T08:32:39.309 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2023-11-05T08:32:39.309 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_export_ephemeral_random": "true", 2023-11-05T08:32:39.309 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_export_ephemeral_random_max": "0.010000", 2023-11-05T08:32:39.309 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_extraordinary_events_dump_interval": "0", 2023-11-05T08:32:39.309 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_forward_all_requests_to_auth": "false", 2023-11-05T08:32:39.309 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_freeze_tree_timeout": "30.000000", 2023-11-05T08:32:39.310 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_fscrypt_last_block_max_size": "4096", 2023-11-05T08:32:39.310 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_go_bad_corrupt_dentry": "true", 2023-11-05T08:32:39.310 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2023-11-05T08:32:39.310 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_health_cache_threshold": "1.500000", 2023-11-05T08:32:39.310 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_health_summarize_threshold": "10", 2023-11-05T08:32:39.310 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_heartbeat_grace": "15.000000", 2023-11-05T08:32:39.311 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_heartbeat_reset_grace": "1000", 2023-11-05T08:32:39.311 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_inject_health_dummy": "false", 2023-11-05T08:32:39.311 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2023-11-05T08:32:39.311 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_inject_migrator_session_race": "false", 2023-11-05T08:32:39.311 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2023-11-05T08:32:39.312 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_inject_skip_replaying_inotable": "false", 2023-11-05T08:32:39.312 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2023-11-05T08:32:39.312 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_join_fs": "", 2023-11-05T08:32:39.312 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_journal_format": "1", 2023-11-05T08:32:39.312 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_kill_create_at": "0", 2023-11-05T08:32:39.312 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_kill_export_at": "0", 2023-11-05T08:32:39.313 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_kill_import_at": "0", 2023-11-05T08:32:39.313 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_kill_journal_at": "0", 2023-11-05T08:32:39.313 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_kill_journal_expire_at": "0", 2023-11-05T08:32:39.313 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_kill_journal_replay_at": "0", 2023-11-05T08:32:39.313 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_kill_link_at": "0", 2023-11-05T08:32:39.314 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_kill_mdstable_at": "0", 2023-11-05T08:32:39.314 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_kill_openc_at": "0", 2023-11-05T08:32:39.314 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_kill_rename_at": "0", 2023-11-05T08:32:39.314 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_kill_shutdown_at": "0", 2023-11-05T08:32:39.314 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_kill_skip_replaying_inotable": "false", 2023-11-05T08:32:39.315 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_log_event_large_threshold": "524288", 2023-11-05T08:32:39.315 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_log_events_per_segment": "1024", 2023-11-05T08:32:39.315 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_log_major_segment_event_ratio": "12", 2023-11-05T08:32:39.315 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_log_max_events": "-1", 2023-11-05T08:32:39.315 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_log_max_segments": "128", 2023-11-05T08:32:39.315 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_log_pause": "false", 2023-11-05T08:32:39.316 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_log_segment_size": "0", 2023-11-05T08:32:39.316 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_log_skip_corrupt_events": "false", 2023-11-05T08:32:39.316 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_log_skip_unbounded_events": "false", 2023-11-05T08:32:39.316 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_log_warn_factor": "2.000000", 2023-11-05T08:32:39.316 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_max_caps_per_client": "1048576", 2023-11-05T08:32:39.317 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_max_completed_flushes": "100000", 2023-11-05T08:32:39.317 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_max_completed_requests": "100000", 2023-11-05T08:32:39.317 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_max_export_size": "20971520", 2023-11-05T08:32:39.317 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_max_file_recover": "32", 2023-11-05T08:32:39.317 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_max_purge_files": "64", 2023-11-05T08:32:39.317 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_max_purge_ops": "8192", 2023-11-05T08:32:39.318 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2023-11-05T08:32:39.318 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_max_scrub_ops_in_progress": "5", 2023-11-05T08:32:39.318 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_max_snaps_per_dir": "100", 2023-11-05T08:32:39.318 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_metrics_update_interval": "2", 2023-11-05T08:32:39.318 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_min_caps_per_client": "100", 2023-11-05T08:32:39.319 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_min_caps_working_set": "10000", 2023-11-05T08:32:39.319 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_mon_shutdown_timeout": "5.000000", 2023-11-05T08:32:39.319 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_numa_node": "-1", 2023-11-05T08:32:39.319 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_oft_prefetch_dirfrags": "false", 2023-11-05T08:32:39.319 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_op_complaint_time": "30.000000", 2023-11-05T08:32:39.319 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_op_history_duration": "600", 2023-11-05T08:32:39.320 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_op_history_size": "20", 2023-11-05T08:32:39.320 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_op_history_slow_op_size": "20", 2023-11-05T08:32:39.320 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_op_history_slow_op_threshold": "10", 2023-11-05T08:32:39.320 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_op_log_threshold": "5", 2023-11-05T08:32:39.320 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_ping_grace": "15", 2023-11-05T08:32:39.320 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_ping_interval": "5", 2023-11-05T08:32:39.321 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2023-11-05T08:32:39.321 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_recall_global_max_decay_threshold": "131072", 2023-11-05T08:32:39.321 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_recall_max_caps": "30000", 2023-11-05T08:32:39.321 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_recall_max_decay_rate": "1.500000", 2023-11-05T08:32:39.321 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_recall_max_decay_threshold": "131072", 2023-11-05T08:32:39.322 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_recall_warning_decay_rate": "60.000000", 2023-11-05T08:32:39.322 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_recall_warning_threshold": "262144", 2023-11-05T08:32:39.322 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_reconnect_timeout": "45.000000", 2023-11-05T08:32:39.322 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_replay_interval": "1.000000", 2023-11-05T08:32:39.322 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_replay_unsafe_with_closed_session": "false", 2023-11-05T08:32:39.323 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_request_load_average_decay_rate": "60.000000", 2023-11-05T08:32:39.323 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_root_ino_gid": "0", 2023-11-05T08:32:39.323 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_root_ino_uid": "0", 2023-11-05T08:32:39.323 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_scatter_nudge_interval": "5.000000", 2023-11-05T08:32:39.323 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_session_blocklist_on_evict": "true", 2023-11-05T08:32:39.324 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_session_blocklist_on_timeout": "true", 2023-11-05T08:32:39.324 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2023-11-05T08:32:39.324 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_session_cache_liveness_magnitude": "10", 2023-11-05T08:32:39.324 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2023-11-05T08:32:39.324 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_session_cap_acquisition_throttle": "100000", 2023-11-05T08:32:39.325 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2023-11-05T08:32:39.325 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_session_metadata_threshold": "16777216", 2023-11-05T08:32:39.325 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_sessionmap_keys_per_op": "1024", 2023-11-05T08:32:39.325 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_shutdown_check": "0", 2023-11-05T08:32:39.325 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_skip_ino": "0", 2023-11-05T08:32:39.326 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_sleep_rank_change": "0.000000", 2023-11-05T08:32:39.326 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_snap_max_uid": "4294967294", 2023-11-05T08:32:39.326 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_snap_min_uid": "0", 2023-11-05T08:32:39.326 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_snap_rstat": "false", 2023-11-05T08:32:39.326 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_standby_replay_damaged": "false", 2023-11-05T08:32:39.327 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_symlink_recovery": "true", 2023-11-05T08:32:39.327 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_task_status_update_interval": "2.000000", 2023-11-05T08:32:39.327 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_thrash_exports": "0", 2023-11-05T08:32:39.327 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_thrash_fragments": "0", 2023-11-05T08:32:39.327 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_tick_interval": "5.000000", 2023-11-05T08:32:39.327 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_valgrind_exit": "false", 2023-11-05T08:32:39.328 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_verify_backtrace": "1", 2023-11-05T08:32:39.328 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_verify_scatter": "false", 2023-11-05T08:32:39.328 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_wipe_ino_prealloc": "false", 2023-11-05T08:32:39.328 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_wipe_sessions": "false", 2023-11-05T08:32:39.328 INFO:teuthology.orchestra.run.smithi002.stdout: "mempool_debug": "false", 2023-11-05T08:32:39.328 INFO:teuthology.orchestra.run.smithi002.stdout: "memstore_debug_omit_block_device_write": "false", 2023-11-05T08:32:39.329 INFO:teuthology.orchestra.run.smithi002.stdout: "memstore_device_bytes": "1073741824", 2023-11-05T08:32:39.329 INFO:teuthology.orchestra.run.smithi002.stdout: "memstore_page_set": "false", 2023-11-05T08:32:39.329 INFO:teuthology.orchestra.run.smithi002.stdout: "memstore_page_size": "65536", 2023-11-05T08:32:39.329 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr_client_bytes": "134217728", 2023-11-05T08:32:39.329 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr_client_messages": "512", 2023-11-05T08:32:39.329 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2023-11-05T08:32:39.330 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr_connect_retry_interval": "1.000000", 2023-11-05T08:32:39.330 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2023-11-05T08:32:39.330 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2023-11-05T08:32:39.330 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr_disabled_modules": "", 2023-11-05T08:32:39.331 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr_initial_modules": "restful iostat nfs", 2023-11-05T08:32:39.331 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr_max_pg_num_change": "128", 2023-11-05T08:32:39.331 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr_mds_bytes": "134217728", 2023-11-05T08:32:39.331 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr_mds_messages": "128", 2023-11-05T08:32:39.331 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2023-11-05T08:32:39.331 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr_mon_bytes": "134217728", 2023-11-05T08:32:39.332 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr_mon_messages": "128", 2023-11-05T08:32:39.332 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr_osd_bytes": "536870912", 2023-11-05T08:32:39.332 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr_osd_messages": "8192", 2023-11-05T08:32:39.332 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr_pool": "true", 2023-11-05T08:32:39.332 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr_service_beacon_grace": "60.000000", 2023-11-05T08:32:39.333 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr_standby_modules": "true", 2023-11-05T08:32:39.333 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr_stats_period": "5", 2023-11-05T08:32:39.333 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr_stats_threshold": "5", 2023-11-05T08:32:39.333 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr_tick_period": "2", 2023-11-05T08:32:39.333 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr_ttl_cache_expire_seconds": "0", 2023-11-05T08:32:39.334 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_accept_timeout_factor": "2.000000", 2023-11-05T08:32:39.334 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_allow_pool_delete": "true", 2023-11-05T08:32:39.334 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_allow_pool_size_one": "true", 2023-11-05T08:32:39.334 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_auth_validate_all_caps": "true", 2023-11-05T08:32:39.334 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2023-11-05T08:32:39.335 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2023-11-05T08:32:39.335 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_client_bytes": "104857600", 2023-11-05T08:32:39.335 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_client_directed_command_retry": "5", 2023-11-05T08:32:39.335 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_client_hunt_interval": "3.000000", 2023-11-05T08:32:39.335 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2023-11-05T08:32:39.336 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2023-11-05T08:32:39.336 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2023-11-05T08:32:39.336 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_client_hunt_parallel": "3", 2023-11-05T08:32:39.336 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_client_log_interval": "1.000000", 2023-11-05T08:32:39.336 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_client_max_log_entries_per_message": "1000", 2023-11-05T08:32:39.337 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_client_ping_interval": "10.000000", 2023-11-05T08:32:39.337 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_client_ping_timeout": "30.000000", 2023-11-05T08:32:39.337 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_client_target_rank": "-1", 2023-11-05T08:32:39.337 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_clock_drift_allowed": "1.000000", 2023-11-05T08:32:39.337 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2023-11-05T08:32:39.337 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2023-11-05T08:32:39.338 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_cluster_log_file_level": "debug", 2023-11-05T08:32:39.338 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_cluster_log_to_file": "true", 2023-11-05T08:32:39.338 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_cluster_log_to_graylog": "false", 2023-11-05T08:32:39.338 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2023-11-05T08:32:39.338 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_cluster_log_to_graylog_port": "12201", 2023-11-05T08:32:39.339 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_cluster_log_to_journald": "false", 2023-11-05T08:32:39.339 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_cluster_log_to_stderr": "false", 2023-11-05T08:32:39.339 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_cluster_log_to_syslog": "default=false", 2023-11-05T08:32:39.339 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2023-11-05T08:32:39.339 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_cluster_log_to_syslog_level": "info", 2023-11-05T08:32:39.340 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_compact_on_bootstrap": "false", 2023-11-05T08:32:39.340 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_compact_on_start": "false", 2023-11-05T08:32:39.340 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_compact_on_trim": "true", 2023-11-05T08:32:39.340 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_con_tracker_persist_interval": "10", 2023-11-05T08:32:39.340 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_con_tracker_score_halflife": "43200", 2023-11-05T08:32:39.341 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_config_key_max_entry_size": "65536", 2023-11-05T08:32:39.341 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_cpu_threads": "4", 2023-11-05T08:32:39.341 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_crush_min_required_version": "hammer", 2023-11-05T08:32:39.341 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_daemon_bytes": "419430400", 2023-11-05T08:32:39.341 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2023-11-05T08:32:39.341 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_data_avail_crit": "5", 2023-11-05T08:32:39.342 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_data_avail_warn": "5", 2023-11-05T08:32:39.342 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_data_size_warn": "16106127360", 2023-11-05T08:32:39.342 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_debug_block_osdmap_trim": "false", 2023-11-05T08:32:39.342 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_debug_deprecated_as_obsolete": "false", 2023-11-05T08:32:39.342 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_debug_dump_json": "false", 2023-11-05T08:32:39.342 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2023-11-05T08:32:39.343 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_debug_dump_transactions": "false", 2023-11-05T08:32:39.343 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_debug_extra_checks": "false", 2023-11-05T08:32:39.343 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_debug_no_initial_persistent_features": "false", 2023-11-05T08:32:39.343 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2023-11-05T08:32:39.343 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_debug_no_require_quincy": "false", 2023-11-05T08:32:39.344 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_debug_no_require_reef": "false", 2023-11-05T08:32:39.344 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2023-11-05T08:32:39.344 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_delta_reset_interval": "10.000000", 2023-11-05T08:32:39.344 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_dns_srv_name": "ceph-mon", 2023-11-05T08:32:39.344 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_down_mkfs_grace": "120", 2023-11-05T08:32:39.345 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_election_default_strategy": "3", 2023-11-05T08:32:39.345 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_election_timeout": "5.000000", 2023-11-05T08:32:39.345 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2023-11-05T08:32:39.345 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_elector_ping_divisor": "2", 2023-11-05T08:32:39.345 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_elector_ping_timeout": "2.000000", 2023-11-05T08:32:39.345 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_enable_op_tracker": "true", 2023-11-05T08:32:39.346 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_fake_pool_delete": "false", 2023-11-05T08:32:39.346 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_force_quorum_join": "false", 2023-11-05T08:32:39.346 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_globalid_prealloc": "10000", 2023-11-05T08:32:39.346 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_health_detail_to_clog": "false", 2023-11-05T08:32:39.346 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_health_log_update_period": "5", 2023-11-05T08:32:39.347 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_health_max_detail": "50", 2023-11-05T08:32:39.347 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_health_to_clog": "true", 2023-11-05T08:32:39.347 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_health_to_clog_interval": "600", 2023-11-05T08:32:39.347 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2023-11-05T08:32:39.347 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_host": "172.21.15.2,172.21.15.53,172.21.15.69", 2023-11-05T08:32:39.348 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_host_override": "", 2023-11-05T08:32:39.348 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_initial_members": "", 2023-11-05T08:32:39.348 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2023-11-05T08:32:39.348 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2023-11-05T08:32:39.349 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_inject_transaction_delay_max": "10.000000", 2023-11-05T08:32:39.349 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2023-11-05T08:32:39.349 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_keyvaluedb": "rocksdb", 2023-11-05T08:32:39.349 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_lease": "5.000000", 2023-11-05T08:32:39.349 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2023-11-05T08:32:39.350 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_lease_renew_interval_factor": "0.600000", 2023-11-05T08:32:39.350 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_log_full_interval": "50", 2023-11-05T08:32:39.350 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_log_max": "10000", 2023-11-05T08:32:39.350 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_log_max_summary": "50", 2023-11-05T08:32:39.351 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_max_log_entries_per_event": "4096", 2023-11-05T08:32:39.351 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_max_log_epochs": "500", 2023-11-05T08:32:39.351 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_max_mdsmap_epochs": "500", 2023-11-05T08:32:39.351 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_max_mgrmap_epochs": "500", 2023-11-05T08:32:39.351 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_max_osd": "10000", 2023-11-05T08:32:39.352 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_max_pg_per_osd": "10000", 2023-11-05T08:32:39.352 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_max_pool_pg_num": "65536", 2023-11-05T08:32:39.352 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_max_snap_prune_per_epoch": "100", 2023-11-05T08:32:39.352 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_mds_blocklist_interval": "86400.000000", 2023-11-05T08:32:39.353 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_mds_force_trim_to": "0", 2023-11-05T08:32:39.353 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_mds_skip_sanity": "false", 2023-11-05T08:32:39.353 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_memory_autotune": "true", 2023-11-05T08:32:39.353 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_memory_target": "2147483648", 2023-11-05T08:32:39.353 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_mgr_beacon_grace": "30", 2023-11-05T08:32:39.354 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2023-11-05T08:32:39.354 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_mgr_digest_period": "5", 2023-11-05T08:32:39.354 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_mgr_inactive_grace": "60", 2023-11-05T08:32:39.354 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_mgr_mkfs_grace": "240", 2023-11-05T08:32:39.354 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2023-11-05T08:32:39.355 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_min_osdmap_epochs": "500", 2023-11-05T08:32:39.355 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_op_complaint_time": "30", 2023-11-05T08:32:39.355 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_op_history_duration": "600", 2023-11-05T08:32:39.355 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_op_history_size": "20", 2023-11-05T08:32:39.355 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_op_history_slow_op_size": "20", 2023-11-05T08:32:39.356 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_op_history_slow_op_threshold": "10", 2023-11-05T08:32:39.356 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_op_log_threshold": "5", 2023-11-05T08:32:39.356 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_osd_adjust_down_out_interval": "true", 2023-11-05T08:32:39.356 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2023-11-05T08:32:39.356 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2023-11-05T08:32:39.357 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_osd_auto_mark_in": "false", 2023-11-05T08:32:39.357 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_osd_auto_mark_new_in": "true", 2023-11-05T08:32:39.357 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2023-11-05T08:32:39.357 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2023-11-05T08:32:39.357 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_osd_cache_size": "500", 2023-11-05T08:32:39.358 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_osd_cache_size_min": "134217728", 2023-11-05T08:32:39.358 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_osd_crush_smoke_test": "true", 2023-11-05T08:32:39.358 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_osd_destroyed_out_interval": "600", 2023-11-05T08:32:39.358 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_osd_down_out_interval": "600", 2023-11-05T08:32:39.358 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_osd_down_out_subtree_limit": "rack", 2023-11-05T08:32:39.359 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2023-11-05T08:32:39.359 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_osd_force_trim_to": "0", 2023-11-05T08:32:39.359 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_osd_full_ratio": "0.950000", 2023-11-05T08:32:39.359 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2023-11-05T08:32:39.359 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_osd_laggy_halflife": "3600", 2023-11-05T08:32:39.360 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_osd_laggy_max_interval": "300", 2023-11-05T08:32:39.360 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_osd_laggy_weight": "0.300000", 2023-11-05T08:32:39.360 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2023-11-05T08:32:39.360 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_osd_max_creating_pgs": "1024", 2023-11-05T08:32:39.360 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_osd_max_initial_pgs": "1024", 2023-11-05T08:32:39.361 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_osd_min_down_reporters": "2", 2023-11-05T08:32:39.361 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_osd_min_in_ratio": "0.750000", 2023-11-05T08:32:39.361 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_osd_min_up_ratio": "0.300000", 2023-11-05T08:32:39.361 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_osd_nearfull_ratio": "0.850000", 2023-11-05T08:32:39.361 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_osd_prime_pg_temp": "true", 2023-11-05T08:32:39.361 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2023-11-05T08:32:39.362 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2023-11-05T08:32:39.362 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_osd_report_timeout": "900", 2023-11-05T08:32:39.362 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_osd_reporter_subtree_level": "osd", 2023-11-05T08:32:39.362 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2023-11-05T08:32:39.362 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_osd_warn_num_repaired": "10", 2023-11-05T08:32:39.363 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_osd_warn_op_age": "32.000000", 2023-11-05T08:32:39.363 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_osdmap_full_prune_enabled": "true", 2023-11-05T08:32:39.363 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_osdmap_full_prune_interval": "10", 2023-11-05T08:32:39.363 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_osdmap_full_prune_min": "10000", 2023-11-05T08:32:39.363 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_osdmap_full_prune_txsize": "100", 2023-11-05T08:32:39.364 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2023-11-05T08:32:39.364 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_pg_stuck_threshold": "60", 2023-11-05T08:32:39.364 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2023-11-05T08:32:39.364 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_pg_warn_min_objects": "10000", 2023-11-05T08:32:39.364 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_pg_warn_min_per_osd": "0", 2023-11-05T08:32:39.365 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_pg_warn_min_pool_objects": "1000", 2023-11-05T08:32:39.365 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_pool_quota_crit_threshold": "0", 2023-11-05T08:32:39.365 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_pool_quota_warn_threshold": "0", 2023-11-05T08:32:39.365 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_probe_timeout": "2.000000", 2023-11-05T08:32:39.365 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_reweight_max_change": "0.050000", 2023-11-05T08:32:39.365 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_reweight_max_osds": "4", 2023-11-05T08:32:39.366 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_reweight_min_bytes_per_osd": "10", 2023-11-05T08:32:39.366 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_reweight_min_pgs_per_osd": "4", 2023-11-05T08:32:39.366 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2023-11-05T08:32:39.366 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2023-11-05T08:32:39.366 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2023-11-05T08:32:39.367 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_scrub_interval": "300", 2023-11-05T08:32:39.367 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_scrub_max_keys": "100", 2023-11-05T08:32:39.367 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_scrub_timeout": "300", 2023-11-05T08:32:39.367 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_session_timeout": "300", 2023-11-05T08:32:39.367 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_smart_report_timeout": "5", 2023-11-05T08:32:39.368 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_stat_smooth_intervals": "6", 2023-11-05T08:32:39.368 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2023-11-05T08:32:39.368 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_stretch_pool_min_size": "2", 2023-11-05T08:32:39.368 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_stretch_pool_size": "4", 2023-11-05T08:32:39.368 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2023-11-05T08:32:39.369 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_subscribe_interval": "86400.000000", 2023-11-05T08:32:39.369 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_sync_debug": "false", 2023-11-05T08:32:39.369 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_sync_max_payload_keys": "2000", 2023-11-05T08:32:39.369 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_sync_max_payload_size": "1048576", 2023-11-05T08:32:39.369 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_sync_provider_kill_at": "0", 2023-11-05T08:32:39.370 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_sync_requester_kill_at": "0", 2023-11-05T08:32:39.370 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_sync_timeout": "60.000000", 2023-11-05T08:32:39.370 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_target_pg_per_osd": "100", 2023-11-05T08:32:39.370 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_tick_interval": "5", 2023-11-05T08:32:39.371 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_timecheck_interval": "300.000000", 2023-11-05T08:32:39.371 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_timecheck_skew_interval": "30.000000", 2023-11-05T08:32:39.371 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_use_min_delay_socket": "false", 2023-11-05T08:32:39.371 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_warn_older_version_delay": "604800", 2023-11-05T08:32:39.371 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2023-11-05T08:32:39.372 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2023-11-05T08:32:39.372 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2023-11-05T08:32:39.372 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_warn_on_filestore_osds": "false", 2023-11-05T08:32:39.372 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2023-11-05T08:32:39.372 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2023-11-05T08:32:39.373 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2023-11-05T08:32:39.373 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_warn_on_misplaced": "false", 2023-11-05T08:32:39.373 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2023-11-05T08:32:39.373 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_warn_on_older_version": "true", 2023-11-05T08:32:39.373 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2023-11-05T08:32:39.373 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_warn_on_pool_no_app": "true", 2023-11-05T08:32:39.374 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_warn_on_pool_no_redundancy": "false", 2023-11-05T08:32:39.374 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2023-11-05T08:32:39.374 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2023-11-05T08:32:39.374 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2023-11-05T08:32:39.374 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_warn_on_too_few_osds": "false", 2023-11-05T08:32:39.375 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2023-11-05T08:32:39.375 INFO:teuthology.orchestra.run.smithi002.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2023-11-05T08:32:39.375 INFO:teuthology.orchestra.run.smithi002.stdout: "monmap": "", 2023-11-05T08:32:39.375 INFO:teuthology.orchestra.run.smithi002.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2023-11-05T08:32:39.375 INFO:teuthology.orchestra.run.smithi002.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2023-11-05T08:32:39.376 INFO:teuthology.orchestra.run.smithi002.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2023-11-05T08:32:39.376 INFO:teuthology.orchestra.run.smithi002.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2023-11-05T08:32:39.376 INFO:teuthology.orchestra.run.smithi002.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2023-11-05T08:32:39.376 INFO:teuthology.orchestra.run.smithi002.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2023-11-05T08:32:39.376 INFO:teuthology.orchestra.run.smithi002.stdout: "motr_tracing_enabled": "false", 2023-11-05T08:32:39.377 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_async_op_threads": "3", 2023-11-05T08:32:39.377 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_async_rdma_buffer_size": "131072", 2023-11-05T08:32:39.377 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_async_rdma_cm": "false", 2023-11-05T08:32:39.377 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_async_rdma_device_name": "", 2023-11-05T08:32:39.377 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_async_rdma_dscp": "96", 2023-11-05T08:32:39.378 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_async_rdma_enable_hugepage": "false", 2023-11-05T08:32:39.378 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_async_rdma_gid_idx": "0", 2023-11-05T08:32:39.378 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_async_rdma_local_gid": "", 2023-11-05T08:32:39.378 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_async_rdma_polling_us": "1000", 2023-11-05T08:32:39.378 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_async_rdma_port_num": "1", 2023-11-05T08:32:39.379 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_async_rdma_receive_buffers": "32768", 2023-11-05T08:32:39.379 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_async_rdma_receive_queue_len": "4096", 2023-11-05T08:32:39.379 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_async_rdma_roce_ver": "1", 2023-11-05T08:32:39.379 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_async_rdma_send_buffers": "1024", 2023-11-05T08:32:39.379 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_async_rdma_sl": "3", 2023-11-05T08:32:39.380 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_async_rdma_support_srq": "true", 2023-11-05T08:32:39.380 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_async_rdma_type": "ib", 2023-11-05T08:32:39.380 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_async_reap_threshold": "5", 2023-11-05T08:32:39.380 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_bind_before_connect": "false", 2023-11-05T08:32:39.380 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_bind_ipv4": "true", 2023-11-05T08:32:39.381 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_bind_ipv6": "false", 2023-11-05T08:32:39.381 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_bind_msgr1": "true", 2023-11-05T08:32:39.381 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_bind_msgr2": "true", 2023-11-05T08:32:39.381 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_bind_port_max": "7568", 2023-11-05T08:32:39.381 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_bind_port_min": "6800", 2023-11-05T08:32:39.382 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_bind_prefer_ipv4": "false", 2023-11-05T08:32:39.382 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_bind_retry_count": "3", 2023-11-05T08:32:39.382 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_bind_retry_delay": "5", 2023-11-05T08:32:39.382 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_blackhole_client": "false", 2023-11-05T08:32:39.382 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_blackhole_mds": "false", 2023-11-05T08:32:39.383 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_blackhole_mgr": "false", 2023-11-05T08:32:39.383 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_blackhole_mon": "false", 2023-11-05T08:32:39.383 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_blackhole_osd": "false", 2023-11-05T08:32:39.383 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_client_mode": "crc secure", 2023-11-05T08:32:39.383 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_cluster_mode": "crc secure", 2023-11-05T08:32:39.383 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_cluster_type": "", 2023-11-05T08:32:39.384 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_compress_secure": "false", 2023-11-05T08:32:39.384 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_connection_idle_timeout": "900", 2023-11-05T08:32:39.384 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_connection_ready_timeout": "10", 2023-11-05T08:32:39.384 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_crc_data": "true", 2023-11-05T08:32:39.384 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_crc_header": "true", 2023-11-05T08:32:39.385 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_die_on_bad_msg": "false", 2023-11-05T08:32:39.385 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_die_on_bug": "true", 2023-11-05T08:32:39.385 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_die_on_old_message": "true", 2023-11-05T08:32:39.385 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_die_on_skipped_message": "false", 2023-11-05T08:32:39.385 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_die_on_unhandled_msg": "false", 2023-11-05T08:32:39.385 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_dispatch_throttle_bytes": "104857600", 2023-11-05T08:32:39.386 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_dpdk_coremask": "0xF", 2023-11-05T08:32:39.386 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_dpdk_debug_allow_loopback": "false", 2023-11-05T08:32:39.386 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_dpdk_devs_allowlist": "", 2023-11-05T08:32:39.386 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_dpdk_enable_tso": "true", 2023-11-05T08:32:39.386 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2023-11-05T08:32:39.386 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_dpdk_host_ipv4_addr": "", 2023-11-05T08:32:39.387 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_dpdk_hugepages": "", 2023-11-05T08:32:39.387 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_dpdk_hw_flow_control": "true", 2023-11-05T08:32:39.387 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2023-11-05T08:32:39.387 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_dpdk_lro": "true", 2023-11-05T08:32:39.387 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_dpdk_memory_channel": "4", 2023-11-05T08:32:39.387 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2023-11-05T08:32:39.388 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_dpdk_pmd": "", 2023-11-05T08:32:39.388 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_dpdk_port_id": "0", 2023-11-05T08:32:39.388 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2023-11-05T08:32:39.388 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_dump_corrupt_message_level": "1", 2023-11-05T08:32:39.388 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_dump_on_send": "false", 2023-11-05T08:32:39.388 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_initial_backoff": "0.200000", 2023-11-05T08:32:39.389 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_inject_delay_max": "1.000000", 2023-11-05T08:32:39.389 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_inject_delay_probability": "0.000000", 2023-11-05T08:32:39.389 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_inject_delay_type": "", 2023-11-05T08:32:39.389 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_inject_internal_delays": "0.000000", 2023-11-05T08:32:39.389 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_inject_network_congestion": "0", 2023-11-05T08:32:39.389 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_inject_socket_failures": "5000", 2023-11-05T08:32:39.390 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_learn_addr_from_peer": "true", 2023-11-05T08:32:39.390 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_max_accept_failures": "4", 2023-11-05T08:32:39.390 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_max_backoff": "15.000000", 2023-11-05T08:32:39.390 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_mon_client_mode": "secure crc", 2023-11-05T08:32:39.390 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_mon_cluster_mode": "secure crc", 2023-11-05T08:32:39.391 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_mon_service_mode": "secure crc", 2023-11-05T08:32:39.391 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_osd_compress_min_size": "1024", 2023-11-05T08:32:39.391 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_osd_compress_mode": "none", 2023-11-05T08:32:39.391 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_osd_compression_algorithm": "snappy", 2023-11-05T08:32:39.391 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2023-11-05T08:32:39.391 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_pq_min_cost": "65536", 2023-11-05T08:32:39.392 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_public_type": "", 2023-11-05T08:32:39.392 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_service_mode": "crc secure", 2023-11-05T08:32:39.392 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_tcp_listen_backlog": "512", 2023-11-05T08:32:39.392 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_tcp_nodelay": "true", 2023-11-05T08:32:39.392 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_tcp_prefetch_max_size": "4096", 2023-11-05T08:32:39.392 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_tcp_rcvbuf": "0", 2023-11-05T08:32:39.393 INFO:teuthology.orchestra.run.smithi002.stdout: "ms_type": "async+posix", 2023-11-05T08:32:39.393 INFO:teuthology.orchestra.run.smithi002.stdout: "no_config_file": "false", 2023-11-05T08:32:39.393 INFO:teuthology.orchestra.run.smithi002.stdout: "objecter_completion_locks_per_session": "32", 2023-11-05T08:32:39.393 INFO:teuthology.orchestra.run.smithi002.stdout: "objecter_debug_inject_relock_delay": "false", 2023-11-05T08:32:39.393 INFO:teuthology.orchestra.run.smithi002.stdout: "objecter_inflight_op_bytes": "104857600", 2023-11-05T08:32:39.394 INFO:teuthology.orchestra.run.smithi002.stdout: "objecter_inflight_ops": "1024", 2023-11-05T08:32:39.394 INFO:teuthology.orchestra.run.smithi002.stdout: "objecter_inject_no_watch_ping": "false", 2023-11-05T08:32:39.394 INFO:teuthology.orchestra.run.smithi002.stdout: "objecter_retry_writes_after_first_reply": "false", 2023-11-05T08:32:39.394 INFO:teuthology.orchestra.run.smithi002.stdout: "objecter_tick_interval": "5.000000", 2023-11-05T08:32:39.394 INFO:teuthology.orchestra.run.smithi002.stdout: "objecter_timeout": "10.000000", 2023-11-05T08:32:39.395 INFO:teuthology.orchestra.run.smithi002.stdout: "objectstore_blackhole": "false", 2023-11-05T08:32:39.395 INFO:teuthology.orchestra.run.smithi002.stdout: "openssl_engine_opts": "", 2023-11-05T08:32:39.395 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_agent_delay_time": "5.000000", 2023-11-05T08:32:39.395 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_agent_hist_halflife": "1000", 2023-11-05T08:32:39.395 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_agent_max_low_ops": "2", 2023-11-05T08:32:39.396 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_agent_max_ops": "4", 2023-11-05T08:32:39.396 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_agent_min_evict_effort": "0.100000", 2023-11-05T08:32:39.396 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_agent_quantize_effort": "0.100000", 2023-11-05T08:32:39.396 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_agent_slop": "0.020000", 2023-11-05T08:32:39.396 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_aggregated_slow_ops_logging": "true", 2023-11-05T08:32:39.396 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_allow_recovery_below_min_size": "true", 2023-11-05T08:32:39.397 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_asio_thread_count": "2", 2023-11-05T08:32:39.397 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_async_recovery_min_cost": "1", 2023-11-05T08:32:39.397 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_auto_mark_unfound_lost": "false", 2023-11-05T08:32:39.397 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_backfill_retry_interval": "30.000000", 2023-11-05T08:32:39.397 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_backfill_scan_max": "512", 2023-11-05T08:32:39.398 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_backfill_scan_min": "64", 2023-11-05T08:32:39.398 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_backoff_on_degraded": "false", 2023-11-05T08:32:39.398 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_backoff_on_peering": "false", 2023-11-05T08:32:39.398 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_backoff_on_unfound": "true", 2023-11-05T08:32:39.398 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_beacon_report_interval": "300", 2023-11-05T08:32:39.399 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_bench_duration": "30", 2023-11-05T08:32:39.399 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_bench_large_size_max_throughput": "104857600", 2023-11-05T08:32:39.399 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_bench_max_block_size": "67108864", 2023-11-05T08:32:39.399 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_bench_small_size_max_iops": "100", 2023-11-05T08:32:39.399 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_blkin_trace_all": "false", 2023-11-05T08:32:39.399 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_blocked_scrub_grace_period": "120", 2023-11-05T08:32:39.400 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2023-11-05T08:32:39.400 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2023-11-05T08:32:39.400 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2023-11-05T08:32:39.400 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_check_for_log_corruption": "false", 2023-11-05T08:32:39.400 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_check_max_object_name_len_on_startup": "true", 2023-11-05T08:32:39.401 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2023-11-05T08:32:39.401 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_class_dir": "/usr/lib/rados-classes", 2023-11-05T08:32:39.401 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2023-11-05T08:32:39.401 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_class_update_on_start": "true", 2023-11-05T08:32:39.401 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_client_message_cap": "256", 2023-11-05T08:32:39.402 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_client_message_size_cap": "524288000", 2023-11-05T08:32:39.402 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_client_op_priority": "63", 2023-11-05T08:32:39.402 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_client_watch_timeout": "30", 2023-11-05T08:32:39.402 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_command_max_records": "256", 2023-11-05T08:32:39.402 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_compact_on_start": "false", 2023-11-05T08:32:39.402 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_copyfrom_max_chunk": "8388608", 2023-11-05T08:32:39.403 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_crush_chooseleaf_type": "0", 2023-11-05T08:32:39.403 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_crush_initial_weight": "-1.000000", 2023-11-05T08:32:39.403 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_crush_update_on_start": "true", 2023-11-05T08:32:39.403 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_crush_update_weight_set": "true", 2023-11-05T08:32:39.403 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2023-11-05T08:32:39.403 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2023-11-05T08:32:39.404 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2023-11-05T08:32:39.404 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_debug_drop_ping_duration": "0", 2023-11-05T08:32:39.404 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_debug_drop_ping_probability": "0.000000", 2023-11-05T08:32:39.404 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_debug_feed_pullee": "-1", 2023-11-05T08:32:39.404 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_debug_inject_copyfrom_error": "false", 2023-11-05T08:32:39.405 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2023-11-05T08:32:39.405 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2023-11-05T08:32:39.405 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_debug_misdirected_ops": "false", 2023-11-05T08:32:39.405 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_debug_no_acting_change": "false", 2023-11-05T08:32:39.405 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_debug_no_purge_strays": "false", 2023-11-05T08:32:39.405 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_debug_op_order": "false", 2023-11-05T08:32:39.406 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_debug_pg_log_writeout": "false", 2023-11-05T08:32:39.406 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_debug_pretend_recovery_active": "false", 2023-11-05T08:32:39.406 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_debug_random_push_read_error": "0.000000", 2023-11-05T08:32:39.406 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2023-11-05T08:32:39.406 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_debug_shutdown": "false", 2023-11-05T08:32:39.406 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2023-11-05T08:32:39.407 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2023-11-05T08:32:39.407 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_debug_trim_objects": "false", 2023-11-05T08:32:39.407 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_debug_verify_cached_snaps": "false", 2023-11-05T08:32:39.407 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_debug_verify_missing_on_start": "false", 2023-11-05T08:32:39.407 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_debug_verify_snaps": "false", 2023-11-05T08:32:39.408 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_debug_verify_stray_on_activate": "false", 2023-11-05T08:32:39.408 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_deep_scrub_interval": "604800.000000", 2023-11-05T08:32:39.408 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_deep_scrub_keys": "1024", 2023-11-05T08:32:39.408 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2023-11-05T08:32:39.408 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2023-11-05T08:32:39.409 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2023-11-05T08:32:39.409 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_deep_scrub_stride": "524288", 2023-11-05T08:32:39.409 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2023-11-05T08:32:39.409 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_default_data_pool_replay_window": "5", 2023-11-05T08:32:39.409 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_default_notify_timeout": "30", 2023-11-05T08:32:39.410 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_delete_sleep": "0.000000", 2023-11-05T08:32:39.410 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_delete_sleep_hdd": "5.000000", 2023-11-05T08:32:39.410 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_delete_sleep_hybrid": "1.000000", 2023-11-05T08:32:39.410 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_delete_sleep_ssd": "1.000000", 2023-11-05T08:32:39.410 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_discard_disconnected_ops": "true", 2023-11-05T08:32:39.411 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_enable_op_tracker": "true", 2023-11-05T08:32:39.411 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2023-11-05T08:32:39.411 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_extblkdev_plugins": "vdo", 2023-11-05T08:32:39.411 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_failsafe_full_ratio": "0.970000", 2023-11-05T08:32:39.412 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_fast_fail_on_connection_refused": "true", 2023-11-05T08:32:39.412 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_fast_info": "true", 2023-11-05T08:32:39.412 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_fast_shutdown": "true", 2023-11-05T08:32:39.412 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_fast_shutdown_notify_mon": "true", 2023-11-05T08:32:39.412 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_fast_shutdown_timeout": "15", 2023-11-05T08:32:39.413 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_find_best_info_ignore_history_les": "false", 2023-11-05T08:32:39.413 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_force_auth_primary_missing_objects": "100", 2023-11-05T08:32:39.413 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2023-11-05T08:32:39.413 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_function_tracing": "false", 2023-11-05T08:32:39.413 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_heartbeat_grace": "20", 2023-11-05T08:32:39.413 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_heartbeat_interval": "6", 2023-11-05T08:32:39.414 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2023-11-05T08:32:39.414 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_heartbeat_min_peers": "10", 2023-11-05T08:32:39.414 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_heartbeat_min_size": "2000", 2023-11-05T08:32:39.414 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_heartbeat_stale": "600", 2023-11-05T08:32:39.415 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2023-11-05T08:32:39.415 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_hit_set_max_size": "100000", 2023-11-05T08:32:39.415 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_hit_set_min_size": "1000", 2023-11-05T08:32:39.415 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_hit_set_namespace": ".ceph-internal", 2023-11-05T08:32:39.415 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_ignore_stale_divergent_priors": "false", 2023-11-05T08:32:39.415 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2023-11-05T08:32:39.416 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_inject_failure_on_pg_removal": "false", 2023-11-05T08:32:39.416 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2023-11-05T08:32:39.416 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_journal_flush_on_shutdown": "true", 2023-11-05T08:32:39.416 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_journal_size": "5120", 2023-11-05T08:32:39.416 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_kill_backfill_at": "0", 2023-11-05T08:32:39.417 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_loop_before_reset_tphandle": "64", 2023-11-05T08:32:39.417 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_map_cache_size": "50", 2023-11-05T08:32:39.417 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_map_dedup": "true", 2023-11-05T08:32:39.417 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_map_message_max": "40", 2023-11-05T08:32:39.417 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_map_message_max_bytes": "10485760", 2023-11-05T08:32:39.418 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_map_share_max_epochs": "40", 2023-11-05T08:32:39.418 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_max_attr_name_len": "100", 2023-11-05T08:32:39.418 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_max_attr_size": "0", 2023-11-05T08:32:39.418 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_max_backfills": "1", 2023-11-05T08:32:39.418 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_max_markdown_count": "5", 2023-11-05T08:32:39.419 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_max_markdown_period": "600", 2023-11-05T08:32:39.419 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_max_object_name_len": "2048", 2023-11-05T08:32:39.419 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_max_object_namespace_len": "256", 2023-11-05T08:32:39.419 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_max_object_size": "134217728", 2023-11-05T08:32:39.419 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2023-11-05T08:32:39.420 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_max_omap_entries_per_request": "1024", 2023-11-05T08:32:39.420 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_max_pg_blocked_by": "16", 2023-11-05T08:32:39.420 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_max_pg_log_entries": "10000", 2023-11-05T08:32:39.420 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2023-11-05T08:32:39.420 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_max_pgls": "1024", 2023-11-05T08:32:39.421 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_max_push_cost": "8388608", 2023-11-05T08:32:39.421 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_max_push_objects": "10", 2023-11-05T08:32:39.421 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_max_scrubs": "3", 2023-11-05T08:32:39.421 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2023-11-05T08:32:39.421 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_max_trimming_pgs": "2", 2023-11-05T08:32:39.422 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_max_write_op_reply_len": "64", 2023-11-05T08:32:39.422 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_max_write_size": "90", 2023-11-05T08:32:39.422 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2023-11-05T08:32:39.422 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2023-11-05T08:32:39.422 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2023-11-05T08:32:39.422 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2023-11-05T08:32:39.423 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2023-11-05T08:32:39.423 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2023-11-05T08:32:39.423 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2023-11-05T08:32:39.423 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mclock_override_recovery_settings": "false", 2023-11-05T08:32:39.423 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mclock_profile": "balanced", 2023-11-05T08:32:39.424 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2023-11-05T08:32:39.424 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2023-11-05T08:32:39.424 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2023-11-05T08:32:39.424 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2023-11-05T08:32:39.424 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2023-11-05T08:32:39.425 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2023-11-05T08:32:39.425 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2023-11-05T08:32:39.425 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2023-11-05T08:32:39.425 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2023-11-05T08:32:39.425 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mclock_scheduler_client_wgt": "1", 2023-11-05T08:32:39.426 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mclock_skip_benchmark": "false", 2023-11-05T08:32:39.426 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_memory_base": "805306368", 2023-11-05T08:32:39.426 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_memory_cache_min": "134217728", 2023-11-05T08:32:39.426 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_memory_cache_resize_interval": "1.000000", 2023-11-05T08:32:39.426 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_memory_expected_fragmentation": "0.150000", 2023-11-05T08:32:39.426 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_memory_target": "4294967296", 2023-11-05T08:32:39.427 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_memory_target_autotune": "false", 2023-11-05T08:32:39.427 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2023-11-05T08:32:39.427 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_min_pg_log_entries": "250", 2023-11-05T08:32:39.427 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_min_recovery_priority": "0", 2023-11-05T08:32:39.427 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mon_heartbeat_interval": "30", 2023-11-05T08:32:39.428 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2023-11-05T08:32:39.428 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mon_report_interval": "5", 2023-11-05T08:32:39.428 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mon_report_max_in_flight": "2", 2023-11-05T08:32:39.428 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mon_shutdown_timeout": "5.000000", 2023-11-05T08:32:39.428 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_num_cache_shards": "32", 2023-11-05T08:32:39.428 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_num_op_tracker_shard": "32", 2023-11-05T08:32:39.429 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_numa_auto_affinity": "true", 2023-11-05T08:32:39.429 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_numa_node": "-1", 2023-11-05T08:32:39.429 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_numa_prefer_iface": "true", 2023-11-05T08:32:39.429 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_object_clean_region_max_num_intervals": "10", 2023-11-05T08:32:39.429 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_objecter_finishers": "1", 2023-11-05T08:32:39.430 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_objectstore": "bluestore", 2023-11-05T08:32:39.430 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_objectstore_fuse": "false", 2023-11-05T08:32:39.430 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_objectstore_tracing": "false", 2023-11-05T08:32:39.430 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_op_complaint_time": "30.000000", 2023-11-05T08:32:39.430 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_op_history_duration": "600", 2023-11-05T08:32:39.431 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_op_history_size": "20", 2023-11-05T08:32:39.431 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_op_history_slow_op_size": "20", 2023-11-05T08:32:39.431 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2023-11-05T08:32:39.431 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_op_log_threshold": "5", 2023-11-05T08:32:39.431 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_op_num_shards": "0", 2023-11-05T08:32:39.432 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_op_num_shards_hdd": "5", 2023-11-05T08:32:39.432 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_op_num_shards_ssd": "8", 2023-11-05T08:32:39.432 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_op_num_threads_per_shard": "0", 2023-11-05T08:32:39.432 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2023-11-05T08:32:39.432 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2023-11-05T08:32:39.432 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2023-11-05T08:32:39.433 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_op_pq_min_cost": "65536", 2023-11-05T08:32:39.433 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_op_queue": "mclock_scheduler", 2023-11-05T08:32:39.433 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_op_queue_cut_off": "high", 2023-11-05T08:32:39.433 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_op_thread_suicide_timeout": "150", 2023-11-05T08:32:39.433 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_op_thread_timeout": "15", 2023-11-05T08:32:39.434 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_open_classes_on_start": "true", 2023-11-05T08:32:39.434 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_os_flags": "0", 2023-11-05T08:32:39.434 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_peering_op_priority": "255", 2023-11-05T08:32:39.434 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pg_delete_cost": "1048576", 2023-11-05T08:32:39.435 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pg_delete_priority": "5", 2023-11-05T08:32:39.435 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2023-11-05T08:32:39.435 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2023-11-05T08:32:39.435 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pg_log_dups_tracked": "3000", 2023-11-05T08:32:39.435 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pg_log_trim_max": "10000", 2023-11-05T08:32:39.435 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pg_log_trim_min": "100", 2023-11-05T08:32:39.436 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2023-11-05T08:32:39.436 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pg_object_context_cache_count": "64", 2023-11-05T08:32:39.436 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pg_stat_report_interval_max": "500", 2023-11-05T08:32:39.436 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2023-11-05T08:32:39.436 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pool_default_cache_min_evict_age": "0", 2023-11-05T08:32:39.436 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pool_default_cache_min_flush_age": "0", 2023-11-05T08:32:39.437 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2023-11-05T08:32:39.437 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2023-11-05T08:32:39.437 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2023-11-05T08:32:39.437 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pool_default_crimson": "false", 2023-11-05T08:32:39.437 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pool_default_crush_rule": "-1", 2023-11-05T08:32:39.438 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pool_default_ec_fast_read": "false", 2023-11-05T08:32:39.438 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2023-11-05T08:32:39.438 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pool_default_flag_bulk": "false", 2023-11-05T08:32:39.438 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pool_default_flag_hashpspool": "true", 2023-11-05T08:32:39.438 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pool_default_flag_nodelete": "false", 2023-11-05T08:32:39.438 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pool_default_flag_nopgchange": "false", 2023-11-05T08:32:39.439 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pool_default_flag_nosizechange": "false", 2023-11-05T08:32:39.439 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pool_default_flags": "0", 2023-11-05T08:32:39.439 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2023-11-05T08:32:39.439 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pool_default_min_size": "0", 2023-11-05T08:32:39.439 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2023-11-05T08:32:39.439 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pool_default_pg_num": "32", 2023-11-05T08:32:39.440 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pool_default_pgp_num": "0", 2023-11-05T08:32:39.440 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2023-11-05T08:32:39.440 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pool_default_size": "2", 2023-11-05T08:32:39.440 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pool_default_type": "replicated", 2023-11-05T08:32:39.441 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2023-11-05T08:32:39.441 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pool_use_gmt_hitset": "true", 2023-11-05T08:32:39.441 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_push_per_object_cost": "1000", 2023-11-05T08:32:39.441 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_read_ec_check_for_errors": "false", 2023-11-05T08:32:39.441 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_recover_clone_overlap": "true", 2023-11-05T08:32:39.441 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_recover_clone_overlap_limit": "10", 2023-11-05T08:32:39.442 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_recovery_cost": "20971520", 2023-11-05T08:32:39.442 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_recovery_delay_start": "0.000000", 2023-11-05T08:32:39.442 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_recovery_max_active": "0", 2023-11-05T08:32:39.442 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_recovery_max_active_hdd": "3", 2023-11-05T08:32:39.443 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_recovery_max_active_ssd": "10", 2023-11-05T08:32:39.443 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_recovery_max_chunk": "8388608", 2023-11-05T08:32:39.443 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2023-11-05T08:32:39.443 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_recovery_max_single_start": "1", 2023-11-05T08:32:39.443 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_recovery_op_priority": "3", 2023-11-05T08:32:39.444 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_recovery_op_warn_multiple": "16", 2023-11-05T08:32:39.444 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_recovery_priority": "5", 2023-11-05T08:32:39.444 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_recovery_retry_interval": "30.000000", 2023-11-05T08:32:39.444 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_recovery_sleep": "0.000000", 2023-11-05T08:32:39.444 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_recovery_sleep_hdd": "0.100000", 2023-11-05T08:32:39.445 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2023-11-05T08:32:39.445 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_recovery_sleep_ssd": "0.000000", 2023-11-05T08:32:39.445 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_repair_during_recovery": "false", 2023-11-05T08:32:39.445 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_requested_scrub_priority": "120", 2023-11-05T08:32:39.445 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2023-11-05T08:32:39.445 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_rollback_to_cluster_snap": "", 2023-11-05T08:32:39.446 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_scrub_auto_repair": "false", 2023-11-05T08:32:39.446 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_scrub_auto_repair_num_errors": "5", 2023-11-05T08:32:39.446 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_scrub_backoff_ratio": "0.660000", 2023-11-05T08:32:39.446 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_scrub_begin_hour": "0", 2023-11-05T08:32:39.446 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_scrub_begin_week_day": "0", 2023-11-05T08:32:39.447 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_scrub_chunk_max": "25", 2023-11-05T08:32:39.447 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_scrub_chunk_min": "5", 2023-11-05T08:32:39.447 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_scrub_cost": "52428800", 2023-11-05T08:32:39.447 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_scrub_during_recovery": "false", 2023-11-05T08:32:39.447 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_scrub_end_hour": "0", 2023-11-05T08:32:39.448 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_scrub_end_week_day": "0", 2023-11-05T08:32:39.448 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_scrub_event_cost": "4096", 2023-11-05T08:32:39.448 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_scrub_extended_sleep": "0.000000", 2023-11-05T08:32:39.448 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2023-11-05T08:32:39.448 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_scrub_invalid_stats": "true", 2023-11-05T08:32:39.448 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_scrub_load_threshold": "0.500000", 2023-11-05T08:32:39.449 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_scrub_max_interval": "604800.000000", 2023-11-05T08:32:39.449 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_scrub_max_preemptions": "5", 2023-11-05T08:32:39.449 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_scrub_min_interval": "86400.000000", 2023-11-05T08:32:39.449 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_scrub_priority": "5", 2023-11-05T08:32:39.449 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_scrub_reservation_timeout": "5000", 2023-11-05T08:32:39.450 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_scrub_sleep": "0.000000", 2023-11-05T08:32:39.450 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_scrub_slow_reservation_response": "2200", 2023-11-05T08:32:39.450 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_shallow_scrub_chunk_max": "100", 2023-11-05T08:32:39.450 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_shallow_scrub_chunk_min": "50", 2023-11-05T08:32:39.450 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_shutdown_pgref_assert": "false", 2023-11-05T08:32:39.450 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_skip_data_digest": "false", 2023-11-05T08:32:39.451 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_smart_report_timeout": "5", 2023-11-05T08:32:39.451 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_snap_trim_cost": "1048576", 2023-11-05T08:32:39.451 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_snap_trim_priority": "5", 2023-11-05T08:32:39.451 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_snap_trim_sleep": "0.000000", 2023-11-05T08:32:39.451 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2023-11-05T08:32:39.452 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2023-11-05T08:32:39.452 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2023-11-05T08:32:39.452 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_stats_update_period_not_scrubbing": "120", 2023-11-05T08:32:39.452 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_stats_update_period_scrubbing": "15", 2023-11-05T08:32:39.452 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2023-11-05T08:32:39.453 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_target_transaction_size": "30", 2023-11-05T08:32:39.453 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_tier_default_cache_hit_set_count": "4", 2023-11-05T08:32:39.453 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2023-11-05T08:32:39.453 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2023-11-05T08:32:39.453 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2023-11-05T08:32:39.454 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2023-11-05T08:32:39.454 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2023-11-05T08:32:39.454 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2023-11-05T08:32:39.454 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_tier_default_cache_mode": "writeback", 2023-11-05T08:32:39.454 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2023-11-05T08:32:39.455 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_tier_promote_max_objects_sec": "25", 2023-11-05T08:32:39.455 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_tracing": "false", 2023-11-05T08:32:39.455 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_use_stale_snap": "false", 2023-11-05T08:32:39.455 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2023-11-05T08:32:39.455 INFO:teuthology.orchestra.run.smithi002.stdout: "osdc_blkin_trace_all": "false", 2023-11-05T08:32:39.456 INFO:teuthology.orchestra.run.smithi002.stdout: "paxos_kill_at": "0", 2023-11-05T08:32:39.456 INFO:teuthology.orchestra.run.smithi002.stdout: "paxos_max_join_drift": "10", 2023-11-05T08:32:39.456 INFO:teuthology.orchestra.run.smithi002.stdout: "paxos_min": "500", 2023-11-05T08:32:39.456 INFO:teuthology.orchestra.run.smithi002.stdout: "paxos_min_wait": "0.050000", 2023-11-05T08:32:39.456 INFO:teuthology.orchestra.run.smithi002.stdout: "paxos_propose_interval": "1.000000", 2023-11-05T08:32:39.457 INFO:teuthology.orchestra.run.smithi002.stdout: "paxos_service_trim_max": "500", 2023-11-05T08:32:39.457 INFO:teuthology.orchestra.run.smithi002.stdout: "paxos_service_trim_max_multiplier": "20", 2023-11-05T08:32:39.457 INFO:teuthology.orchestra.run.smithi002.stdout: "paxos_service_trim_min": "250", 2023-11-05T08:32:39.457 INFO:teuthology.orchestra.run.smithi002.stdout: "paxos_stash_full_interval": "25", 2023-11-05T08:32:39.458 INFO:teuthology.orchestra.run.smithi002.stdout: "paxos_trim_max": "500", 2023-11-05T08:32:39.458 INFO:teuthology.orchestra.run.smithi002.stdout: "paxos_trim_min": "250", 2023-11-05T08:32:39.458 INFO:teuthology.orchestra.run.smithi002.stdout: "perf": "true", 2023-11-05T08:32:39.458 INFO:teuthology.orchestra.run.smithi002.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2023-11-05T08:32:39.458 INFO:teuthology.orchestra.run.smithi002.stdout: "plugin_crypto_accelerator": "crypto_isal", 2023-11-05T08:32:39.459 INFO:teuthology.orchestra.run.smithi002.stdout: "plugin_dir": "/usr/lib/ceph", 2023-11-05T08:32:39.459 INFO:teuthology.orchestra.run.smithi002.stdout: "public_addr": "-", 2023-11-05T08:32:39.459 INFO:teuthology.orchestra.run.smithi002.stdout: "public_addrv": "", 2023-11-05T08:32:39.459 INFO:teuthology.orchestra.run.smithi002.stdout: "public_bind_addr": "-", 2023-11-05T08:32:39.459 INFO:teuthology.orchestra.run.smithi002.stdout: "public_network": "", 2023-11-05T08:32:39.460 INFO:teuthology.orchestra.run.smithi002.stdout: "public_network_interface": "", 2023-11-05T08:32:39.460 INFO:teuthology.orchestra.run.smithi002.stdout: "qat_compressor_enabled": "false", 2023-11-05T08:32:39.460 INFO:teuthology.orchestra.run.smithi002.stdout: "qat_compressor_session_max_number": "256", 2023-11-05T08:32:39.460 INFO:teuthology.orchestra.run.smithi002.stdout: "rados_mon_op_timeout": "0", 2023-11-05T08:32:39.461 INFO:teuthology.orchestra.run.smithi002.stdout: "rados_osd_op_timeout": "0", 2023-11-05T08:32:39.461 INFO:teuthology.orchestra.run.smithi002.stdout: "rados_tracing": "false", 2023-11-05T08:32:39.461 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_atime_update_interval": "60", 2023-11-05T08:32:39.461 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2023-11-05T08:32:39.461 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_balance_parent_reads": "false", 2023-11-05T08:32:39.462 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_balance_snap_reads": "false", 2023-11-05T08:32:39.462 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_blkin_trace_all": "false", 2023-11-05T08:32:39.462 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_blocklist_expire_seconds": "0", 2023-11-05T08:32:39.462 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_blocklist_on_break_lock": "true", 2023-11-05T08:32:39.462 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_cache": "true", 2023-11-05T08:32:39.463 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_cache_block_writes_upfront": "false", 2023-11-05T08:32:39.463 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_cache_max_dirty": "25165824", 2023-11-05T08:32:39.463 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_cache_max_dirty_age": "1.000000", 2023-11-05T08:32:39.463 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_cache_max_dirty_object": "0", 2023-11-05T08:32:39.463 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_cache_policy": "writearound", 2023-11-05T08:32:39.463 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_cache_size": "33554432", 2023-11-05T08:32:39.464 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_cache_target_dirty": "16777216", 2023-11-05T08:32:39.464 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_cache_writethrough_until_flush": "true", 2023-11-05T08:32:39.464 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_clone_copy_on_read": "false", 2023-11-05T08:32:39.464 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_compression_hint": "none", 2023-11-05T08:32:39.464 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_concurrent_management_ops": "10", 2023-11-05T08:32:39.465 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_config_pool_override_update_timestamp": "0", 2023-11-05T08:32:39.465 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_default_clone_format": "auto", 2023-11-05T08:32:39.465 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_default_data_pool": "", 2023-11-05T08:32:39.465 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_default_features": "61", 2023-11-05T08:32:39.465 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_default_format": "2", 2023-11-05T08:32:39.465 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_default_map_options": "", 2023-11-05T08:32:39.466 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_default_order": "22", 2023-11-05T08:32:39.466 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_default_pool": "rbd", 2023-11-05T08:32:39.466 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2023-11-05T08:32:39.466 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_default_stripe_count": "0", 2023-11-05T08:32:39.466 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_default_stripe_unit": "0", 2023-11-05T08:32:39.467 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_disable_zero_copy_writes": "true", 2023-11-05T08:32:39.467 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_discard_granularity_bytes": "65536", 2023-11-05T08:32:39.467 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_discard_on_zeroed_write_same": "true", 2023-11-05T08:32:39.467 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_enable_alloc_hint": "true", 2023-11-05T08:32:39.467 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2023-11-05T08:32:39.468 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_io_scheduler": "simple", 2023-11-05T08:32:39.468 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2023-11-05T08:32:39.468 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_journal_commit_age": "5.000000", 2023-11-05T08:32:39.468 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2023-11-05T08:32:39.468 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_journal_max_payload_bytes": "16384", 2023-11-05T08:32:39.468 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_journal_object_flush_age": "0.000000", 2023-11-05T08:32:39.469 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_journal_object_flush_bytes": "1048576", 2023-11-05T08:32:39.469 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_journal_object_flush_interval": "0", 2023-11-05T08:32:39.469 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2023-11-05T08:32:39.469 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2023-11-05T08:32:39.469 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_journal_order": "24", 2023-11-05T08:32:39.469 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_journal_pool": "", 2023-11-05T08:32:39.470 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_journal_splay_width": "4", 2023-11-05T08:32:39.470 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_localize_parent_reads": "false", 2023-11-05T08:32:39.470 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_localize_snap_reads": "false", 2023-11-05T08:32:39.470 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2023-11-05T08:32:39.470 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2023-11-05T08:32:39.470 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2023-11-05T08:32:39.471 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2023-11-05T08:32:39.471 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2023-11-05T08:32:39.471 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2023-11-05T08:32:39.471 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_mirror_image_policy_type": "simple", 2023-11-05T08:32:39.471 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2023-11-05T08:32:39.472 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_mirror_image_state_check_interval": "30", 2023-11-05T08:32:39.472 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2023-11-05T08:32:39.472 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2023-11-05T08:32:39.472 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2023-11-05T08:32:39.472 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2023-11-05T08:32:39.473 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2023-11-05T08:32:39.473 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_mirror_memory_autotune": "true", 2023-11-05T08:32:39.473 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_mirror_memory_base": "805306368", 2023-11-05T08:32:39.473 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2023-11-05T08:32:39.473 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_mirror_memory_cache_min": "134217728", 2023-11-05T08:32:39.474 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2023-11-05T08:32:39.474 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2023-11-05T08:32:39.474 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_mirror_memory_target": "4294967296", 2023-11-05T08:32:39.474 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_mirror_perf_stats_prio": "5", 2023-11-05T08:32:39.474 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2023-11-05T08:32:39.475 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2023-11-05T08:32:39.475 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_mirroring_delete_delay": "0", 2023-11-05T08:32:39.475 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2023-11-05T08:32:39.475 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_mirroring_replay_delay": "0", 2023-11-05T08:32:39.475 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2023-11-05T08:32:39.476 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2023-11-05T08:32:39.476 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_move_to_trash_on_remove": "false", 2023-11-05T08:32:39.476 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2023-11-05T08:32:39.476 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_mtime_update_interval": "60", 2023-11-05T08:32:39.476 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_non_blocking_aio": "true", 2023-11-05T08:32:39.477 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_op_thread_timeout": "60", 2023-11-05T08:32:39.477 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_op_threads": "1", 2023-11-05T08:32:39.477 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_parent_cache_enabled": "false", 2023-11-05T08:32:39.477 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_persistent_cache_mode": "disabled", 2023-11-05T08:32:39.477 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_persistent_cache_path": "/tmp", 2023-11-05T08:32:39.478 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_persistent_cache_size": "1073741824", 2023-11-05T08:32:39.478 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_plugins": "", 2023-11-05T08:32:39.478 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_qos_bps_burst": "0", 2023-11-05T08:32:39.478 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_qos_bps_burst_seconds": "1", 2023-11-05T08:32:39.478 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_qos_bps_limit": "0", 2023-11-05T08:32:39.479 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_qos_exclude_ops": "0", 2023-11-05T08:32:39.479 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_qos_iops_burst": "0", 2023-11-05T08:32:39.479 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_qos_iops_burst_seconds": "1", 2023-11-05T08:32:39.479 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_qos_iops_limit": "0", 2023-11-05T08:32:39.479 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_qos_read_bps_burst": "0", 2023-11-05T08:32:39.480 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2023-11-05T08:32:39.480 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_qos_read_bps_limit": "0", 2023-11-05T08:32:39.480 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_qos_read_iops_burst": "0", 2023-11-05T08:32:39.480 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2023-11-05T08:32:39.480 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_qos_read_iops_limit": "0", 2023-11-05T08:32:39.481 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_qos_schedule_tick_min": "50", 2023-11-05T08:32:39.481 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_qos_write_bps_burst": "0", 2023-11-05T08:32:39.481 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2023-11-05T08:32:39.481 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_qos_write_bps_limit": "0", 2023-11-05T08:32:39.481 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_qos_write_iops_burst": "0", 2023-11-05T08:32:39.482 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2023-11-05T08:32:39.482 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_qos_write_iops_limit": "0", 2023-11-05T08:32:39.482 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_quiesce_notification_attempts": "10", 2023-11-05T08:32:39.482 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_read_from_replica_policy": "default", 2023-11-05T08:32:39.482 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2023-11-05T08:32:39.483 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_readahead_max_bytes": "524288", 2023-11-05T08:32:39.483 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_readahead_trigger_requests": "10", 2023-11-05T08:32:39.483 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_request_timed_out_seconds": "30", 2023-11-05T08:32:39.483 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_skip_partial_discard": "true", 2023-11-05T08:32:39.483 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2023-11-05T08:32:39.484 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_tracing": "false", 2023-11-05T08:32:39.484 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_validate_names": "true", 2023-11-05T08:32:39.484 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_validate_pool": "true", 2023-11-05T08:32:39.484 INFO:teuthology.orchestra.run.smithi002.stdout: "restapi_base_url": "", 2023-11-05T08:32:39.484 INFO:teuthology.orchestra.run.smithi002.stdout: "restapi_log_level": "", 2023-11-05T08:32:39.485 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_acl_grants_max_num": "100", 2023-11-05T08:32:39.485 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_admin_entry": "admin", 2023-11-05T08:32:39.485 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2023-11-05T08:32:39.485 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_backend_store": "rados", 2023-11-05T08:32:39.485 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_barbican_url": "", 2023-11-05T08:32:39.485 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_beast_enable_async": "true", 2023-11-05T08:32:39.486 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2023-11-05T08:32:39.486 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_bucket_default_quota_max_size": "-1", 2023-11-05T08:32:39.486 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_bucket_index_max_aio": "128", 2023-11-05T08:32:39.486 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2023-11-05T08:32:39.487 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_bucket_quota_cache_size": "10000", 2023-11-05T08:32:39.487 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_bucket_quota_ttl": "600", 2023-11-05T08:32:39.487 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_bucket_sync_spawn_window": "20", 2023-11-05T08:32:39.487 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_cache_enabled": "true", 2023-11-05T08:32:39.487 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_cache_expiry_interval": "900", 2023-11-05T08:32:39.488 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_cache_lru_size": "10000", 2023-11-05T08:32:39.488 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_config_store": "rados", 2023-11-05T08:32:39.488 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_content_length_compat": "false", 2023-11-05T08:32:39.488 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_copy_obj_progress": "true", 2023-11-05T08:32:39.488 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2023-11-05T08:32:39.488 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_cors_rules_max_num": "100", 2023-11-05T08:32:39.489 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_cross_domain_policy": "", 2023-11-05T08:32:39.489 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_crypt_default_encryption_key": "", 2023-11-05T08:32:39.489 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_crypt_kmip_addr": "", 2023-11-05T08:32:39.489 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_crypt_kmip_ca_path": "", 2023-11-05T08:32:39.489 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_crypt_kmip_client_cert": "", 2023-11-05T08:32:39.490 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_crypt_kmip_client_key": "", 2023-11-05T08:32:39.490 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_crypt_kmip_kms_key_template": "", 2023-11-05T08:32:39.490 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_crypt_kmip_password": "", 2023-11-05T08:32:39.490 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2023-11-05T08:32:39.490 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_crypt_kmip_username": "", 2023-11-05T08:32:39.491 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_crypt_require_ssl": "true", 2023-11-05T08:32:39.491 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2023-11-05T08:32:39.491 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2023-11-05T08:32:39.491 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_crypt_sse_s3_backend": "vault", 2023-11-05T08:32:39.491 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2023-11-05T08:32:39.491 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2023-11-05T08:32:39.492 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2023-11-05T08:32:39.492 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2023-11-05T08:32:39.492 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2023-11-05T08:32:39.492 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2023-11-05T08:32:39.492 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2023-11-05T08:32:39.492 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2023-11-05T08:32:39.493 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2023-11-05T08:32:39.493 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2023-11-05T08:32:39.493 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2023-11-05T08:32:39.493 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_crypt_suppress_logs": "true", 2023-11-05T08:32:39.493 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_crypt_vault_addr": "", 2023-11-05T08:32:39.494 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_crypt_vault_auth": "token", 2023-11-05T08:32:39.494 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_crypt_vault_namespace": "", 2023-11-05T08:32:39.494 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_crypt_vault_prefix": "", 2023-11-05T08:32:39.494 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_crypt_vault_secret_engine": "transit", 2023-11-05T08:32:39.494 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_crypt_vault_ssl_cacert": "", 2023-11-05T08:32:39.495 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2023-11-05T08:32:39.495 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2023-11-05T08:32:39.495 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_crypt_vault_token_file": "", 2023-11-05T08:32:39.495 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_crypt_vault_verify_ssl": "true", 2023-11-05T08:32:39.495 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_curl_buffersize": "524288", 2023-11-05T08:32:39.495 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_curl_low_speed_limit": "1024", 2023-11-05T08:32:39.496 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_curl_low_speed_time": "300", 2023-11-05T08:32:39.496 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_curl_tcp_keepalive": "0", 2023-11-05T08:32:39.496 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_curl_wait_timeout_ms": "1000", 2023-11-05T08:32:39.496 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2023-11-05T08:32:39.496 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2023-11-05T08:32:39.496 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2023-11-05T08:32:39.497 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2023-11-05T08:32:39.497 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_d3n_l1_fadvise": "4", 2023-11-05T08:32:39.497 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2023-11-05T08:32:39.497 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_d3n_libaio_aio_num": "64", 2023-11-05T08:32:39.497 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_d3n_libaio_aio_threads": "20", 2023-11-05T08:32:39.498 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_d4n_host": "127.0.0.1", 2023-11-05T08:32:39.498 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_d4n_port": "6379", 2023-11-05T08:32:39.498 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2023-11-05T08:32:39.498 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_data_log_changes_size": "1000", 2023-11-05T08:32:39.498 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_data_log_num_shards": "128", 2023-11-05T08:32:39.499 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_data_log_obj_prefix": "data_log", 2023-11-05T08:32:39.499 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_data_log_window": "30", 2023-11-05T08:32:39.499 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_data_notify_interval_msec": "0", 2023-11-05T08:32:39.499 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_data_sync_poll_interval": "20", 2023-11-05T08:32:39.499 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_data_sync_spawn_window": "20", 2023-11-05T08:32:39.499 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2023-11-05T08:32:39.500 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_debug_inject_set_olh_err": "0", 2023-11-05T08:32:39.500 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_default_data_log_backing": "fifo", 2023-11-05T08:32:39.500 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_default_realm_info_oid": "default.realm", 2023-11-05T08:32:39.500 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_default_region_info_oid": "default.region", 2023-11-05T08:32:39.500 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_default_zone_info_oid": "default.zone", 2023-11-05T08:32:39.501 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2023-11-05T08:32:39.501 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_defer_to_bucket_acls": "", 2023-11-05T08:32:39.501 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_delete_multi_obj_max_num": "1000", 2023-11-05T08:32:39.501 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_dmclock_admin_lim": "0.000000", 2023-11-05T08:32:39.501 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_dmclock_admin_res": "100.000000", 2023-11-05T08:32:39.502 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2023-11-05T08:32:39.502 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_dmclock_auth_lim": "0.000000", 2023-11-05T08:32:39.502 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_dmclock_auth_res": "200.000000", 2023-11-05T08:32:39.502 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2023-11-05T08:32:39.502 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_dmclock_data_lim": "0.000000", 2023-11-05T08:32:39.503 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_dmclock_data_res": "500.000000", 2023-11-05T08:32:39.503 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_dmclock_data_wgt": "500.000000", 2023-11-05T08:32:39.503 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2023-11-05T08:32:39.503 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_dmclock_metadata_res": "500.000000", 2023-11-05T08:32:39.504 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2023-11-05T08:32:39.504 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_dns_name": "", 2023-11-05T08:32:39.504 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_dns_s3website_name": "", 2023-11-05T08:32:39.504 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_dynamic_resharding": "true", 2023-11-05T08:32:39.504 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2023-11-05T08:32:39.505 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_enable_gc_threads": "true", 2023-11-05T08:32:39.505 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_enable_lc_threads": "true", 2023-11-05T08:32:39.505 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_enable_ops_log": "false", 2023-11-05T08:32:39.505 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_enable_quota_threads": "true", 2023-11-05T08:32:39.505 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_enable_static_website": "false", 2023-11-05T08:32:39.506 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_enable_usage_log": "false", 2023-11-05T08:32:39.506 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_enforce_swift_acls": "true", 2023-11-05T08:32:39.506 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_exit_timeout_secs": "120", 2023-11-05T08:32:39.506 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_expose_bucket": "false", 2023-11-05T08:32:39.507 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_extended_http_attrs": "", 2023-11-05T08:32:39.507 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_filter": "none", 2023-11-05T08:32:39.507 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2023-11-05T08:32:39.507 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_frontends": "beast port=7480", 2023-11-05T08:32:39.507 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_gc_max_concurrent_io": "10", 2023-11-05T08:32:39.507 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_gc_max_deferred": "50", 2023-11-05T08:32:39.508 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2023-11-05T08:32:39.508 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_gc_max_objs": "32", 2023-11-05T08:32:39.508 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_gc_max_queue_size": "134213632", 2023-11-05T08:32:39.508 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_gc_max_trim_chunk": "16", 2023-11-05T08:32:39.508 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_gc_obj_min_wait": "7200", 2023-11-05T08:32:39.509 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_gc_processor_max_time": "3600", 2023-11-05T08:32:39.509 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_gc_processor_period": "3600", 2023-11-05T08:32:39.509 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_get_obj_max_req_size": "4194304", 2023-11-05T08:32:39.509 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_get_obj_window_size": "16777216", 2023-11-05T08:32:39.509 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_healthcheck_disabling_path": "", 2023-11-05T08:32:39.509 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_ignore_get_invalid_range": "false", 2023-11-05T08:32:39.510 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_init_timeout": "300", 2023-11-05T08:32:39.510 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2023-11-05T08:32:39.510 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2023-11-05T08:32:39.510 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_keystone_accepted_admin_roles": "", 2023-11-05T08:32:39.510 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_keystone_accepted_reader_roles": "", 2023-11-05T08:32:39.511 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2023-11-05T08:32:39.511 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_keystone_admin_domain": "", 2023-11-05T08:32:39.511 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_keystone_admin_password": "", 2023-11-05T08:32:39.511 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_keystone_admin_password_path": "", 2023-11-05T08:32:39.511 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_keystone_admin_project": "", 2023-11-05T08:32:39.512 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_keystone_admin_tenant": "", 2023-11-05T08:32:39.512 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_keystone_admin_token": "", 2023-11-05T08:32:39.512 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_keystone_admin_token_path": "", 2023-11-05T08:32:39.512 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_keystone_admin_user": "", 2023-11-05T08:32:39.512 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_keystone_api_version": "2", 2023-11-05T08:32:39.513 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_keystone_barbican_domain": "", 2023-11-05T08:32:39.513 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_keystone_barbican_password": "", 2023-11-05T08:32:39.513 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_keystone_barbican_project": "", 2023-11-05T08:32:39.513 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_keystone_barbican_tenant": "", 2023-11-05T08:32:39.513 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_keystone_barbican_user": "", 2023-11-05T08:32:39.513 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2023-11-05T08:32:39.514 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_keystone_implicit_tenants": "false", 2023-11-05T08:32:39.514 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2023-11-05T08:32:39.514 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_keystone_service_token_enabled": "false", 2023-11-05T08:32:39.514 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_keystone_token_cache_size": "10000", 2023-11-05T08:32:39.514 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_keystone_url": "", 2023-11-05T08:32:39.515 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_keystone_verify_ssl": "true", 2023-11-05T08:32:39.515 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_lc_debug_interval": "-1", 2023-11-05T08:32:39.515 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_lc_lock_max_time": "90", 2023-11-05T08:32:39.515 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_lc_max_objs": "32", 2023-11-05T08:32:39.515 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_lc_max_rules": "1000", 2023-11-05T08:32:39.515 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_lc_max_worker": "3", 2023-11-05T08:32:39.516 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_lc_max_wp_worker": "3", 2023-11-05T08:32:39.516 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_lc_thread_delay": "0", 2023-11-05T08:32:39.516 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2023-11-05T08:32:39.516 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_ldap_dnattr": "uid", 2023-11-05T08:32:39.516 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2023-11-05T08:32:39.517 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_ldap_searchfilter": "", 2023-11-05T08:32:39.517 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2023-11-05T08:32:39.517 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_ldap_uri": "ldaps://", 2023-11-05T08:32:39.517 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2023-11-05T08:32:39.517 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_list_bucket_min_readahead": "1000", 2023-11-05T08:32:39.517 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_list_buckets_max_chunk": "1000", 2023-11-05T08:32:39.518 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_log_http_headers": "", 2023-11-05T08:32:39.518 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_log_nonexistent_bucket": "false", 2023-11-05T08:32:39.518 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2023-11-05T08:32:39.518 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_log_object_name_utc": "false", 2023-11-05T08:32:39.518 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_lua_max_memory_per_state": "128000", 2023-11-05T08:32:39.519 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2023-11-05T08:32:39.519 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_max_attr_name_len": "0", 2023-11-05T08:32:39.519 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_max_attr_size": "0", 2023-11-05T08:32:39.519 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_max_attrs_num_in_req": "0", 2023-11-05T08:32:39.519 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_max_chunk_size": "4194304", 2023-11-05T08:32:39.519 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_max_concurrent_requests": "1024", 2023-11-05T08:32:39.520 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2023-11-05T08:32:39.520 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_max_dynamic_shards": "1999", 2023-11-05T08:32:39.520 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_max_listing_results": "1000", 2023-11-05T08:32:39.520 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_max_notify_retries": "10", 2023-11-05T08:32:39.520 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_max_objs_per_shard": "100000", 2023-11-05T08:32:39.520 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_max_put_param_size": "1048576", 2023-11-05T08:32:39.521 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_max_put_size": "5368709120", 2023-11-05T08:32:39.521 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_max_slo_entries": "1000", 2023-11-05T08:32:39.521 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_md_log_max_shards": "64", 2023-11-05T08:32:39.521 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_md_notify_interval_msec": "200", 2023-11-05T08:32:39.521 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_meta_sync_poll_interval": "20", 2023-11-05T08:32:39.522 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_meta_sync_spawn_window": "20", 2023-11-05T08:32:39.522 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_mime_types_file": "/etc/mime.types", 2023-11-05T08:32:39.522 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_mp_lock_max_time": "600", 2023-11-05T08:32:39.522 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_multi_obj_del_max_aio": "16", 2023-11-05T08:32:39.522 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_multipart_min_part_size": "5242880", 2023-11-05T08:32:39.523 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_multipart_part_upload_limit": "10000", 2023-11-05T08:32:39.523 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_nfs_fhcache_partitions": "3", 2023-11-05T08:32:39.523 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_nfs_fhcache_size": "2017", 2023-11-05T08:32:39.523 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_nfs_frontends": "rgw-nfs", 2023-11-05T08:32:39.523 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2023-11-05T08:32:39.524 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_nfs_lru_lanes": "5", 2023-11-05T08:32:39.524 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_nfs_max_gc": "300", 2023-11-05T08:32:39.524 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_nfs_namespace_expire_secs": "300", 2023-11-05T08:32:39.524 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_nfs_run_gc_threads": "false", 2023-11-05T08:32:39.524 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_nfs_run_lc_threads": "false", 2023-11-05T08:32:39.525 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_nfs_run_quota_threads": "false", 2023-11-05T08:32:39.525 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_nfs_run_sync_thread": "false", 2023-11-05T08:32:39.525 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_nfs_s3_fast_attrs": "false", 2023-11-05T08:32:39.525 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_nfs_write_completion_interval_s": "10", 2023-11-05T08:32:39.525 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_num_async_rados_threads": "32", 2023-11-05T08:32:39.526 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_num_control_oids": "8", 2023-11-05T08:32:39.526 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_numa_node": "-1", 2023-11-05T08:32:39.526 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_obj_stripe_size": "4194304", 2023-11-05T08:32:39.526 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_obj_tombstone_cache_size": "1000", 2023-11-05T08:32:39.526 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_objexp_chunk_size": "100", 2023-11-05T08:32:39.527 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_objexp_gc_interval": "600", 2023-11-05T08:32:39.527 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_objexp_hints_num_shards": "127", 2023-11-05T08:32:39.527 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_olh_pending_timeout_sec": "3600", 2023-11-05T08:32:39.527 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_op_thread_suicide_timeout": "0", 2023-11-05T08:32:39.527 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_op_thread_timeout": "600", 2023-11-05T08:32:39.528 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_op_tracing": "false", 2023-11-05T08:32:39.528 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_opa_token": "", 2023-11-05T08:32:39.528 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_opa_url": "", 2023-11-05T08:32:39.528 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_opa_verify_ssl": "true", 2023-11-05T08:32:39.528 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_ops_log_data_backlog": "5242880", 2023-11-05T08:32:39.528 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2023-11-05T08:32:39.529 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_ops_log_rados": "false", 2023-11-05T08:32:39.529 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_ops_log_socket_path": "", 2023-11-05T08:32:39.529 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_override_bucket_index_max_shards": "0", 2023-11-05T08:32:39.529 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2023-11-05T08:32:39.529 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2023-11-05T08:32:39.530 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_period_push_interval": "2.000000", 2023-11-05T08:32:39.530 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_period_push_interval_max": "30.000000", 2023-11-05T08:32:39.530 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_period_root_pool": ".rgw.root", 2023-11-05T08:32:39.530 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_policy_reject_invalid_principals": "true", 2023-11-05T08:32:39.530 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2023-11-05T08:32:39.531 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_posix_cache_lanes": "3", 2023-11-05T08:32:39.531 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_posix_cache_lmdb_count": "3", 2023-11-05T08:32:39.531 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_posix_cache_max_buckets": "100", 2023-11-05T08:32:39.531 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_posix_cache_partitions": "3", 2023-11-05T08:32:39.531 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2023-11-05T08:32:39.531 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_print_continue": "true", 2023-11-05T08:32:39.532 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_print_prohibited_content_length": "false", 2023-11-05T08:32:39.532 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_put_obj_max_window_size": "67108864", 2023-11-05T08:32:39.532 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_put_obj_min_window_size": "16777216", 2023-11-05T08:32:39.532 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2023-11-05T08:32:39.532 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_rados_pool_recovery_priority": "5", 2023-11-05T08:32:39.532 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_rados_tracing": "false", 2023-11-05T08:32:39.533 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_realm": "", 2023-11-05T08:32:39.533 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_realm_id": "", 2023-11-05T08:32:39.533 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_realm_root_pool": ".rgw.root", 2023-11-05T08:32:39.533 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_region": "", 2023-11-05T08:32:39.533 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_region_root_pool": ".rgw.root", 2023-11-05T08:32:39.534 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_relaxed_region_enforcement": "false", 2023-11-05T08:32:39.534 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_relaxed_s3_bucket_names": "false", 2023-11-05T08:32:39.534 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2023-11-05T08:32:39.534 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_request_uri": "", 2023-11-05T08:32:39.534 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_reshard_batch_size": "64", 2023-11-05T08:32:39.535 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_reshard_bucket_lock_duration": "360", 2023-11-05T08:32:39.535 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_reshard_max_aio": "128", 2023-11-05T08:32:39.535 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_reshard_num_logs": "16", 2023-11-05T08:32:39.535 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_reshard_thread_interval": "600", 2023-11-05T08:32:39.535 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_resolve_cname": "false", 2023-11-05T08:32:39.536 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_rest_getusage_op_compat": "false", 2023-11-05T08:32:39.536 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_run_sync_thread": "true", 2023-11-05T08:32:39.536 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_s3_auth_order": "sts, external, local", 2023-11-05T08:32:39.536 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_s3_auth_use_keystone": "false", 2023-11-05T08:32:39.536 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_s3_auth_use_ldap": "false", 2023-11-05T08:32:39.537 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_s3_auth_use_rados": "true", 2023-11-05T08:32:39.537 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_s3_auth_use_sts": "false", 2023-11-05T08:32:39.537 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_s3_client_max_sig_ver": "-1", 2023-11-05T08:32:39.537 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_s3_success_create_obj_status": "0", 2023-11-05T08:32:39.537 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_safe_max_objects_per_shard": "102400", 2023-11-05T08:32:39.538 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_scheduler_type": "throttler", 2023-11-05T08:32:39.538 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_script_uri": "", 2023-11-05T08:32:39.538 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_service_provider_name": "", 2023-11-05T08:32:39.538 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_shard_warning_threshold": "90.000000", 2023-11-05T08:32:39.538 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_sts_client_id": "", 2023-11-05T08:32:39.539 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_sts_client_secret": "", 2023-11-05T08:32:39.539 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_sts_entry": "sts", 2023-11-05T08:32:39.539 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_sts_key": "sts", 2023-11-05T08:32:39.539 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_sts_max_session_duration": "43200", 2023-11-05T08:32:39.539 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_sts_min_session_duration": "900", 2023-11-05T08:32:39.539 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_sts_token_introspection_url": "", 2023-11-05T08:32:39.540 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_swift_account_in_url": "false", 2023-11-05T08:32:39.540 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_swift_auth_entry": "auth", 2023-11-05T08:32:39.540 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_swift_auth_url": "", 2023-11-05T08:32:39.540 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_swift_custom_header": "", 2023-11-05T08:32:39.540 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_swift_enforce_content_length": "false", 2023-11-05T08:32:39.541 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_swift_need_stats": "true", 2023-11-05T08:32:39.541 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_swift_tenant_name": "", 2023-11-05T08:32:39.541 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_swift_token_expiration": "86400", 2023-11-05T08:32:39.541 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_swift_url": "", 2023-11-05T08:32:39.541 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_swift_url_prefix": "swift", 2023-11-05T08:32:39.542 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_swift_versioning_enabled": "false", 2023-11-05T08:32:39.542 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2023-11-05T08:32:39.542 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2023-11-05T08:32:39.542 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_sync_lease_period": "120", 2023-11-05T08:32:39.542 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2023-11-05T08:32:39.542 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_sync_log_trim_interval": "1200", 2023-11-05T08:32:39.543 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_sync_log_trim_max_buckets": "16", 2023-11-05T08:32:39.543 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2023-11-05T08:32:39.543 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2023-11-05T08:32:39.543 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_sync_obj_etag_verify": "false", 2023-11-05T08:32:39.543 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_sync_trace_history_size": "4096", 2023-11-05T08:32:39.544 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_sync_trace_per_node_log_size": "32", 2023-11-05T08:32:39.544 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2023-11-05T08:32:39.544 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_thread_pool_size": "512", 2023-11-05T08:32:39.544 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_topic_persistency_max_retries": "0", 2023-11-05T08:32:39.544 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_topic_persistency_sleep_duration": "0", 2023-11-05T08:32:39.545 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_topic_persistency_time_to_live": "0", 2023-11-05T08:32:39.545 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_torrent_comment": "", 2023-11-05T08:32:39.545 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_torrent_createby": "", 2023-11-05T08:32:39.545 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_torrent_encoding": "", 2023-11-05T08:32:39.545 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_torrent_flag": "false", 2023-11-05T08:32:39.546 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_torrent_max_size": "5368709120", 2023-11-05T08:32:39.546 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_torrent_origin": "", 2023-11-05T08:32:39.546 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_torrent_sha_unit": "524288", 2023-11-05T08:32:39.546 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_torrent_tracker": "", 2023-11-05T08:32:39.546 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_trust_forwarded_https": "false", 2023-11-05T08:32:39.547 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_usage_log_flush_threshold": "1024", 2023-11-05T08:32:39.547 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_usage_log_tick_interval": "30", 2023-11-05T08:32:39.547 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_usage_max_shards": "32", 2023-11-05T08:32:39.547 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_usage_max_user_shards": "1", 2023-11-05T08:32:39.547 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_use_opa_authz": "false", 2023-11-05T08:32:39.548 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_user_default_quota_max_objects": "-1", 2023-11-05T08:32:39.548 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_user_default_quota_max_size": "-1", 2023-11-05T08:32:39.548 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_user_max_buckets": "1000", 2023-11-05T08:32:39.548 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_user_policies_max_num": "100", 2023-11-05T08:32:39.548 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2023-11-05T08:32:39.549 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_user_quota_sync_idle_users": "false", 2023-11-05T08:32:39.549 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_user_quota_sync_interval": "86400", 2023-11-05T08:32:39.549 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_user_quota_sync_wait_time": "86400", 2023-11-05T08:32:39.549 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_user_unique_email": "true", 2023-11-05T08:32:39.549 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_verify_ssl": "true", 2023-11-05T08:32:39.550 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_website_routing_rules_max_num": "50", 2023-11-05T08:32:39.550 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_zone": "", 2023-11-05T08:32:39.550 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_zone_id": "", 2023-11-05T08:32:39.550 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_zone_root_pool": ".rgw.root", 2023-11-05T08:32:39.550 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_zonegroup": "", 2023-11-05T08:32:39.551 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_zonegroup_id": "", 2023-11-05T08:32:39.551 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2023-11-05T08:32:39.551 INFO:teuthology.orchestra.run.smithi002.stdout: "rgwlc_auto_session_clear": "true", 2023-11-05T08:32:39.551 INFO:teuthology.orchestra.run.smithi002.stdout: "rgwlc_skip_bucket_step": "false", 2023-11-05T08:32:39.551 INFO:teuthology.orchestra.run.smithi002.stdout: "rocksdb_block_size": "4096", 2023-11-05T08:32:39.551 INFO:teuthology.orchestra.run.smithi002.stdout: "rocksdb_bloom_bits_per_key": "20", 2023-11-05T08:32:39.552 INFO:teuthology.orchestra.run.smithi002.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2023-11-05T08:32:39.552 INFO:teuthology.orchestra.run.smithi002.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2023-11-05T08:32:39.552 INFO:teuthology.orchestra.run.smithi002.stdout: "rocksdb_cache_row_ratio": "0.000000", 2023-11-05T08:32:39.552 INFO:teuthology.orchestra.run.smithi002.stdout: "rocksdb_cache_shard_bits": "4", 2023-11-05T08:32:39.552 INFO:teuthology.orchestra.run.smithi002.stdout: "rocksdb_cache_size": "536870912", 2023-11-05T08:32:39.553 INFO:teuthology.orchestra.run.smithi002.stdout: "rocksdb_cache_type": "binned_lru", 2023-11-05T08:32:39.553 INFO:teuthology.orchestra.run.smithi002.stdout: "rocksdb_cf_compact_on_deletion": "true", 2023-11-05T08:32:39.553 INFO:teuthology.orchestra.run.smithi002.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2023-11-05T08:32:39.553 INFO:teuthology.orchestra.run.smithi002.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2023-11-05T08:32:39.553 INFO:teuthology.orchestra.run.smithi002.stdout: "rocksdb_collect_compaction_stats": "false", 2023-11-05T08:32:39.554 INFO:teuthology.orchestra.run.smithi002.stdout: "rocksdb_collect_extended_stats": "false", 2023-11-05T08:32:39.554 INFO:teuthology.orchestra.run.smithi002.stdout: "rocksdb_collect_memory_stats": "false", 2023-11-05T08:32:39.554 INFO:teuthology.orchestra.run.smithi002.stdout: "rocksdb_delete_range_threshold": "1048576", 2023-11-05T08:32:39.554 INFO:teuthology.orchestra.run.smithi002.stdout: "rocksdb_index_type": "binary_search", 2023-11-05T08:32:39.554 INFO:teuthology.orchestra.run.smithi002.stdout: "rocksdb_log_to_ceph_log": "true", 2023-11-05T08:32:39.555 INFO:teuthology.orchestra.run.smithi002.stdout: "rocksdb_metadata_block_size": "4096", 2023-11-05T08:32:39.555 INFO:teuthology.orchestra.run.smithi002.stdout: "rocksdb_partition_filters": "false", 2023-11-05T08:32:39.555 INFO:teuthology.orchestra.run.smithi002.stdout: "rocksdb_perf": "false", 2023-11-05T08:32:39.555 INFO:teuthology.orchestra.run.smithi002.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2023-11-05T08:32:39.555 INFO:teuthology.orchestra.run.smithi002.stdout: "rotating_keys_bootstrap_timeout": "30", 2023-11-05T08:32:39.556 INFO:teuthology.orchestra.run.smithi002.stdout: "rotating_keys_renewal_timeout": "10", 2023-11-05T08:32:39.556 INFO:teuthology.orchestra.run.smithi002.stdout: "run_dir": "/var/run/ceph", 2023-11-05T08:32:39.556 INFO:teuthology.orchestra.run.smithi002.stdout: "seastore_block_create": "true", 2023-11-05T08:32:39.556 INFO:teuthology.orchestra.run.smithi002.stdout: "seastore_cache_lru_size": "67108864", 2023-11-05T08:32:39.557 INFO:teuthology.orchestra.run.smithi002.stdout: "seastore_cbjournal_size": "5368709120", 2023-11-05T08:32:39.557 INFO:teuthology.orchestra.run.smithi002.stdout: "seastore_default_max_object_size": "16777216", 2023-11-05T08:32:39.557 INFO:teuthology.orchestra.run.smithi002.stdout: "seastore_default_object_metadata_reservation": "16777216", 2023-11-05T08:32:39.557 INFO:teuthology.orchestra.run.smithi002.stdout: "seastore_device_size": "53687091200", 2023-11-05T08:32:39.557 INFO:teuthology.orchestra.run.smithi002.stdout: "seastore_journal_batch_capacity": "16", 2023-11-05T08:32:39.558 INFO:teuthology.orchestra.run.smithi002.stdout: "seastore_journal_batch_flush_size": "16777216", 2023-11-05T08:32:39.558 INFO:teuthology.orchestra.run.smithi002.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2023-11-05T08:32:39.558 INFO:teuthology.orchestra.run.smithi002.stdout: "seastore_journal_iodepth_limit": "5", 2023-11-05T08:32:39.558 INFO:teuthology.orchestra.run.smithi002.stdout: "seastore_main_device_type": "SSD", 2023-11-05T08:32:39.559 INFO:teuthology.orchestra.run.smithi002.stdout: "seastore_max_concurrent_transactions": "8", 2023-11-05T08:32:39.559 INFO:teuthology.orchestra.run.smithi002.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2023-11-05T08:32:39.559 INFO:teuthology.orchestra.run.smithi002.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2023-11-05T08:32:39.559 INFO:teuthology.orchestra.run.smithi002.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2023-11-05T08:32:39.559 INFO:teuthology.orchestra.run.smithi002.stdout: "seastore_obj_data_write_amplification": "1.250000", 2023-11-05T08:32:39.560 INFO:teuthology.orchestra.run.smithi002.stdout: "seastore_segment_size": "67108864", 2023-11-05T08:32:39.560 INFO:teuthology.orchestra.run.smithi002.stdout: "set_keepcaps": "false", 2023-11-05T08:32:39.560 INFO:teuthology.orchestra.run.smithi002.stdout: "setgroup": "", 2023-11-05T08:32:39.560 INFO:teuthology.orchestra.run.smithi002.stdout: "setuser": "", 2023-11-05T08:32:39.560 INFO:teuthology.orchestra.run.smithi002.stdout: "setuser_match_path": "", 2023-11-05T08:32:39.561 INFO:teuthology.orchestra.run.smithi002.stdout: "target_max_misplaced_ratio": "0.050000", 2023-11-05T08:32:39.561 INFO:teuthology.orchestra.run.smithi002.stdout: "thp": "false", 2023-11-05T08:32:39.561 INFO:teuthology.orchestra.run.smithi002.stdout: "threadpool_default_timeout": "60", 2023-11-05T08:32:39.561 INFO:teuthology.orchestra.run.smithi002.stdout: "threadpool_empty_queue_max_wait": "2", 2023-11-05T08:32:39.561 INFO:teuthology.orchestra.run.smithi002.stdout: "throttler_perf_counter": "true" 2023-11-05T08:32:39.562 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:32:39.564 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2023-11-05T08:32:39.719 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a: {} 2023-11-05T08:32:39.720 INFO:teuthology.orchestra.run.smithi002.stderr:mon.a: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2023-11-05T08:32:39.732 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b: {} 2023-11-05T08:32:39.732 INFO:teuthology.orchestra.run.smithi002.stderr:mon.b: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2023-11-05T08:32:39.746 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c: {} 2023-11-05T08:32:39.746 INFO:teuthology.orchestra.run.smithi002.stderr:mon.c: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2023-11-05T08:32:39.762 DEBUG:teuthology.orchestra.run.smithi090:> type ceph-objectstore-tool 2023-11-05T08:32:39.768 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2023-11-05T08:32:39.768 DEBUG:teuthology.orchestra.run.smithi053:> type ceph-objectstore-tool 2023-11-05T08:32:39.774 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2023-11-05T08:32:39.774 DEBUG:teuthology.orchestra.run.smithi069:> type ceph-objectstore-tool 2023-11-05T08:32:39.780 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2023-11-05T08:32:39.781 DEBUG:teuthology.orchestra.run.smithi002:> type ceph-objectstore-tool 2023-11-05T08:32:39.785 INFO:teuthology.orchestra.run.smithi002.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2023-11-05T08:32:39.786 INFO:teuthology.run_tasks:Running task rados... 2023-11-05T08:32:39.796 INFO:tasks.rados:Beginning rados... 2023-11-05T08:32:39.797 DEBUG:teuthology.run_tasks:Unwinding manager rados 2023-11-05T08:32:39.814 INFO:tasks.rados:joining rados 2023-11-05T08:32:39.814 INFO:tasks.thrashosds.thrasher:starting do_thrash 2023-11-05T08:32:39.815 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2023-11-05T08:32:39.815 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-11-05T08:32:39.815 INFO:tasks.thrashosds.thrasher:inject_pause on osd.8 2023-11-05T08:32:39.815 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-11-05T08:32:39.816 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-11-05T08:32:39.816 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok config set filestore_inject_stall 3 2023-11-05T08:32:39.816 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2023-11-05T08:32:39.817 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2023-11-05T08:32:39.817 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-05T08:32:39.817 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2023-11-05T08:32:39.818 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-11-05T08:32:39.818 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2023-11-05T08:32:39.819 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-11-05T08:32:39.819 INFO:tasks.rados:clients are ['client.0'] 2023-11-05T08:32:39.819 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set shecprofile c=2 crush-failure-domain=osd k=4 m=3 name=shecprofile plugin=shec 2023-11-05T08:32:39.822 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:32:39.820+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:32:39.918 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:32:39.919+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:32:39.951 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:32:39.951 INFO:teuthology.orchestra.run.smithi002.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2023-11-05T08:32:39.952 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:32:39.977 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0: {} 2023-11-05T08:32:39.977 INFO:teuthology.orchestra.run.smithi002.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-11-05T08:32:39.997 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1: {} 2023-11-05T08:32:39.998 INFO:teuthology.orchestra.run.smithi002.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-11-05T08:32:40.004 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:40.004 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:40.019 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-11-05T08:32:40.017+0000 7f966619d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35976) UID: 0 2023-11-05T08:32:40.021 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-11-05T08:32:40.024 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2: {} 2023-11-05T08:32:40.024 INFO:teuthology.orchestra.run.smithi002.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-11-05T08:32:40.044 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3: {} 2023-11-05T08:32:40.045 INFO:teuthology.orchestra.run.smithi002.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-11-05T08:32:40.061 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4: {} 2023-11-05T08:32:40.062 INFO:teuthology.orchestra.run.smithi002.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-11-05T08:32:40.082 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5: {} 2023-11-05T08:32:40.082 INFO:teuthology.orchestra.run.smithi002.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-11-05T08:32:40.102 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6: {} 2023-11-05T08:32:40.102 INFO:teuthology.orchestra.run.smithi002.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-11-05T08:32:40.120 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:32:40.118+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:32:40.121 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7: {} 2023-11-05T08:32:40.122 INFO:teuthology.orchestra.run.smithi002.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-11-05T08:32:40.135 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:40.136 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:40.137 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8: {} 2023-11-05T08:32:40.137 INFO:teuthology.orchestra.run.smithi002.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-11-05T08:32:40.149 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-11-05T08:32:40.159 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9: {} 2023-11-05T08:32:40.160 INFO:teuthology.orchestra.run.smithi002.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-11-05T08:32:40.179 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10: {} 2023-11-05T08:32:40.179 INFO:teuthology.orchestra.run.smithi002.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-11-05T08:32:40.199 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11: {} 2023-11-05T08:32:40.199 INFO:teuthology.orchestra.run.smithi002.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-11-05T08:32:40.214 INFO:teuthology.orchestra.run.smithi002.stdout:osd.12: {} 2023-11-05T08:32:40.215 INFO:teuthology.orchestra.run.smithi002.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-11-05T08:32:40.221 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:32:40.218+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:32:40.233 INFO:teuthology.orchestra.run.smithi002.stdout:osd.13: {} 2023-11-05T08:32:40.233 INFO:teuthology.orchestra.run.smithi002.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-11-05T08:32:40.253 INFO:teuthology.orchestra.run.smithi002.stdout:osd.14: {} 2023-11-05T08:32:40.253 INFO:teuthology.orchestra.run.smithi002.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-11-05T08:32:40.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-05T08:32:40.251+0000 7fe319ad5640 0 -- 172.21.15.2:0/184908875 >> [v2:172.21.15.90:6824/739151808,v1:172.21.15.90:6825/739151808] conn(0x7fe2f0033320 msgr2=0x7fe2f0035710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-11-05T08:32:40.267 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:40.267 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:40.279 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-11-05T08:32:40.323 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:32:40.323+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:32:40.423 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:32:40.421+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:32:40.440 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:40.440 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:40.452 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-11-05T08:32:40.474 INFO:teuthology.orchestra.run.smithi002.stdout:osd.15: {} 2023-11-05T08:32:40.475 INFO:teuthology.orchestra.run.smithi002.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-11-05T08:32:40.525 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:32:40.522+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:32:40.594 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:40.594 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:40.607 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-11-05T08:32:40.625 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:32:40.622+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:32:40.726 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:32:40.726+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:32:40.727 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:40.727 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:40.742 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2023-11-05T08:32:40.827 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:32:40.828+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:32:40.862 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:40.863 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:40.877 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2023-11-05T08:32:40.928 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:32:40.927+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:32:40.996 INFO:teuthology.orchestra.run.smithi002.stderr:load: shec 2023-11-05T08:32:40.997 INFO:teuthology.orchestra.run.smithi002.stderr:noscrub is set 2023-11-05T08:32:41.005 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:41.006 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:41.020 INFO:tasks.rados:starting run 0 out of 1 2023-11-05T08:32:41.020 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2023-11-05T08:32:41.021 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 16 erasure shecprofile 2023-11-05T08:32:41.023 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2023-11-05T08:32:41.030 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:32:41.028+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:32:41.130 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:32:41.128+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:32:41.173 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:41.173 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:41.186 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2023-11-05T08:32:41.231 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:32:41.228+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:32:41.298 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:41.298 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:41.312 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2023-11-05T08:32:41.359 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:32:41.359+0000 7fdc5cd58640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:32:41.425 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:41.425 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:41.433 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:32:41.428+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:32:41.437 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2023-11-05T08:32:41.534 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:32:41.531+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:32:41.603 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:41.603 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:41.617 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-11-05T08:32:41.659 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:32:41.659+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:32:41.736 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-11-05T08:32:41.733+0000 7f966619d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35976) UID: 0 2023-11-05T08:32:41.753 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:41.753 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:41.767 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-11-05T08:32:41.836 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:32:41.835+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:32:41.891 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:41.891 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:41.906 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-11-05T08:32:41.937 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:32:41.932+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:32:42.032 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:42.033 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:42.038 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:32:42.037+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:32:42.047 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-11-05T08:32:42.139 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:32:42.137+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:32:42.216 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:42.217 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:42.232 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-11-05T08:32:42.240 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:32:42.236+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:32:42.341 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:32:42.339+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:32:42.356 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:42.356 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:42.370 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-11-05T08:32:42.442 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:32:42.441+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:32:42.491 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-05T08:32:42.494 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:42.495 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:42.509 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2023-11-05T08:32:42.554 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:32:42.553+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:32:42.643 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:32:42.641+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:32:42.652 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0: {} 2023-11-05T08:32:42.653 INFO:teuthology.orchestra.run.smithi002.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-11-05T08:32:42.673 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1: {} 2023-11-05T08:32:42.673 INFO:teuthology.orchestra.run.smithi002.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-11-05T08:32:42.676 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:42.676 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:42.690 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2023-11-05T08:32:42.693 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2: {} 2023-11-05T08:32:42.694 INFO:teuthology.orchestra.run.smithi002.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-11-05T08:32:42.715 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3: {} 2023-11-05T08:32:42.715 INFO:teuthology.orchestra.run.smithi002.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-11-05T08:32:42.736 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4: {} 2023-11-05T08:32:42.736 INFO:teuthology.orchestra.run.smithi002.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-11-05T08:32:42.744 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:32:42.741+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:32:42.754 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5: {} 2023-11-05T08:32:42.755 INFO:teuthology.orchestra.run.smithi002.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-11-05T08:32:42.775 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6: {} 2023-11-05T08:32:42.776 INFO:teuthology.orchestra.run.smithi002.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-11-05T08:32:42.795 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7: {} 2023-11-05T08:32:42.796 INFO:teuthology.orchestra.run.smithi002.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-11-05T08:32:42.814 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8: {} 2023-11-05T08:32:42.815 INFO:teuthology.orchestra.run.smithi002.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-11-05T08:32:42.818 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:42.818 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:42.833 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2023-11-05T08:32:42.837 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9: {} 2023-11-05T08:32:42.838 INFO:teuthology.orchestra.run.smithi002.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-11-05T08:32:42.845 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:32:42.841+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:32:42.860 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10: {} 2023-11-05T08:32:42.860 INFO:teuthology.orchestra.run.smithi002.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-11-05T08:32:42.883 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11: {} 2023-11-05T08:32:42.884 INFO:teuthology.orchestra.run.smithi002.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-11-05T08:32:42.903 INFO:teuthology.orchestra.run.smithi002.stdout:osd.12: {} 2023-11-05T08:32:42.903 INFO:teuthology.orchestra.run.smithi002.stderr:osd.12: osd_enable_op_tracker = 'true' 2023-11-05T08:32:42.924 INFO:teuthology.orchestra.run.smithi002.stdout:osd.13: {} 2023-11-05T08:32:42.924 INFO:teuthology.orchestra.run.smithi002.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-11-05T08:32:42.946 INFO:teuthology.orchestra.run.smithi002.stdout:osd.14: {} 2023-11-05T08:32:42.947 INFO:teuthology.orchestra.run.smithi002.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-11-05T08:32:42.947 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:32:42.947+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:32:42.961 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:32:42.961 INFO:teuthology.orchestra.run.smithi053.stdout: "size": 20, 2023-11-05T08:32:42.961 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 600, 2023-11-05T08:32:42.961 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [] 2023-11-05T08:32:42.961 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:32:42.970 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-05T08:32:42.967+0000 7f62ff7fe640 0 -- 172.21.15.2:0/745103531 >> [v2:172.21.15.90:6824/739151808,v1:172.21.15.90:6825/739151808] conn(0x7f63002237b0 msgr2=0x7f6300225ba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-11-05T08:32:42.975 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2023-11-05T08:32:42.976 INFO:teuthology.orchestra.run.smithi002.stdout:osd.15: {} 2023-11-05T08:32:43.002 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'unique_pool_0' created 2023-11-05T08:32:43.020 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-11-05T08:32:43.028 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2023-11-05T08:32:43.047 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:32:43.044+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:32:43.101 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:32:43.102 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2023-11-05T08:32:43.102 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 0 2023-11-05T08:32:43.102 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:32:43.116 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2023-11-05T08:32:43.148 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:32:43.148+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:32:43.241 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:32:43.241 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2023-11-05T08:32:43.242 INFO:teuthology.orchestra.run.smithi053.stdout: "complaint_time": 30, 2023-11-05T08:32:43.242 INFO:teuthology.orchestra.run.smithi053.stdout: "num_blocked_ops": 0 2023-11-05T08:32:43.242 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:32:43.249 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:32:43.245+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:32:43.255 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2023-11-05T08:32:43.349 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:32:43.347+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:32:43.386 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:32:43.387 INFO:teuthology.orchestra.run.smithi053.stdout: "size": 20, 2023-11-05T08:32:43.387 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 600, 2023-11-05T08:32:43.387 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2023-11-05T08:32:43.387 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.388 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:32:43.388 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:28.710073+0000", 2023-11-05T08:32:43.388 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 14.671958844000001, 2023-11-05T08:32:43.388 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.052834791999999998, 2023-11-05T08:32:43.388 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:43.389 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:43.389 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:32:43.389 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:32:43.389 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:32:43.389 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 1 2023-11-05T08:32:43.390 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.390 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:43.390 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.390 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:43.391 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710073+0000", 2023-11-05T08:32:43.391 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.391 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.391 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.391 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:43.391 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710073+0000", 2023-11-05T08:32:43.392 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.392 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.392 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.392 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:43.392 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710076+0000", 2023-11-05T08:32:43.392 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1729999999999999e-06 2023-11-05T08:32:43.393 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.393 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.393 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:43.393 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710108+0000", 2023-11-05T08:32:43.393 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1458000000000003e-05 2023-11-05T08:32:43.393 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.394 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.394 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:43.394 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710140+0000", 2023-11-05T08:32:43.394 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2532e-05 2023-11-05T08:32:43.394 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.395 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.395 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:43.395 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710165+0000", 2023-11-05T08:32:43.395 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5205e-05 2023-11-05T08:32:43.395 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.396 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.396 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:43.396 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710311+0000", 2023-11-05T08:32:43.396 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014532799999999999 2023-11-05T08:32:43.396 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.396 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.397 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2023-11-05T08:32:43.397 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710327+0000", 2023-11-05T08:32:43.397 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6059e-05 2023-11-05T08:32:43.397 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.397 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.397 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:43.398 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.759348+0000", 2023-11-05T08:32:43.398 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.049021481999999998 2023-11-05T08:32:43.398 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.398 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.399 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:32:43.399 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.759880+0000", 2023-11-05T08:32:43.399 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00053197400000000005 2023-11-05T08:32:43.399 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.399 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.399 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:32:43.400 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.760478+0000", 2023-11-05T08:32:43.400 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00059839300000000002 2023-11-05T08:32:43.400 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.400 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.400 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:32:43.401 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.761955+0000", 2023-11-05T08:32:43.401 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001476991 2023-11-05T08:32:43.401 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.401 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.401 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:32:43.402 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.762611+0000", 2023-11-05T08:32:43.402 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00065598700000000002 2023-11-05T08:32:43.402 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.402 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.402 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:32:43.403 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.762782+0000", 2023-11-05T08:32:43.403 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00017064000000000001 2023-11-05T08:32:43.403 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.403 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.404 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:43.404 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.762908+0000", 2023-11-05T08:32:43.404 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012557000000000001 2023-11-05T08:32:43.404 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.404 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:43.405 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.405 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.405 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.405 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:32:43.405 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.151546+0000", 2023-11-05T08:32:43.406 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 14.230485496, 2023-11-05T08:32:43.406 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.018837386000000001, 2023-11-05T08:32:43.406 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:43.406 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:43.406 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:32:43.407 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:32:43.407 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:32:43.407 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 113 2023-11-05T08:32:43.407 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.407 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:43.408 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.408 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:43.408 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151546+0000", 2023-11-05T08:32:43.408 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.408 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.409 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.409 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:43.409 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151546+0000", 2023-11-05T08:32:43.409 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.409 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.409 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.410 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:43.410 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151547+0000", 2023-11-05T08:32:43.410 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9699999999999995e-07 2023-11-05T08:32:43.410 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.410 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.411 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:43.411 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151553+0000", 2023-11-05T08:32:43.411 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9830000000000001e-06 2023-11-05T08:32:43.411 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.411 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.412 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:43.413 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151562+0000", 2023-11-05T08:32:43.413 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.5060000000000001e-06 2023-11-05T08:32:43.413 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.414 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.414 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:43.414 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151569+0000", 2023-11-05T08:32:43.414 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9160000000000004e-06 2023-11-05T08:32:43.414 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.415 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.415 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:43.415 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.163944+0000", 2023-11-05T08:32:43.415 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012374703000000001 2023-11-05T08:32:43.415 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.416 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.416 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:32:43.416 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.164091+0000", 2023-11-05T08:32:43.416 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014739199999999999 2023-11-05T08:32:43.416 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.417 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.417 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:32:43.417 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.164295+0000", 2023-11-05T08:32:43.417 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00020334899999999999 2023-11-05T08:32:43.417 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.418 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.418 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:32:43.418 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.167412+0000", 2023-11-05T08:32:43.418 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.003117529 2023-11-05T08:32:43.418 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.419 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.419 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:32:43.419 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.170241+0000", 2023-11-05T08:32:43.419 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0028283890000000002 2023-11-05T08:32:43.419 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.419 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.420 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:32:43.420 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.170326+0000", 2023-11-05T08:32:43.420 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.5581999999999995e-05 2023-11-05T08:32:43.420 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.420 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.421 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:43.421 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.170384+0000", 2023-11-05T08:32:43.421 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7340000000000003e-05 2023-11-05T08:32:43.421 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.421 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:43.421 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.421 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.422 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.422 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:32:43.422 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.151583+0000", 2023-11-05T08:32:43.422 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 14.230449051000001, 2023-11-05T08:32:43.422 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.020286660000000001, 2023-11-05T08:32:43.423 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:43.423 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:43.423 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:32:43.423 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:32:43.423 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:32:43.424 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 114 2023-11-05T08:32:43.424 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.424 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:43.424 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.424 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:43.425 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151583+0000", 2023-11-05T08:32:43.425 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.425 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.425 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.425 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:43.425 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151583+0000", 2023-11-05T08:32:43.426 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.426 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.426 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.426 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:43.426 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151583+0000", 2023-11-05T08:32:43.427 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.2600000000000002e-07 2023-11-05T08:32:43.427 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.427 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.428 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:43.428 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151589+0000", 2023-11-05T08:32:43.428 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5099999999999998e-06 2023-11-05T08:32:43.428 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.428 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.429 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:43.429 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151598+0000", 2023-11-05T08:32:43.429 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1730000000000001e-06 2023-11-05T08:32:43.429 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.430 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.430 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:43.431 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151608+0000", 2023-11-05T08:32:43.431 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0059e-05 2023-11-05T08:32:43.431 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.432 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.432 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:43.432 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.165015+0000", 2023-11-05T08:32:43.433 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.013407207000000001 2023-11-05T08:32:43.433 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.433 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.433 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:32:43.434 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.165268+0000", 2023-11-05T08:32:43.434 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00025284999999999999 2023-11-05T08:32:43.435 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.435 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.435 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:32:43.435 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.165635+0000", 2023-11-05T08:32:43.436 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00036695899999999998 2023-11-05T08:32:43.436 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.436 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.436 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:32:43.436 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.170105+0000", 2023-11-05T08:32:43.437 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0044696960000000004 2023-11-05T08:32:43.437 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.437 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.437 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:32:43.437 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.171719+0000", 2023-11-05T08:32:43.437 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001613855 2023-11-05T08:32:43.438 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.438 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.438 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:32:43.438 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.171810+0000", 2023-11-05T08:32:43.438 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1625000000000005e-05 2023-11-05T08:32:43.439 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.439 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.439 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:43.439 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.171869+0000", 2023-11-05T08:32:43.439 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.91e-05 2023-11-05T08:32:43.439 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.440 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:43.440 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.440 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.440 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.440 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:32:43.441 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.151660+0000", 2023-11-05T08:32:43.441 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 14.230371251999999, 2023-11-05T08:32:43.441 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022553719, 2023-11-05T08:32:43.441 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:43.441 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:43.441 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:32:43.442 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:32:43.442 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:32:43.442 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 115 2023-11-05T08:32:43.442 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.442 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:43.443 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.443 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:43.443 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151660+0000", 2023-11-05T08:32:43.443 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.443 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.444 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.444 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:43.444 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151660+0000", 2023-11-05T08:32:43.444 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.444 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.444 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.444 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:43.445 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151661+0000", 2023-11-05T08:32:43.445 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.5600000000000004e-07 2023-11-05T08:32:43.445 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.445 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.445 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:43.446 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151799+0000", 2023-11-05T08:32:43.446 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000137207 2023-11-05T08:32:43.446 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.446 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.446 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:43.447 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151814+0000", 2023-11-05T08:32:43.447 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4885000000000001e-05 2023-11-05T08:32:43.447 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.447 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.447 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:43.448 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151827+0000", 2023-11-05T08:32:43.448 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3413e-05 2023-11-05T08:32:43.448 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.448 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.448 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:43.448 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.167513+0000", 2023-11-05T08:32:43.449 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.015686091999999999 2023-11-05T08:32:43.449 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.449 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.449 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:32:43.449 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.167652+0000", 2023-11-05T08:32:43.450 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013890500000000001 2023-11-05T08:32:43.450 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.450 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.450 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:32:43.450 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.167860+0000", 2023-11-05T08:32:43.450 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00020838599999999999 2023-11-05T08:32:43.451 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.451 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.451 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:32:43.451 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.171602+0000", 2023-11-05T08:32:43.451 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0037417449999999999 2023-11-05T08:32:43.452 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.452 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.452 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:32:43.452 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174068+0000", 2023-11-05T08:32:43.452 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0024658010000000001 2023-11-05T08:32:43.453 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.453 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.453 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:32:43.453 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174152+0000", 2023-11-05T08:32:43.453 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.3752999999999996e-05 2023-11-05T08:32:43.454 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.454 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.454 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:43.454 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174214+0000", 2023-11-05T08:32:43.454 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.2576000000000005e-05 2023-11-05T08:32:43.454 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.455 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:43.455 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.455 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.455 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.455 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:32:43.456 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.151849+0000", 2023-11-05T08:32:43.456 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 14.230182896000001, 2023-11-05T08:32:43.456 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022629176000000001, 2023-11-05T08:32:43.456 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:43.456 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:43.457 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:32:43.457 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:32:43.457 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:32:43.457 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 116 2023-11-05T08:32:43.457 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.458 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:43.458 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.458 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:43.458 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151849+0000", 2023-11-05T08:32:43.458 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.459 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.459 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.459 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:43.459 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151849+0000", 2023-11-05T08:32:43.459 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.459 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.460 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.460 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:43.460 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151850+0000", 2023-11-05T08:32:43.460 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.037e-06 2023-11-05T08:32:43.460 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.461 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.461 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:43.461 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151858+0000", 2023-11-05T08:32:43.461 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.3389999999999993e-06 2023-11-05T08:32:43.461 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.462 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.462 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:43.462 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151868+0000", 2023-11-05T08:32:43.462 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6670000000000008e-06 2023-11-05T08:32:43.462 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.463 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.463 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:43.463 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151875+0000", 2023-11-05T08:32:43.463 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9229999999999996e-06 2023-11-05T08:32:43.463 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.463 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.464 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:43.464 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.168777+0000", 2023-11-05T08:32:43.464 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.016901884999999998 2023-11-05T08:32:43.464 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.464 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.464 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:32:43.465 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.169009+0000", 2023-11-05T08:32:43.465 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00023236400000000001 2023-11-05T08:32:43.465 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.465 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.465 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:32:43.466 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.169365+0000", 2023-11-05T08:32:43.466 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00035625300000000001 2023-11-05T08:32:43.466 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.467 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.467 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:32:43.467 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.171624+0000", 2023-11-05T08:32:43.467 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0022587169999999999 2023-11-05T08:32:43.467 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.467 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.468 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:32:43.468 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174314+0000", 2023-11-05T08:32:43.468 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002690263 2023-11-05T08:32:43.468 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.468 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.469 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:32:43.469 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174405+0000", 2023-11-05T08:32:43.469 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1064999999999996e-05 2023-11-05T08:32:43.469 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.469 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.469 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:43.470 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174478+0000", 2023-11-05T08:32:43.470 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.2663000000000005e-05 2023-11-05T08:32:43.470 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.470 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:43.470 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.470 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.471 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.471 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:32:43.471 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.151886+0000", 2023-11-05T08:32:43.471 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 14.230145480999999, 2023-11-05T08:32:43.471 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.023174162000000002, 2023-11-05T08:32:43.471 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:43.472 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:43.472 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:32:43.472 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:32:43.472 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:32:43.472 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 117 2023-11-05T08:32:43.472 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.473 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:43.473 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.473 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:43.473 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151886+0000", 2023-11-05T08:32:43.473 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.474 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.474 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.474 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:43.474 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151886+0000", 2023-11-05T08:32:43.474 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.474 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.475 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.475 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:43.475 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151887+0000", 2023-11-05T08:32:43.475 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.2300000000000001e-07 2023-11-05T08:32:43.475 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.476 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.476 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:43.476 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151893+0000", 2023-11-05T08:32:43.476 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.19e-06 2023-11-05T08:32:43.476 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.476 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.477 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:43.477 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151905+0000", 2023-11-05T08:32:43.477 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1883e-05 2023-11-05T08:32:43.477 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.477 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.477 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:43.478 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151912+0000", 2023-11-05T08:32:43.478 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.0690000000000004e-06 2023-11-05T08:32:43.478 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.478 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.478 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:43.479 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.170509+0000", 2023-11-05T08:32:43.479 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.018596686000000001 2023-11-05T08:32:43.479 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.479 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.479 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:32:43.479 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.170791+0000", 2023-11-05T08:32:43.480 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00028221799999999999 2023-11-05T08:32:43.480 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.480 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.480 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:32:43.480 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.171137+0000", 2023-11-05T08:32:43.480 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000346454 2023-11-05T08:32:43.481 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.481 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.481 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:32:43.481 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.172413+0000", 2023-11-05T08:32:43.481 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0012752340000000001 2023-11-05T08:32:43.481 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.482 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.482 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:32:43.482 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174926+0000", 2023-11-05T08:32:43.482 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0025137050000000002 2023-11-05T08:32:43.482 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.483 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.483 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:32:43.483 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.175007+0000", 2023-11-05T08:32:43.483 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.0413000000000004e-05 2023-11-05T08:32:43.483 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.483 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.484 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:43.484 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.175060+0000", 2023-11-05T08:32:43.484 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3687000000000003e-05 2023-11-05T08:32:43.484 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.484 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:43.485 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.485 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.485 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.485 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:32:43.485 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.352803+0000", 2023-11-05T08:32:43.485 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 14.029228549000001, 2023-11-05T08:32:43.486 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017043243, 2023-11-05T08:32:43.486 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:43.486 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:43.486 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:32:43.486 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:32:43.486 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:32:43.487 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 140 2023-11-05T08:32:43.487 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.487 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:43.487 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.487 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:43.488 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.352803+0000", 2023-11-05T08:32:43.488 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.488 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.488 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.488 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:43.488 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.352803+0000", 2023-11-05T08:32:43.489 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.489 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.489 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.489 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:43.489 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.352804+0000", 2023-11-05T08:32:43.490 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1600000000000001e-07 2023-11-05T08:32:43.490 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.490 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.490 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:43.490 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.352932+0000", 2023-11-05T08:32:43.491 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012813399999999999 2023-11-05T08:32:43.491 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.491 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.491 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:43.491 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.352943+0000", 2023-11-05T08:32:43.491 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0601000000000001e-05 2023-11-05T08:32:43.492 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.492 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.492 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:43.492 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.352948+0000", 2023-11-05T08:32:43.492 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3029999999999999e-06 2023-11-05T08:32:43.493 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.493 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.493 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:43.493 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.362447+0000", 2023-11-05T08:32:43.493 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0094990089999999992 2023-11-05T08:32:43.493 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.494 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.494 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:32:43.494 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.362610+0000", 2023-11-05T08:32:43.494 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00016328600000000001 2023-11-05T08:32:43.494 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.495 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.495 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:32:43.495 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.362809+0000", 2023-11-05T08:32:43.495 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00019889199999999999 2023-11-05T08:32:43.495 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.495 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.496 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:32:43.496 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.364900+0000", 2023-11-05T08:32:43.496 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002091369 2023-11-05T08:32:43.496 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.496 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.496 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:32:43.497 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.369777+0000", 2023-11-05T08:32:43.497 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0048767860000000001 2023-11-05T08:32:43.497 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.497 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.497 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:32:43.497 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.369819+0000", 2023-11-05T08:32:43.498 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1653999999999998e-05 2023-11-05T08:32:43.498 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.498 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.498 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:43.498 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.369846+0000", 2023-11-05T08:32:43.498 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7593000000000002e-05 2023-11-05T08:32:43.499 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.499 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:43.499 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.499 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.499 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.500 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:32:43.500 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.352968+0000", 2023-11-05T08:32:43.500 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 14.029063526, 2023-11-05T08:32:43.500 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017162192, 2023-11-05T08:32:43.500 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:43.500 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:43.501 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:32:43.501 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:32:43.501 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:32:43.501 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 141 2023-11-05T08:32:43.501 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.501 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:43.502 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.502 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:43.502 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.352968+0000", 2023-11-05T08:32:43.502 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.502 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.503 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.503 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:43.503 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.352968+0000", 2023-11-05T08:32:43.503 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.503 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.503 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.504 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:43.504 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.352969+0000", 2023-11-05T08:32:43.504 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5200000000000002e-07 2023-11-05T08:32:43.504 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.504 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.504 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:43.505 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.352972+0000", 2023-11-05T08:32:43.505 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.7249999999999999e-06 2023-11-05T08:32:43.505 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.505 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.505 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:43.506 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.352981+0000", 2023-11-05T08:32:43.506 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9749999999999996e-06 2023-11-05T08:32:43.506 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.506 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.506 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:43.507 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.352985+0000", 2023-11-05T08:32:43.507 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.0400000000000003e-06 2023-11-05T08:32:43.507 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.507 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.507 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:43.507 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.364169+0000", 2023-11-05T08:32:43.508 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011183451000000001 2023-11-05T08:32:43.508 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.508 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.508 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:32:43.508 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.364285+0000", 2023-11-05T08:32:43.509 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00011637900000000001 2023-11-05T08:32:43.509 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.509 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.509 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:32:43.509 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.364464+0000", 2023-11-05T08:32:43.510 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00017891900000000001 2023-11-05T08:32:43.510 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.510 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.510 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:32:43.510 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.365555+0000", 2023-11-05T08:32:43.511 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0010906539999999999 2023-11-05T08:32:43.511 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.511 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.511 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:32:43.511 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.369947+0000", 2023-11-05T08:32:43.511 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.004391697 2023-11-05T08:32:43.512 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.512 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.512 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:32:43.512 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.370078+0000", 2023-11-05T08:32:43.512 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013115699999999999 2023-11-05T08:32:43.512 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.513 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.513 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:43.513 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.370130+0000", 2023-11-05T08:32:43.513 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2743e-05 2023-11-05T08:32:43.513 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.514 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:43.514 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.514 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.514 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.515 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:161 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:32:43.515 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.670703+0000", 2023-11-05T08:32:43.515 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 13.711328588000001, 2023-11-05T08:32:43.515 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.016216886, 2023-11-05T08:32:43.516 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:43.516 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:43.516 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:32:43.516 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:32:43.516 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:32:43.517 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 161 2023-11-05T08:32:43.517 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.517 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:43.517 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.517 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:43.518 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670703+0000", 2023-11-05T08:32:43.518 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.518 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.518 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.518 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:43.519 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670703+0000", 2023-11-05T08:32:43.519 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.519 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.519 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.519 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:43.520 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670704+0000", 2023-11-05T08:32:43.520 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.0900000000000001e-07 2023-11-05T08:32:43.520 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.520 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.520 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:43.521 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670819+0000", 2023-11-05T08:32:43.521 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000115011 2023-11-05T08:32:43.521 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.521 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.521 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:43.521 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670831+0000", 2023-11-05T08:32:43.522 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2509e-05 2023-11-05T08:32:43.522 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.522 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.522 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:43.522 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670840+0000", 2023-11-05T08:32:43.523 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.2970000000000004e-06 2023-11-05T08:32:43.523 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.523 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.523 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:43.523 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.682384+0000", 2023-11-05T08:32:43.524 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011544538 2023-11-05T08:32:43.524 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.524 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.524 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:32:43.524 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.682535+0000", 2023-11-05T08:32:43.525 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00015100199999999999 2023-11-05T08:32:43.525 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.525 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.526 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:32:43.526 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.682757+0000", 2023-11-05T08:32:43.527 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00022158400000000001 2023-11-05T08:32:43.527 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.527 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.528 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:32:43.528 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.685826+0000", 2023-11-05T08:32:43.528 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0030695990000000001 2023-11-05T08:32:43.528 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.529 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.529 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:32:43.529 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.686670+0000", 2023-11-05T08:32:43.530 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00084375400000000005 2023-11-05T08:32:43.530 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.530 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.530 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:32:43.530 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.686821+0000", 2023-11-05T08:32:43.531 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00015110500000000001 2023-11-05T08:32:43.531 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.531 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.531 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:43.531 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.686920+0000", 2023-11-05T08:32:43.532 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.8777999999999995e-05 2023-11-05T08:32:43.532 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.532 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:43.532 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.532 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.533 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.533 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:162 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:32:43.533 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.670856+0000", 2023-11-05T08:32:43.533 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 13.71117609, 2023-11-05T08:32:43.534 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.016361348000000001, 2023-11-05T08:32:43.534 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:43.534 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:43.534 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:32:43.534 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:32:43.535 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:32:43.535 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 162 2023-11-05T08:32:43.535 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.535 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:43.535 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.536 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:43.536 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670856+0000", 2023-11-05T08:32:43.536 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.536 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.536 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.537 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:43.537 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670856+0000", 2023-11-05T08:32:43.537 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.537 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.537 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.538 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:43.538 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670857+0000", 2023-11-05T08:32:43.538 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.8999999999999995e-07 2023-11-05T08:32:43.538 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.538 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.539 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:43.539 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670862+0000", 2023-11-05T08:32:43.539 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9030000000000001e-06 2023-11-05T08:32:43.539 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.539 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.539 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:43.540 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670872+0000", 2023-11-05T08:32:43.540 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.5659999999999996e-06 2023-11-05T08:32:43.540 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.540 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.541 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:43.541 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670879+0000", 2023-11-05T08:32:43.541 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.08e-06 2023-11-05T08:32:43.541 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.542 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.542 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:43.542 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.683898+0000", 2023-11-05T08:32:43.542 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.013018897 2023-11-05T08:32:43.542 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.543 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.543 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:32:43.543 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.684137+0000", 2023-11-05T08:32:43.544 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00023945699999999999 2023-11-05T08:32:43.544 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.545 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.545 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:32:43.545 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.684487+0000", 2023-11-05T08:32:43.546 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00034995099999999998 2023-11-05T08:32:43.546 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.546 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.546 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:32:43.547 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.686952+0000", 2023-11-05T08:32:43.547 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002464548 2023-11-05T08:32:43.547 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.548 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.548 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:32:43.548 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.687067+0000", 2023-11-05T08:32:43.549 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00011500799999999999 2023-11-05T08:32:43.549 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.549 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.549 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:32:43.550 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.687157+0000", 2023-11-05T08:32:43.550 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.0271000000000004e-05 2023-11-05T08:32:43.550 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.550 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.550 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:43.551 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.687217+0000", 2023-11-05T08:32:43.551 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9777e-05 2023-11-05T08:32:43.551 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.551 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:43.551 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.552 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.552 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.552 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:163 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:32:43.552 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.670890+0000", 2023-11-05T08:32:43.552 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 13.711141252000001, 2023-11-05T08:32:43.552 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.02132307, 2023-11-05T08:32:43.553 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:43.553 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:43.553 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:32:43.553 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:32:43.553 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:32:43.554 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 163 2023-11-05T08:32:43.554 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.554 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:43.554 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.554 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:43.555 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670890+0000", 2023-11-05T08:32:43.555 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.555 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.555 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.555 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:43.555 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670890+0000", 2023-11-05T08:32:43.556 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.556 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.556 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.556 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:43.556 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670891+0000", 2023-11-05T08:32:43.557 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3300000000000002e-07 2023-11-05T08:32:43.557 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.557 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.557 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:43.557 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670897+0000", 2023-11-05T08:32:43.557 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.4600000000000002e-06 2023-11-05T08:32:43.558 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.558 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.558 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:43.558 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670906+0000", 2023-11-05T08:32:43.559 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.3130000000000004e-06 2023-11-05T08:32:43.559 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.559 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.559 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:43.559 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670913+0000", 2023-11-05T08:32:43.560 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9859999999999997e-06 2023-11-05T08:32:43.560 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.560 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.560 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:43.560 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.687360+0000", 2023-11-05T08:32:43.560 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.016447244 2023-11-05T08:32:43.561 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.561 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.561 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:32:43.561 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.687588+0000", 2023-11-05T08:32:43.562 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00022757699999999999 2023-11-05T08:32:43.562 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.562 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.562 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:32:43.562 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.687946+0000", 2023-11-05T08:32:43.562 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00035789399999999999 2023-11-05T08:32:43.563 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.563 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.563 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:32:43.563 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.691929+0000", 2023-11-05T08:32:43.563 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0039835690000000002 2023-11-05T08:32:43.564 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.564 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.564 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:32:43.564 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.691970+0000", 2023-11-05T08:32:43.564 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.0351e-05 2023-11-05T08:32:43.565 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.565 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.565 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:32:43.565 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692111+0000", 2023-11-05T08:32:43.565 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014114499999999999 2023-11-05T08:32:43.566 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.566 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.566 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:43.566 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692214+0000", 2023-11-05T08:32:43.566 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000102898 2023-11-05T08:32:43.567 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.567 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:43.567 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.567 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.567 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.567 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:32:43.568 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.670924+0000", 2023-11-05T08:32:43.568 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 13.711107844000001, 2023-11-05T08:32:43.568 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.021621676999999999, 2023-11-05T08:32:43.568 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:43.568 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:43.569 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:32:43.569 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:32:43.569 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:32:43.569 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 164 2023-11-05T08:32:43.569 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.569 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:43.570 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.570 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:43.570 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670924+0000", 2023-11-05T08:32:43.570 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.570 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.571 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.571 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:43.571 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670924+0000", 2023-11-05T08:32:43.571 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.571 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.571 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.572 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:43.572 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670925+0000", 2023-11-05T08:32:43.572 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.8299999999999996e-07 2023-11-05T08:32:43.572 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.572 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.572 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:43.573 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671022+0000", 2023-11-05T08:32:43.573 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.7445e-05 2023-11-05T08:32:43.573 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.573 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.573 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:43.573 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671033+0000", 2023-11-05T08:32:43.574 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1206000000000001e-05 2023-11-05T08:32:43.574 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.574 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.574 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:43.574 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671045+0000", 2023-11-05T08:32:43.575 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1797000000000001e-05 2023-11-05T08:32:43.575 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.575 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.575 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:43.575 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.688874+0000", 2023-11-05T08:32:43.575 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017829424 2023-11-05T08:32:43.576 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.576 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.576 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:32:43.576 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.689097+0000", 2023-11-05T08:32:43.576 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000222828 2023-11-05T08:32:43.576 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.577 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.577 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:32:43.577 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.689444+0000", 2023-11-05T08:32:43.577 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00034672099999999998 2023-11-05T08:32:43.577 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.578 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.578 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:32:43.578 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692240+0000", 2023-11-05T08:32:43.578 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0027959030000000002 2023-11-05T08:32:43.578 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.579 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.579 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:32:43.579 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692339+0000", 2023-11-05T08:32:43.579 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.8801000000000005e-05 2023-11-05T08:32:43.579 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.579 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.580 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:32:43.580 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692435+0000", 2023-11-05T08:32:43.580 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6311999999999995e-05 2023-11-05T08:32:43.580 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.580 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.580 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:43.581 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692546+0000", 2023-11-05T08:32:43.581 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00011055700000000001 2023-11-05T08:32:43.581 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.581 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:43.581 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.581 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.582 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.582 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:32:43.582 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.671060+0000", 2023-11-05T08:32:43.582 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 13.710972178, 2023-11-05T08:32:43.583 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022080292000000001, 2023-11-05T08:32:43.583 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:43.583 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:43.583 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:32:43.583 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:32:43.585 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:32:43.586 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 165 2023-11-05T08:32:43.586 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.586 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:43.586 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.586 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:43.587 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671060+0000", 2023-11-05T08:32:43.587 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.587 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.587 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.587 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:43.587 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671060+0000", 2023-11-05T08:32:43.588 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.588 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.588 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.588 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:43.588 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671060+0000", 2023-11-05T08:32:43.588 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.1699999999999997e-07 2023-11-05T08:32:43.589 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.589 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.589 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:43.589 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671067+0000", 2023-11-05T08:32:43.589 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5690000000000001e-06 2023-11-05T08:32:43.589 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.590 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.590 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:43.590 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671076+0000", 2023-11-05T08:32:43.590 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.5270000000000004e-06 2023-11-05T08:32:43.590 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.591 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.591 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:43.591 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671084+0000", 2023-11-05T08:32:43.591 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.3459999999999996e-06 2023-11-05T08:32:43.591 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.592 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.592 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:43.592 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.690832+0000", 2023-11-05T08:32:43.592 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.019747793 2023-11-05T08:32:43.592 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.592 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.593 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:32:43.593 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.690971+0000", 2023-11-05T08:32:43.593 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000139245 2023-11-05T08:32:43.593 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.593 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.593 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:32:43.594 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.691178+0000", 2023-11-05T08:32:43.594 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000206892 2023-11-05T08:32:43.594 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.594 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.595 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:32:43.595 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692704+0000", 2023-11-05T08:32:43.595 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00152655 2023-11-05T08:32:43.596 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.596 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.596 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:32:43.596 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692904+0000", 2023-11-05T08:32:43.597 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000199376 2023-11-05T08:32:43.597 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.597 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.597 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:32:43.597 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.693044+0000", 2023-11-05T08:32:43.598 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013989199999999999 2023-11-05T08:32:43.598 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.598 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.598 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:43.598 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.693140+0000", 2023-11-05T08:32:43.599 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6285000000000004e-05 2023-11-05T08:32:43.599 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.599 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:43.599 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.599 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.599 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.600 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:183 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:32:43.600 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.866106+0000", 2023-11-05T08:32:43.600 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 13.515925714, 2023-11-05T08:32:43.600 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.016027553, 2023-11-05T08:32:43.600 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:43.601 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:43.601 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:32:43.601 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:32:43.601 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:32:43.601 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 183 2023-11-05T08:32:43.601 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.602 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:43.602 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.602 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:43.602 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866106+0000", 2023-11-05T08:32:43.602 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.602 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.603 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.603 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:43.603 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866106+0000", 2023-11-05T08:32:43.603 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.603 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.603 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.604 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:43.604 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866107+0000", 2023-11-05T08:32:43.604 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.47e-07 2023-11-05T08:32:43.604 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.604 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.604 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:43.605 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866114+0000", 2023-11-05T08:32:43.605 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.0520000000000004e-06 2023-11-05T08:32:43.605 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.605 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.605 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:43.605 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866127+0000", 2023-11-05T08:32:43.606 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.331e-05 2023-11-05T08:32:43.606 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.606 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.606 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:43.606 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866136+0000", 2023-11-05T08:32:43.607 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.456e-06 2023-11-05T08:32:43.607 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.607 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.607 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:43.607 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.876274+0000", 2023-11-05T08:32:43.607 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010138632 2023-11-05T08:32:43.608 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.608 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.608 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:32:43.608 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.876533+0000", 2023-11-05T08:32:43.608 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00025816199999999999 2023-11-05T08:32:43.609 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.609 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.609 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:32:43.609 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.876953+0000", 2023-11-05T08:32:43.609 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00042021400000000003 2023-11-05T08:32:43.610 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.610 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.610 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:32:43.610 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.881834+0000", 2023-11-05T08:32:43.610 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0048814590000000003 2023-11-05T08:32:43.611 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.611 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.611 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:32:43.611 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.881947+0000", 2023-11-05T08:32:43.611 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000112911 2023-11-05T08:32:43.611 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.612 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.612 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:32:43.612 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.882030+0000", 2023-11-05T08:32:43.612 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.2855999999999994e-05 2023-11-05T08:32:43.612 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.613 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.613 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:43.613 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.882134+0000", 2023-11-05T08:32:43.613 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000103554 2023-11-05T08:32:43.613 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.614 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:43.614 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.614 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.614 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.614 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:184 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:32:43.615 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.866152+0000", 2023-11-05T08:32:43.615 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 13.515879903, 2023-11-05T08:32:43.615 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.019116587000000001, 2023-11-05T08:32:43.615 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:43.615 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:43.616 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:32:43.616 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:32:43.616 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:32:43.617 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 184 2023-11-05T08:32:43.617 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.618 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:43.618 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.618 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:43.618 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866152+0000", 2023-11-05T08:32:43.619 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.619 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.619 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.620 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:43.620 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866152+0000", 2023-11-05T08:32:43.620 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.621 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.621 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.621 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:43.622 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866152+0000", 2023-11-05T08:32:43.622 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4600000000000004e-07 2023-11-05T08:32:43.622 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.622 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.623 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:43.623 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866158+0000", 2023-11-05T08:32:43.623 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5629999999999999e-06 2023-11-05T08:32:43.624 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.624 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.624 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:43.624 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866167+0000", 2023-11-05T08:32:43.625 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.2229999999999996e-06 2023-11-05T08:32:43.625 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.625 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.625 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:43.625 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866175+0000", 2023-11-05T08:32:43.625 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.3699999999999997e-06 2023-11-05T08:32:43.626 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.626 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.626 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:43.626 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.878285+0000", 2023-11-05T08:32:43.626 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012110164 2023-11-05T08:32:43.627 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.627 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.627 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:32:43.627 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.878584+0000", 2023-11-05T08:32:43.627 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00029961300000000001 2023-11-05T08:32:43.628 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.628 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.628 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:32:43.629 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.878942+0000", 2023-11-05T08:32:43.629 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00035795100000000002 2023-11-05T08:32:43.629 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.629 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.630 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:32:43.630 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.883845+0000", 2023-11-05T08:32:43.630 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0049021450000000001 2023-11-05T08:32:43.630 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.630 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.631 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:32:43.631 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885018+0000", 2023-11-05T08:32:43.631 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0011738160000000001 2023-11-05T08:32:43.631 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.631 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.632 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:32:43.632 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885167+0000", 2023-11-05T08:32:43.632 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014895800000000001 2023-11-05T08:32:43.632 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.632 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.633 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:43.633 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885268+0000", 2023-11-05T08:32:43.633 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010113800000000001 2023-11-05T08:32:43.633 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.634 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:43.634 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.634 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.634 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.634 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:185 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:32:43.635 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.866212+0000", 2023-11-05T08:32:43.635 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 13.515820183000001, 2023-11-05T08:32:43.635 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.019448929, 2023-11-05T08:32:43.635 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:43.635 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:43.635 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:32:43.636 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:32:43.636 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:32:43.636 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 185 2023-11-05T08:32:43.636 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.636 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:43.636 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.637 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:43.637 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866212+0000", 2023-11-05T08:32:43.637 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.637 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.637 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.638 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:43.638 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866212+0000", 2023-11-05T08:32:43.638 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.638 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.638 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.639 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:43.639 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866213+0000", 2023-11-05T08:32:43.639 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0559999999999999e-06 2023-11-05T08:32:43.639 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.640 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.640 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:43.640 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866308+0000", 2023-11-05T08:32:43.640 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.4984999999999999e-05 2023-11-05T08:32:43.640 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.641 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.641 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:43.641 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866320+0000", 2023-11-05T08:32:43.641 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2639e-05 2023-11-05T08:32:43.641 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.642 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.642 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:43.642 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866328+0000", 2023-11-05T08:32:43.643 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.1240000000000005e-06 2023-11-05T08:32:43.643 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.644 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.644 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:43.644 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.879899+0000", 2023-11-05T08:32:43.644 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.013571059999999999 2023-11-05T08:32:43.645 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.645 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.645 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:32:43.645 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.880034+0000", 2023-11-05T08:32:43.646 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000134676 2023-11-05T08:32:43.646 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.646 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.647 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:32:43.647 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.880258+0000", 2023-11-05T08:32:43.647 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00022358100000000001 2023-11-05T08:32:43.647 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.647 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.648 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:32:43.648 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885299+0000", 2023-11-05T08:32:43.648 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.005041678 2023-11-05T08:32:43.648 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.649 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.649 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:32:43.649 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885436+0000", 2023-11-05T08:32:43.649 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000136472 2023-11-05T08:32:43.649 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.650 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.650 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:32:43.650 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885557+0000", 2023-11-05T08:32:43.650 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000121644 2023-11-05T08:32:43.650 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.651 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.651 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:43.651 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885660+0000", 2023-11-05T08:32:43.651 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000103014 2023-11-05T08:32:43.651 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.652 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:43.652 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.652 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.652 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.652 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:186 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:32:43.652 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.866343+0000", 2023-11-05T08:32:43.653 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 13.515688239999999, 2023-11-05T08:32:43.653 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.021691589000000001, 2023-11-05T08:32:43.653 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:43.653 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:43.653 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:32:43.653 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:32:43.654 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:32:43.654 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 186 2023-11-05T08:32:43.654 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.654 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:43.654 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.654 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:43.655 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866343+0000", 2023-11-05T08:32:43.655 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.655 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.655 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.655 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:43.656 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866343+0000", 2023-11-05T08:32:43.656 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.656 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.656 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.656 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:43.657 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866344+0000", 2023-11-05T08:32:43.657 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.6000000000000003e-07 2023-11-05T08:32:43.657 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.657 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.657 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:43.657 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866351+0000", 2023-11-05T08:32:43.658 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3090000000000001e-06 2023-11-05T08:32:43.658 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.658 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.658 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:43.658 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866360+0000", 2023-11-05T08:32:43.659 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.2599999999999994e-06 2023-11-05T08:32:43.659 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.659 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.659 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:43.659 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866367+0000", 2023-11-05T08:32:43.660 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.7599999999999997e-06 2023-11-05T08:32:43.660 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.660 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.660 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:43.660 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.882269+0000", 2023-11-05T08:32:43.661 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.015902218999999999 2023-11-05T08:32:43.661 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.661 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.661 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:32:43.661 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.882545+0000", 2023-11-05T08:32:43.662 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00027617199999999998 2023-11-05T08:32:43.662 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.662 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.662 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:32:43.662 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.882925+0000", 2023-11-05T08:32:43.663 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00037995900000000003 2023-11-05T08:32:43.663 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.663 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.663 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:32:43.663 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.887718+0000", 2023-11-05T08:32:43.663 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0047935069999999998 2023-11-05T08:32:43.664 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.664 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.664 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:32:43.664 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.887836+0000", 2023-11-05T08:32:43.664 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000117401 2023-11-05T08:32:43.665 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.665 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.665 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:32:43.665 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.887931+0000", 2023-11-05T08:32:43.665 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.4802000000000005e-05 2023-11-05T08:32:43.666 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.666 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.666 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:43.666 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888035+0000", 2023-11-05T08:32:43.667 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010444 2023-11-05T08:32:43.667 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.667 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:43.667 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.667 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.668 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.668 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:187 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:32:43.668 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.866377+0000", 2023-11-05T08:32:43.668 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 13.515654279, 2023-11-05T08:32:43.668 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022216751, 2023-11-05T08:32:43.668 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:43.669 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:43.669 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:32:43.669 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:32:43.669 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:32:43.669 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 187 2023-11-05T08:32:43.670 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.670 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:43.670 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.670 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:43.670 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866377+0000", 2023-11-05T08:32:43.670 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.671 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.671 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.671 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:43.671 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866377+0000", 2023-11-05T08:32:43.671 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.671 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.672 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.672 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:43.672 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866378+0000", 2023-11-05T08:32:43.672 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9999999999999997e-07 2023-11-05T08:32:43.672 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.672 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.673 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:43.673 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866384+0000", 2023-11-05T08:32:43.673 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.4099999999999999e-06 2023-11-05T08:32:43.673 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.673 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.674 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:43.674 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866393+0000", 2023-11-05T08:32:43.674 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9719999999999998e-06 2023-11-05T08:32:43.674 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.674 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.675 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:43.675 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866400+0000", 2023-11-05T08:32:43.675 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.2740000000000002e-06 2023-11-05T08:32:43.675 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.675 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.675 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:43.676 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.883936+0000", 2023-11-05T08:32:43.676 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017536513 2023-11-05T08:32:43.676 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.676 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.676 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:32:43.676 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.884061+0000", 2023-11-05T08:32:43.677 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012435300000000001 2023-11-05T08:32:43.677 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.677 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.677 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:32:43.677 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.884277+0000", 2023-11-05T08:32:43.678 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00021665500000000001 2023-11-05T08:32:43.678 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.678 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.678 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:32:43.678 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888193+0000", 2023-11-05T08:32:43.678 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0039160319999999998 2023-11-05T08:32:43.679 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.679 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.679 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:32:43.679 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888235+0000", 2023-11-05T08:32:43.679 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1267999999999999e-05 2023-11-05T08:32:43.679 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.680 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.680 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:32:43.680 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888373+0000", 2023-11-05T08:32:43.680 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000138789 2023-11-05T08:32:43.680 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.681 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.681 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:43.681 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888594+0000", 2023-11-05T08:32:43.681 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00022078500000000001 2023-11-05T08:32:43.681 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.681 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:43.682 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.682 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.682 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.682 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:188 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:32:43.682 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.866411+0000", 2023-11-05T08:32:43.682 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 13.515621018000001, 2023-11-05T08:32:43.683 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.023678187999999999, 2023-11-05T08:32:43.683 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:43.683 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:43.683 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:32:43.683 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:32:43.684 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:32:43.684 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 188 2023-11-05T08:32:43.684 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.684 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:43.684 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.684 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:43.685 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866411+0000", 2023-11-05T08:32:43.685 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.685 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.685 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.685 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:43.685 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866411+0000", 2023-11-05T08:32:43.686 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.686 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.686 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.686 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:43.686 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866411+0000", 2023-11-05T08:32:43.686 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.44e-07 2023-11-05T08:32:43.687 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.687 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.687 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:43.687 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866531+0000", 2023-11-05T08:32:43.687 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000119664 2023-11-05T08:32:43.687 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.688 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.688 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:43.688 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866545+0000", 2023-11-05T08:32:43.688 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3875e-05 2023-11-05T08:32:43.688 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.688 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.689 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:43.689 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866554+0000", 2023-11-05T08:32:43.689 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9099999999999994e-06 2023-11-05T08:32:43.689 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.689 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.690 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:43.690 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885784+0000", 2023-11-05T08:32:43.690 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.019230651000000001 2023-11-05T08:32:43.690 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.690 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.690 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:32:43.691 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.886020+0000", 2023-11-05T08:32:43.691 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00023538300000000001 2023-11-05T08:32:43.691 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.691 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.691 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:32:43.691 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.886386+0000", 2023-11-05T08:32:43.692 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00036659399999999998 2023-11-05T08:32:43.692 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.692 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.692 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:32:43.692 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888662+0000", 2023-11-05T08:32:43.693 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0022756019999999998 2023-11-05T08:32:43.693 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.693 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.693 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:32:43.693 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.889922+0000", 2023-11-05T08:32:43.693 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0012603149999999999 2023-11-05T08:32:43.694 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.694 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.694 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:32:43.694 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.890034+0000", 2023-11-05T08:32:43.694 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000111283 2023-11-05T08:32:43.695 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.695 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.695 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:43.695 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.890089+0000", 2023-11-05T08:32:43.695 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5266999999999998e-05 2023-11-05T08:32:43.695 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.696 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:43.696 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.696 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.696 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.696 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:189 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:32:43.696 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.866572+0000", 2023-11-05T08:32:43.697 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 13.515460060000001, 2023-11-05T08:32:43.697 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.024895937999999999, 2023-11-05T08:32:43.697 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:43.697 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:43.697 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:32:43.697 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:32:43.698 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:32:43.698 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 189 2023-11-05T08:32:43.698 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.698 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:43.698 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.699 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:43.699 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866572+0000", 2023-11-05T08:32:43.699 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.699 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.699 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.699 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:43.700 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866572+0000", 2023-11-05T08:32:43.700 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:43.700 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.700 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.700 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:43.700 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866572+0000", 2023-11-05T08:32:43.701 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.8000000000000005e-07 2023-11-05T08:32:43.701 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.701 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.701 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:43.701 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866581+0000", 2023-11-05T08:32:43.702 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9800000000000004e-06 2023-11-05T08:32:43.702 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.702 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.702 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:43.702 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866591+0000", 2023-11-05T08:32:43.703 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6059999999999992e-06 2023-11-05T08:32:43.703 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.703 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.703 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:43.703 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866598+0000", 2023-11-05T08:32:43.703 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.1029999999999996e-06 2023-11-05T08:32:43.704 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.704 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.704 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:43.704 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888850+0000", 2023-11-05T08:32:43.704 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022251615999999998 2023-11-05T08:32:43.705 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.705 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.705 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:32:43.705 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888988+0000", 2023-11-05T08:32:43.705 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000138436 2023-11-05T08:32:43.705 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.706 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.706 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:32:43.706 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.889206+0000", 2023-11-05T08:32:43.706 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00021818099999999999 2023-11-05T08:32:43.706 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.707 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.707 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:32:43.707 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.890621+0000", 2023-11-05T08:32:43.707 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001414219 2023-11-05T08:32:43.707 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.708 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.708 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:32:43.708 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.891316+0000", 2023-11-05T08:32:43.708 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00069570500000000002 2023-11-05T08:32:43.708 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.709 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.709 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:32:43.709 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.891406+0000", 2023-11-05T08:32:43.709 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9901999999999994e-05 2023-11-05T08:32:43.709 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:43.709 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:43.710 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:43.710 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.891468+0000", 2023-11-05T08:32:43.710 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1409999999999996e-05 2023-11-05T08:32:43.710 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.710 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:43.711 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.711 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:43.711 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:43.711 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:32:43.713 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-11-05T08:32:43.714 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:32:43.708+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:32:43.814 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-11-05T08:32:43.809+0000 7f966619d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35976) UID: 0 2023-11-05T08:32:43.844 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:32:43.844 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2023-11-05T08:32:43.844 INFO:teuthology.orchestra.run.smithi069.stdout: "num_ops": 0 2023-11-05T08:32:43.844 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:32:43.858 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-11-05T08:32:43.915 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:32:43.915+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:32:43.979 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:32:43.979 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2023-11-05T08:32:43.979 INFO:teuthology.orchestra.run.smithi069.stdout: "complaint_time": 30, 2023-11-05T08:32:43.980 INFO:teuthology.orchestra.run.smithi069.stdout: "num_blocked_ops": 0 2023-11-05T08:32:43.980 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:32:43.994 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-11-05T08:32:44.006 INFO:teuthology.orchestra.run.smithi002.stderr:nodeep-scrub is set 2023-11-05T08:32:44.006 INFO:teuthology.orchestra.run.smithi002.stderr:enabled application 'rados' on pool 'unique_pool_0' 2023-11-05T08:32:44.016 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:32:44.013+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:32:44.116 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:32:44.112+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:32:44.126 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:32:44.126 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 20, 2023-11-05T08:32:44.126 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 600, 2023-11-05T08:32:44.126 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [ 2023-11-05T08:32:44.127 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.127 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(client.4281.0:110 1.0 e23/22 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 23'72, mlcod=23'67)", 2023-11-05T08:32:44.127 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:29.161696+0000", 2023-11-05T08:32:44.127 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 14.962531553, 2023-11-05T08:32:44.127 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0042069070000000002, 2023-11-05T08:32:44.127 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:32:44.128 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:44.128 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:32:44.128 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.128 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:32:44.128 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.161696+0000", 2023-11-05T08:32:44.129 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.129 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.129 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.129 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:32:44.129 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.161696+0000", 2023-11-05T08:32:44.129 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.130 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.130 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.130 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:32:44.130 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.161698+0000", 2023-11-05T08:32:44.130 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0559999999999999e-06 2023-11-05T08:32:44.131 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.131 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.131 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:32:44.131 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.161708+0000", 2023-11-05T08:32:44.131 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0638999999999999e-05 2023-11-05T08:32:44.132 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.132 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.132 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:32:44.132 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.161730+0000", 2023-11-05T08:32:44.132 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1883999999999999e-05 2023-11-05T08:32:44.133 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.133 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.133 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:32:44.133 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.161742+0000", 2023-11-05T08:32:44.134 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2386e-05 2023-11-05T08:32:44.134 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.134 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.135 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:32:44.135 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.161822+0000", 2023-11-05T08:32:44.135 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.9937000000000004e-05 2023-11-05T08:32:44.135 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.135 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.135 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:32:44.136 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.161859+0000", 2023-11-05T08:32:44.136 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.6609999999999997e-05 2023-11-05T08:32:44.136 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.136 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.136 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:32:44.136 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.165815+0000", 2023-11-05T08:32:44.137 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0039564149999999996 2023-11-05T08:32:44.137 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.137 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.137 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:32:44.137 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.165903+0000", 2023-11-05T08:32:44.138 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.7979999999999995e-05 2023-11-05T08:32:44.138 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.138 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:44.138 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.138 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.139 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.139 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(client.4281.0:114 1.0 e23/22 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 23'76, mlcod=23'70)", 2023-11-05T08:32:44.139 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:29.167656+0000", 2023-11-05T08:32:44.139 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 14.956571609999999, 2023-11-05T08:32:44.139 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0049778720000000004, 2023-11-05T08:32:44.140 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:32:44.140 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:44.140 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:32:44.140 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.140 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:32:44.141 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.167656+0000", 2023-11-05T08:32:44.141 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.141 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.142 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.142 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:32:44.143 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.167656+0000", 2023-11-05T08:32:44.143 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.143 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.144 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.144 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:32:44.144 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.167658+0000", 2023-11-05T08:32:44.144 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.389e-06 2023-11-05T08:32:44.145 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.145 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.145 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:32:44.146 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.167669+0000", 2023-11-05T08:32:44.146 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0732e-05 2023-11-05T08:32:44.146 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.146 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.147 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:32:44.147 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.167692+0000", 2023-11-05T08:32:44.147 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2977999999999999e-05 2023-11-05T08:32:44.147 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.147 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.148 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:32:44.148 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.167707+0000", 2023-11-05T08:32:44.148 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5882000000000001e-05 2023-11-05T08:32:44.148 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.148 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.148 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:32:44.149 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.167815+0000", 2023-11-05T08:32:44.149 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00010768399999999999 2023-11-05T08:32:44.149 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.149 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.149 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:32:44.150 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.167850+0000", 2023-11-05T08:32:44.150 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.4663000000000003e-05 2023-11-05T08:32:44.150 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.150 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.150 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:32:44.151 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.172556+0000", 2023-11-05T08:32:44.151 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0047065960000000004 2023-11-05T08:32:44.151 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.151 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.151 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:32:44.151 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.172634+0000", 2023-11-05T08:32:44.152 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.7948000000000007e-05 2023-11-05T08:32:44.152 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.152 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:44.152 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.152 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.152 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.153 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(client.4281.0:115 1.0 e23/22 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 23'77, mlcod=23'73)", 2023-11-05T08:32:44.153 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:29.169871+0000", 2023-11-05T08:32:44.153 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 14.954356908999999, 2023-11-05T08:32:44.153 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.005638522, 2023-11-05T08:32:44.153 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:32:44.153 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:44.154 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:32:44.154 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.154 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:32:44.154 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.169871+0000", 2023-11-05T08:32:44.154 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.155 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.155 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.155 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:32:44.155 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.169871+0000", 2023-11-05T08:32:44.155 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.156 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.156 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.156 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:32:44.156 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.169873+0000", 2023-11-05T08:32:44.156 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.61e-06 2023-11-05T08:32:44.156 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.157 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.157 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:32:44.157 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.169934+0000", 2023-11-05T08:32:44.157 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.1292999999999997e-05 2023-11-05T08:32:44.157 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.157 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.158 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:32:44.158 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.169962+0000", 2023-11-05T08:32:44.158 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7716999999999999e-05 2023-11-05T08:32:44.158 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.158 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.158 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:32:44.159 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.169976+0000", 2023-11-05T08:32:44.159 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3815000000000001e-05 2023-11-05T08:32:44.159 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.159 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.159 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:32:44.160 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.170058+0000", 2023-11-05T08:32:44.160 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.2885000000000003e-05 2023-11-05T08:32:44.160 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.160 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.160 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:32:44.160 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.170089+0000", 2023-11-05T08:32:44.161 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0579e-05 2023-11-05T08:32:44.161 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.161 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.161 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:32:44.161 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.175416+0000", 2023-11-05T08:32:44.161 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0053266169999999996 2023-11-05T08:32:44.162 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.162 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.162 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:32:44.162 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.175510+0000", 2023-11-05T08:32:44.162 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.4005999999999994e-05 2023-11-05T08:32:44.162 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.163 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:44.163 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.163 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.163 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.163 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(client.4281.0:116 1.0 e23/22 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 23'78, mlcod=23'73)", 2023-11-05T08:32:44.163 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:29.171288+0000", 2023-11-05T08:32:44.164 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 14.952939622000001, 2023-11-05T08:32:44.164 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0042687630000000001, 2023-11-05T08:32:44.164 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:32:44.164 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:44.164 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:32:44.164 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.165 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:32:44.165 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.171288+0000", 2023-11-05T08:32:44.165 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.165 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.165 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.166 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:32:44.166 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.171288+0000", 2023-11-05T08:32:44.166 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.166 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.166 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.166 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:32:44.167 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.171290+0000", 2023-11-05T08:32:44.167 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1230000000000001e-06 2023-11-05T08:32:44.167 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.167 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.167 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:32:44.168 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.171298+0000", 2023-11-05T08:32:44.168 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.9460000000000004e-06 2023-11-05T08:32:44.168 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.168 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.168 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:32:44.168 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.171316+0000", 2023-11-05T08:32:44.169 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.7944e-05 2023-11-05T08:32:44.169 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.169 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.169 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:32:44.169 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.171328+0000", 2023-11-05T08:32:44.170 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1328999999999999e-05 2023-11-05T08:32:44.170 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.170 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.170 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:32:44.170 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.171442+0000", 2023-11-05T08:32:44.171 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000114184 2023-11-05T08:32:44.171 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.171 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.171 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:32:44.171 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.171509+0000", 2023-11-05T08:32:44.172 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.7231999999999996e-05 2023-11-05T08:32:44.172 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.172 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.172 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:32:44.172 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.175515+0000", 2023-11-05T08:32:44.172 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0040060399999999998 2023-11-05T08:32:44.173 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.173 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.173 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:32:44.173 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.175557+0000", 2023-11-05T08:32:44.173 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.1965000000000002e-05 2023-11-05T08:32:44.174 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.174 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:44.174 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.174 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.174 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.175 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(client.4281.0:131 1.0 e23/22 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 23'89, mlcod=23'85)", 2023-11-05T08:32:44.175 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:29.355619+0000", 2023-11-05T08:32:44.175 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 14.768608711000001, 2023-11-05T08:32:44.175 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0056090239999999998, 2023-11-05T08:32:44.175 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:32:44.176 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:44.176 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:32:44.176 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.176 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:32:44.176 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.355619+0000", 2023-11-05T08:32:44.176 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.177 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.177 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.177 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:32:44.177 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.355619+0000", 2023-11-05T08:32:44.177 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.178 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.178 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.178 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:32:44.178 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.355621+0000", 2023-11-05T08:32:44.178 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6670000000000001e-06 2023-11-05T08:32:44.179 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.179 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.179 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:32:44.179 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.355680+0000", 2023-11-05T08:32:44.179 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.9426999999999999e-05 2023-11-05T08:32:44.180 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.180 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.180 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:32:44.180 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.355704+0000", 2023-11-05T08:32:44.180 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3841000000000001e-05 2023-11-05T08:32:44.181 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.181 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.181 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:32:44.181 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.355718+0000", 2023-11-05T08:32:44.181 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3858e-05 2023-11-05T08:32:44.181 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.182 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.182 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:32:44.182 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.355840+0000", 2023-11-05T08:32:44.182 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00012216000000000001 2023-11-05T08:32:44.182 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.183 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.183 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:32:44.183 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.355872+0000", 2023-11-05T08:32:44.183 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.1346000000000001e-05 2023-11-05T08:32:44.183 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.184 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.184 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:32:44.184 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.361119+0000", 2023-11-05T08:32:44.184 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0052472229999999996 2023-11-05T08:32:44.184 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.185 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.185 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:32:44.185 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.361228+0000", 2023-11-05T08:32:44.185 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000109502 2023-11-05T08:32:44.185 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.185 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:44.186 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.186 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.186 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.186 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(client.4281.0:132 1.0 e23/22 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 23'90, mlcod=23'85)", 2023-11-05T08:32:44.186 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:29.356513+0000", 2023-11-05T08:32:44.187 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 14.767715368999999, 2023-11-05T08:32:44.187 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00475757, 2023-11-05T08:32:44.187 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:32:44.187 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:44.188 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:32:44.188 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.188 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:32:44.188 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.356513+0000", 2023-11-05T08:32:44.188 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.189 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.189 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.189 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:32:44.189 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.356513+0000", 2023-11-05T08:32:44.189 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.189 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.190 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.190 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:32:44.190 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.356514+0000", 2023-11-05T08:32:44.190 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.663e-06 2023-11-05T08:32:44.190 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.191 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.191 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:32:44.191 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.356530+0000", 2023-11-05T08:32:44.191 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5424999999999999e-05 2023-11-05T08:32:44.191 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.192 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.192 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:32:44.192 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.356558+0000", 2023-11-05T08:32:44.192 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8070000000000001e-05 2023-11-05T08:32:44.192 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.192 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.193 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:32:44.193 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.356571+0000", 2023-11-05T08:32:44.193 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3122000000000001e-05 2023-11-05T08:32:44.193 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.193 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.193 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:32:44.194 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.356700+0000", 2023-11-05T08:32:44.194 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00012898900000000001 2023-11-05T08:32:44.194 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.195 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.195 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:32:44.195 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.356733+0000", 2023-11-05T08:32:44.198 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.3250000000000002e-05 2023-11-05T08:32:44.198 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.199 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.199 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:32:44.200 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.361232+0000", 2023-11-05T08:32:44.200 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0044988950000000002 2023-11-05T08:32:44.200 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.201 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.201 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:32:44.201 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.361270+0000", 2023-11-05T08:32:44.202 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.8155999999999999e-05 2023-11-05T08:32:44.202 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.202 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:44.203 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.203 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.203 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.204 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(client.4281.0:133 1.0 e23/22 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 23'91, mlcod=23'85)", 2023-11-05T08:32:44.204 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:29.357346+0000", 2023-11-05T08:32:44.204 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 14.766881955000001, 2023-11-05T08:32:44.205 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0039582619999999997, 2023-11-05T08:32:44.205 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:32:44.205 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:44.205 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:32:44.205 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.206 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:32:44.206 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.357346+0000", 2023-11-05T08:32:44.206 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.206 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.206 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.207 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:32:44.207 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.357346+0000", 2023-11-05T08:32:44.207 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.207 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.208 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.208 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:32:44.208 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.357348+0000", 2023-11-05T08:32:44.208 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.657e-06 2023-11-05T08:32:44.208 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.209 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.209 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:32:44.209 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.357360+0000", 2023-11-05T08:32:44.209 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2305e-05 2023-11-05T08:32:44.209 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.209 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.210 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:32:44.210 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.357388+0000", 2023-11-05T08:32:44.210 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7804e-05 2023-11-05T08:32:44.210 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.210 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.211 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:32:44.211 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.357404+0000", 2023-11-05T08:32:44.211 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6268000000000001e-05 2023-11-05T08:32:44.211 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.211 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.212 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:32:44.212 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.357576+0000", 2023-11-05T08:32:44.212 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000171471 2023-11-05T08:32:44.212 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.212 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.213 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:32:44.213 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.357609+0000", 2023-11-05T08:32:44.213 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.3492999999999999e-05 2023-11-05T08:32:44.213 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.213 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.214 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:32:44.214 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.361271+0000", 2023-11-05T08:32:44.214 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0036623850000000002 2023-11-05T08:32:44.214 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.214 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.215 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:32:44.215 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.361304+0000", 2023-11-05T08:32:44.215 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2879000000000001e-05 2023-11-05T08:32:44.215 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.215 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:44.216 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.216 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.216 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.216 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(client.4281.0:134 1.0 e23/22 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 23'92, mlcod=23'86)", 2023-11-05T08:32:44.216 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:29.358423+0000", 2023-11-05T08:32:44.217 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 14.765804580999999, 2023-11-05T08:32:44.217 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0048857099999999997, 2023-11-05T08:32:44.217 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:32:44.217 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:44.217 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:32:44.218 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.218 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:32:44.218 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.358423+0000", 2023-11-05T08:32:44.218 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.218 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.218 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.219 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:32:44.219 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.358423+0000", 2023-11-05T08:32:44.219 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.219 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.219 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.220 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:32:44.220 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.358425+0000", 2023-11-05T08:32:44.220 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.544e-06 2023-11-05T08:32:44.220 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.220 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.220 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:32:44.221 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.358501+0000", 2023-11-05T08:32:44.221 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.5834000000000006e-05 2023-11-05T08:32:44.221 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.221 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.221 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:32:44.222 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.358533+0000", 2023-11-05T08:32:44.222 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.1699999999999998e-05 2023-11-05T08:32:44.222 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.222 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.223 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:32:44.223 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.358547+0000", 2023-11-05T08:32:44.223 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4239e-05 2023-11-05T08:32:44.223 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.224 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.224 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:32:44.224 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.358660+0000", 2023-11-05T08:32:44.224 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000113261 2023-11-05T08:32:44.224 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.225 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.225 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:32:44.225 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.358690+0000", 2023-11-05T08:32:44.225 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0035999999999998e-05 2023-11-05T08:32:44.225 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.226 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.226 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:32:44.226 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.363240+0000", 2023-11-05T08:32:44.226 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0045500130000000003 2023-11-05T08:32:44.226 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.227 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.227 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:32:44.227 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.363309+0000", 2023-11-05T08:32:44.227 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.9083000000000002e-05 2023-11-05T08:32:44.227 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.228 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:44.228 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.228 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.228 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.228 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(client.4281.0:135 1.0 e23/22 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 23'93, mlcod=23'86)", 2023-11-05T08:32:44.228 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:29.359330+0000", 2023-11-05T08:32:44.229 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 14.764898074, 2023-11-05T08:32:44.229 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0040202520000000002, 2023-11-05T08:32:44.229 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:32:44.229 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:44.229 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:32:44.229 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.230 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:32:44.230 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.359330+0000", 2023-11-05T08:32:44.230 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.230 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.230 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.231 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:32:44.231 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.359330+0000", 2023-11-05T08:32:44.231 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.231 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.231 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.231 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:32:44.232 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.359332+0000", 2023-11-05T08:32:44.232 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.643e-06 2023-11-05T08:32:44.232 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.232 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.232 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:32:44.233 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.359344+0000", 2023-11-05T08:32:44.233 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1962e-05 2023-11-05T08:32:44.233 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.233 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.234 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:32:44.234 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.359372+0000", 2023-11-05T08:32:44.234 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8240000000000001e-05 2023-11-05T08:32:44.235 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.235 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.235 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:32:44.236 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.359385+0000", 2023-11-05T08:32:44.236 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3219e-05 2023-11-05T08:32:44.236 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.236 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.236 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:32:44.236 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.359521+0000", 2023-11-05T08:32:44.237 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000135685 2023-11-05T08:32:44.237 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.237 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.237 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:32:44.238 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.359554+0000", 2023-11-05T08:32:44.238 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.3485999999999998e-05 2023-11-05T08:32:44.238 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.238 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.238 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:32:44.239 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.363314+0000", 2023-11-05T08:32:44.239 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0037597339999999998 2023-11-05T08:32:44.239 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.239 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.239 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:32:44.240 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.363350+0000", 2023-11-05T08:32:44.240 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.6282999999999998e-05 2023-11-05T08:32:44.240 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.240 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:44.240 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.240 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.241 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.241 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(client.4281.0:136 1.0 e23/22 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 23'94, mlcod=23'87)", 2023-11-05T08:32:44.241 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:29.360460+0000", 2023-11-05T08:32:44.241 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 14.763768019, 2023-11-05T08:32:44.241 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0056144380000000002, 2023-11-05T08:32:44.241 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:32:44.242 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:44.242 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:32:44.242 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.242 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:32:44.242 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.360460+0000", 2023-11-05T08:32:44.242 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.243 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.243 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.243 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:32:44.243 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.360460+0000", 2023-11-05T08:32:44.243 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.243 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.244 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.244 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:32:44.244 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.360462+0000", 2023-11-05T08:32:44.244 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6500000000000001e-06 2023-11-05T08:32:44.244 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.245 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.245 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:32:44.245 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.360474+0000", 2023-11-05T08:32:44.245 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2208000000000001e-05 2023-11-05T08:32:44.245 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.245 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.246 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:32:44.246 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.360498+0000", 2023-11-05T08:32:44.246 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4176999999999999e-05 2023-11-05T08:32:44.246 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.246 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.247 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:32:44.247 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.360511+0000", 2023-11-05T08:32:44.248 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3236e-05 2023-11-05T08:32:44.248 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.248 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.248 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:32:44.248 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.360646+0000", 2023-11-05T08:32:44.249 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00013491100000000001 2023-11-05T08:32:44.249 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.249 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.250 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:32:44.250 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.360679+0000", 2023-11-05T08:32:44.250 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.3033e-05 2023-11-05T08:32:44.251 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.251 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.251 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:32:44.252 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.366010+0000", 2023-11-05T08:32:44.252 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0053304629999999997 2023-11-05T08:32:44.252 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.252 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.253 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:32:44.253 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.366074+0000", 2023-11-05T08:32:44.253 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.4759999999999997e-05 2023-11-05T08:32:44.253 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.254 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:44.254 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.254 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.255 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.255 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(client.4281.0:137 1.0 e23/22 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 23'95, mlcod=23'87)", 2023-11-05T08:32:44.255 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:29.361240+0000", 2023-11-05T08:32:44.255 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 14.762987652, 2023-11-05T08:32:44.255 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.004876526, 2023-11-05T08:32:44.255 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:32:44.256 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:44.256 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:32:44.256 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.257 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:32:44.257 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.361240+0000", 2023-11-05T08:32:44.257 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.257 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.257 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.257 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:32:44.258 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.361240+0000", 2023-11-05T08:32:44.258 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.258 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.258 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.258 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:32:44.259 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.361242+0000", 2023-11-05T08:32:44.259 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6759999999999999e-06 2023-11-05T08:32:44.259 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.259 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.259 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:32:44.259 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.361296+0000", 2023-11-05T08:32:44.260 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.3776999999999997e-05 2023-11-05T08:32:44.260 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.260 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.260 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:32:44.260 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.361323+0000", 2023-11-05T08:32:44.260 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6906999999999998e-05 2023-11-05T08:32:44.261 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.261 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.261 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:32:44.261 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.361336+0000", 2023-11-05T08:32:44.261 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3536000000000001e-05 2023-11-05T08:32:44.261 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.262 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.262 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:32:44.262 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.361466+0000", 2023-11-05T08:32:44.262 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000129665 2023-11-05T08:32:44.262 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.263 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.263 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:32:44.263 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.361498+0000", 2023-11-05T08:32:44.263 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2277000000000003e-05 2023-11-05T08:32:44.263 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.263 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.264 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:32:44.264 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.366078+0000", 2023-11-05T08:32:44.264 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0045793029999999998 2023-11-05T08:32:44.264 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.264 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.264 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:32:44.265 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.366117+0000", 2023-11-05T08:32:44.265 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.9385000000000003e-05 2023-11-05T08:32:44.265 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.265 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:44.265 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.266 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.266 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.266 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(client.4281.0:138 1.0 e23/22 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 23'96, mlcod=23'87)", 2023-11-05T08:32:44.266 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:29.362320+0000", 2023-11-05T08:32:44.266 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 14.761908441999999, 2023-11-05T08:32:44.267 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0038375219999999999, 2023-11-05T08:32:44.267 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:32:44.267 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:44.267 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:32:44.267 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.268 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:32:44.268 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.362320+0000", 2023-11-05T08:32:44.268 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.268 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.268 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.269 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:32:44.269 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.362320+0000", 2023-11-05T08:32:44.269 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.269 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.269 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.269 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:32:44.270 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.362321+0000", 2023-11-05T08:32:44.270 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.657e-06 2023-11-05T08:32:44.270 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.270 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.271 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:32:44.271 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.362336+0000", 2023-11-05T08:32:44.271 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4955e-05 2023-11-05T08:32:44.271 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.271 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.271 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:32:44.272 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.362364+0000", 2023-11-05T08:32:44.272 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8160000000000001e-05 2023-11-05T08:32:44.272 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.272 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.272 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:32:44.272 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.362378+0000", 2023-11-05T08:32:44.273 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3229e-05 2023-11-05T08:32:44.273 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.273 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.273 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:32:44.273 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.362496+0000", 2023-11-05T08:32:44.273 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000118373 2023-11-05T08:32:44.274 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.274 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.274 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:32:44.274 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.362534+0000", 2023-11-05T08:32:44.274 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.7568999999999999e-05 2023-11-05T08:32:44.275 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.275 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.275 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:32:44.275 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.366118+0000", 2023-11-05T08:32:44.275 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0035847700000000001 2023-11-05T08:32:44.275 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.276 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.276 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:32:44.276 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.366157+0000", 2023-11-05T08:32:44.276 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.8809000000000001e-05 2023-11-05T08:32:44.276 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.277 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:44.277 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.277 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.277 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.277 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(client.4281.0:139 1.0 e23/22 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 23'97, mlcod=23'90)", 2023-11-05T08:32:44.277 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:29.363764+0000", 2023-11-05T08:32:44.278 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 14.760463562, 2023-11-05T08:32:44.278 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0046955740000000001, 2023-11-05T08:32:44.278 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:32:44.278 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:44.278 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:32:44.279 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.279 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:32:44.279 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.363764+0000", 2023-11-05T08:32:44.279 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.279 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.279 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.280 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:32:44.280 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.363764+0000", 2023-11-05T08:32:44.280 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.280 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.280 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.280 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:32:44.281 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.363766+0000", 2023-11-05T08:32:44.281 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.314e-06 2023-11-05T08:32:44.281 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.281 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.281 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:32:44.282 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.363775+0000", 2023-11-05T08:32:44.282 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.0890000000000006e-06 2023-11-05T08:32:44.282 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.282 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.282 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:32:44.283 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.363798+0000", 2023-11-05T08:32:44.283 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3609999999999999e-05 2023-11-05T08:32:44.283 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.283 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.283 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:32:44.283 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.363812+0000", 2023-11-05T08:32:44.284 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3122000000000001e-05 2023-11-05T08:32:44.284 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.284 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.284 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:32:44.284 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.363930+0000", 2023-11-05T08:32:44.285 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000118097 2023-11-05T08:32:44.285 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.285 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.285 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:32:44.285 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.363960+0000", 2023-11-05T08:32:44.286 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0366999999999999e-05 2023-11-05T08:32:44.286 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.286 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.286 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:32:44.286 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.368374+0000", 2023-11-05T08:32:44.287 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0044138199999999997 2023-11-05T08:32:44.287 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.287 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.287 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:32:44.287 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.368460+0000", 2023-11-05T08:32:44.288 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.6155000000000004e-05 2023-11-05T08:32:44.288 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.288 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:44.288 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.288 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.288 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.289 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(client.4281.0:140 1.0 e23/22 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 23'98, mlcod=23'90)", 2023-11-05T08:32:44.289 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:29.364836+0000", 2023-11-05T08:32:44.289 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 14.759392033999999, 2023-11-05T08:32:44.289 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0063794710000000003, 2023-11-05T08:32:44.289 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:32:44.290 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:44.290 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:32:44.290 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.290 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:32:44.290 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.364836+0000", 2023-11-05T08:32:44.291 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.291 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.291 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.292 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:32:44.292 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.364836+0000", 2023-11-05T08:32:44.292 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.292 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.293 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.293 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:32:44.293 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.364838+0000", 2023-11-05T08:32:44.293 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.663e-06 2023-11-05T08:32:44.293 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.294 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.294 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:32:44.295 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.364935+0000", 2023-11-05T08:32:44.295 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.7565999999999994e-05 2023-11-05T08:32:44.295 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.296 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.296 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:32:44.297 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.364964+0000", 2023-11-05T08:32:44.297 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8626999999999999e-05 2023-11-05T08:32:44.297 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.297 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.298 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:32:44.298 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.364977+0000", 2023-11-05T08:32:44.298 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3412e-05 2023-11-05T08:32:44.299 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.299 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.299 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:32:44.299 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.365131+0000", 2023-11-05T08:32:44.299 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00015339300000000001 2023-11-05T08:32:44.300 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.300 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.300 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:32:44.300 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.365170+0000", 2023-11-05T08:32:44.300 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.9632000000000003e-05 2023-11-05T08:32:44.301 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.301 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.301 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:32:44.301 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.371117+0000", 2023-11-05T08:32:44.301 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0059463249999999997 2023-11-05T08:32:44.301 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.302 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.302 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:32:44.302 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.371215+0000", 2023-11-05T08:32:44.302 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.8852999999999997e-05 2023-11-05T08:32:44.302 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.303 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:44.303 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.303 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.303 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.303 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(client.4281.0:141 1.0 e23/22 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 23'99, mlcod=23'92)", 2023-11-05T08:32:44.304 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:29.366385+0000", 2023-11-05T08:32:44.304 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 14.757843422000001, 2023-11-05T08:32:44.304 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0048805560000000003, 2023-11-05T08:32:44.304 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:32:44.304 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:44.305 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:32:44.305 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.305 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:32:44.305 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.366385+0000", 2023-11-05T08:32:44.305 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.306 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.306 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.306 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:32:44.306 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.366385+0000", 2023-11-05T08:32:44.306 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.307 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.307 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.307 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:32:44.307 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.366386+0000", 2023-11-05T08:32:44.307 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.226e-06 2023-11-05T08:32:44.308 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.308 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.308 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:32:44.308 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.366397+0000", 2023-11-05T08:32:44.308 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0749e-05 2023-11-05T08:32:44.309 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.309 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.309 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:32:44.309 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.366422+0000", 2023-11-05T08:32:44.309 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5581000000000001e-05 2023-11-05T08:32:44.310 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.310 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.310 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:32:44.310 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.366436+0000", 2023-11-05T08:32:44.310 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3502e-05 2023-11-05T08:32:44.311 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.311 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.311 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:32:44.311 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.366567+0000", 2023-11-05T08:32:44.311 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000131168 2023-11-05T08:32:44.312 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.312 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.312 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:32:44.312 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.366603+0000", 2023-11-05T08:32:44.312 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.6340000000000001e-05 2023-11-05T08:32:44.313 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.313 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.313 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:32:44.313 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.371219+0000", 2023-11-05T08:32:44.313 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0046159920000000002 2023-11-05T08:32:44.314 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.314 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.314 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:32:44.314 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.371265+0000", 2023-11-05T08:32:44.314 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.5998000000000002e-05 2023-11-05T08:32:44.315 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.315 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:44.315 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.315 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.315 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.316 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(client.4281.0:144 1.0 e23/22 1:30306672:devicehealth::main.db.0000000000000000:head v 23'102, mlcod=23'98)", 2023-11-05T08:32:44.316 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:29.374946+0000", 2023-11-05T08:32:44.316 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 14.749282317, 2023-11-05T08:32:44.316 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.005255582, 2023-11-05T08:32:44.316 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:32:44.317 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:44.317 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:32:44.317 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.317 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:32:44.317 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.374946+0000", 2023-11-05T08:32:44.317 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.318 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.318 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.318 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:32:44.318 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.374946+0000", 2023-11-05T08:32:44.318 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.319 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.319 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.319 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:32:44.319 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.374947+0000", 2023-11-05T08:32:44.319 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.7400000000000001e-06 2023-11-05T08:32:44.320 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.320 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.320 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:32:44.321 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.375016+0000", 2023-11-05T08:32:44.321 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.8208999999999997e-05 2023-11-05T08:32:44.321 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.321 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.321 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:32:44.322 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.375051+0000", 2023-11-05T08:32:44.322 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.5785999999999999e-05 2023-11-05T08:32:44.322 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.322 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.323 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:32:44.323 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.375070+0000", 2023-11-05T08:32:44.323 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8745000000000001e-05 2023-11-05T08:32:44.324 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.324 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.325 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:32:44.325 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.375195+0000", 2023-11-05T08:32:44.325 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000124599 2023-11-05T08:32:44.326 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.326 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.326 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:32:44.326 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.375229+0000", 2023-11-05T08:32:44.327 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.4140000000000002e-05 2023-11-05T08:32:44.327 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.327 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.328 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:32:44.328 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.380118+0000", 2023-11-05T08:32:44.328 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0048891890000000004 2023-11-05T08:32:44.328 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.328 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.329 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:32:44.329 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.380201+0000", 2023-11-05T08:32:44.329 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.3174e-05 2023-11-05T08:32:44.329 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.329 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:44.330 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.330 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.330 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.330 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(client.4281.0:145 1.0 e23/22 1:30306672:devicehealth::main.db.0000000000000000:head v 23'103, mlcod=23'98)", 2023-11-05T08:32:44.330 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:29.376016+0000", 2023-11-05T08:32:44.331 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 14.748212365000001, 2023-11-05T08:32:44.331 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0042304250000000003, 2023-11-05T08:32:44.331 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:32:44.331 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:44.331 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:32:44.332 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.332 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:32:44.332 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.376016+0000", 2023-11-05T08:32:44.332 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.332 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.333 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.333 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:32:44.333 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.376016+0000", 2023-11-05T08:32:44.333 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.333 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.334 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.334 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:32:44.334 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.376017+0000", 2023-11-05T08:32:44.334 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.703e-06 2023-11-05T08:32:44.334 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.335 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.335 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:32:44.335 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.376075+0000", 2023-11-05T08:32:44.335 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.7902999999999998e-05 2023-11-05T08:32:44.335 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.336 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.336 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:32:44.336 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.376104+0000", 2023-11-05T08:32:44.336 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.9003999999999999e-05 2023-11-05T08:32:44.336 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.337 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.337 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:32:44.338 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.376117+0000", 2023-11-05T08:32:44.338 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3165e-05 2023-11-05T08:32:44.338 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.338 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.339 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:32:44.339 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.376232+0000", 2023-11-05T08:32:44.339 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00011467399999999999 2023-11-05T08:32:44.339 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.339 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.339 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:32:44.340 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.376264+0000", 2023-11-05T08:32:44.340 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2209999999999998e-05 2023-11-05T08:32:44.340 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.340 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.340 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:32:44.341 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.380205+0000", 2023-11-05T08:32:44.341 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0039408230000000004 2023-11-05T08:32:44.341 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.341 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.341 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:32:44.341 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.380246+0000", 2023-11-05T08:32:44.342 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.0942999999999997e-05 2023-11-05T08:32:44.342 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.342 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:44.342 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.342 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.343 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.343 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(client.4281.0:147 1.0 e23/22 1:30306672:devicehealth::main.db.0000000000000000:head v 23'105, mlcod=23'103)", 2023-11-05T08:32:44.343 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:29.384549+0000", 2023-11-05T08:32:44.343 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 14.73967865, 2023-11-05T08:32:44.343 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0042092070000000004, 2023-11-05T08:32:44.343 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:32:44.344 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:44.344 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:32:44.344 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.344 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:32:44.344 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.384549+0000", 2023-11-05T08:32:44.345 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.345 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.345 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.345 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:32:44.345 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.384549+0000", 2023-11-05T08:32:44.345 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.346 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.346 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.346 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:32:44.346 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.384551+0000", 2023-11-05T08:32:44.346 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6500000000000001e-06 2023-11-05T08:32:44.347 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.347 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.347 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:32:44.347 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.384564+0000", 2023-11-05T08:32:44.347 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2704999999999999e-05 2023-11-05T08:32:44.348 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.348 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.348 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:32:44.348 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.384592+0000", 2023-11-05T08:32:44.348 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7780000000000002e-05 2023-11-05T08:32:44.349 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.349 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.349 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:32:44.349 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.384604+0000", 2023-11-05T08:32:44.349 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2989e-05 2023-11-05T08:32:44.349 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.350 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.350 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:32:44.350 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.384733+0000", 2023-11-05T08:32:44.350 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000128832 2023-11-05T08:32:44.350 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.351 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.351 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:32:44.351 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.384796+0000", 2023-11-05T08:32:44.352 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.2323000000000006e-05 2023-11-05T08:32:44.352 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.353 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.353 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:32:44.353 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.388679+0000", 2023-11-05T08:32:44.354 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.003883727 2023-11-05T08:32:44.354 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.354 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.354 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:32:44.355 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.388759+0000", 2023-11-05T08:32:44.355 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.9201000000000003e-05 2023-11-05T08:32:44.355 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.356 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:44.356 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.356 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.356 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.357 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(client.4281.0:158 1.0 e23/22 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 23'112, mlcod=23'107)", 2023-11-05T08:32:44.357 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:29.679773+0000", 2023-11-05T08:32:44.357 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 14.444454679, 2023-11-05T08:32:44.357 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0046210729999999998, 2023-11-05T08:32:44.358 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:32:44.358 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:44.358 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:32:44.358 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.359 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:32:44.359 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.679773+0000", 2023-11-05T08:32:44.359 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.359 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.359 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.360 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:32:44.360 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.679773+0000", 2023-11-05T08:32:44.360 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.360 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.360 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.360 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:32:44.361 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.679774+0000", 2023-11-05T08:32:44.361 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.16e-07 2023-11-05T08:32:44.361 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.361 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.361 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:32:44.362 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.679833+0000", 2023-11-05T08:32:44.362 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.8513e-05 2023-11-05T08:32:44.362 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.362 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.362 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:32:44.363 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.679846+0000", 2023-11-05T08:32:44.363 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3067e-05 2023-11-05T08:32:44.363 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.363 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.363 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:32:44.364 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.679852+0000", 2023-11-05T08:32:44.364 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.5019999999999998e-06 2023-11-05T08:32:44.364 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.364 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.364 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:32:44.364 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.679913+0000", 2023-11-05T08:32:44.365 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.1208000000000001e-05 2023-11-05T08:32:44.365 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.365 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.365 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:32:44.365 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.679943+0000", 2023-11-05T08:32:44.366 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.9348000000000001e-05 2023-11-05T08:32:44.366 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.366 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.366 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:32:44.366 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.684260+0000", 2023-11-05T08:32:44.367 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0043174240000000003 2023-11-05T08:32:44.367 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.367 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.367 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:32:44.367 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.684394+0000", 2023-11-05T08:32:44.368 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00013419500000000001 2023-11-05T08:32:44.368 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.368 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:44.368 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.368 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.369 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.369 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(client.4281.0:159 1.0 e23/22 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 23'113, mlcod=23'109)", 2023-11-05T08:32:44.369 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:29.681850+0000", 2023-11-05T08:32:44.369 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 14.442377866999999, 2023-11-05T08:32:44.369 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0038382680000000001, 2023-11-05T08:32:44.370 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:32:44.370 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:44.370 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:32:44.371 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.371 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:32:44.371 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.681850+0000", 2023-11-05T08:32:44.371 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.372 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.372 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.372 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:32:44.373 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.681850+0000", 2023-11-05T08:32:44.373 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:44.373 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.373 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.373 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:32:44.374 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.681851+0000", 2023-11-05T08:32:44.374 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.6700000000000003e-07 2023-11-05T08:32:44.374 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.374 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.374 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:32:44.375 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.681859+0000", 2023-11-05T08:32:44.375 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.1470000000000001e-06 2023-11-05T08:32:44.375 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.375 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.375 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:32:44.375 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.681872+0000", 2023-11-05T08:32:44.376 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2668e-05 2023-11-05T08:32:44.376 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.376 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.376 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:32:44.376 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.681878+0000", 2023-11-05T08:32:44.377 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.9499999999999998e-06 2023-11-05T08:32:44.377 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.377 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.377 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:32:44.377 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.681966+0000", 2023-11-05T08:32:44.378 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.8487999999999997e-05 2023-11-05T08:32:44.378 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.378 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.378 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:32:44.378 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.681992+0000", 2023-11-05T08:32:44.379 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6251999999999999e-05 2023-11-05T08:32:44.379 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.379 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.379 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:32:44.379 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.685651+0000", 2023-11-05T08:32:44.380 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0036590479999999998 2023-11-05T08:32:44.380 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:44.380 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:44.380 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:32:44.380 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.685688+0000", 2023-11-05T08:32:44.380 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.6947999999999999e-05 2023-11-05T08:32:44.381 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.381 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:44.381 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.381 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:44.381 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:44.382 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:32:44.383 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-11-05T08:32:44.384 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:32:44.381+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:32:44.484 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:32:44.480+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:32:44.561 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:32:44.561 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2023-11-05T08:32:44.561 INFO:teuthology.orchestra.run.smithi069.stdout: "num_ops": 0 2023-11-05T08:32:44.562 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:32:44.575 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-11-05T08:32:44.585 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:32:44.581+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:32:44.686 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:32:44.685+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:32:44.742 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:32:44.742 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2023-11-05T08:32:44.742 INFO:teuthology.orchestra.run.smithi069.stdout: "complaint_time": 30, 2023-11-05T08:32:44.743 INFO:teuthology.orchestra.run.smithi069.stdout: "num_blocked_ops": 0 2023-11-05T08:32:44.743 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:32:44.756 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-11-05T08:32:44.787 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:32:44.785+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:32:44.878 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:32:44.879 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 20, 2023-11-05T08:32:44.879 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 600, 2023-11-05T08:32:44.879 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [] 2023-11-05T08:32:44.879 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:32:44.888 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:32:44.884+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:32:44.893 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2023-11-05T08:32:44.970 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2023-11-05T08:32:44.970 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-11-05T08:32:44.970 INFO:tasks.thrashosds.thrasher:primary_affinity 2023-11-05T08:32:44.971 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 0.046311 2023-11-05T08:32:44.971 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 0.04631098273860701 2023-11-05T08:32:44.988 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:32:44.985+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:32:44.998 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-05T08:32:45.042 DEBUG:teuthology.orchestra.run.smithi090:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0 2023-11-05T08:32:45.055 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:32:45.055 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2023-11-05T08:32:45.056 INFO:teuthology.orchestra.run.smithi069.stdout: "num_ops": 0 2023-11-05T08:32:45.056 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:32:45.063 INFO:tasks.rados.rados.0.smithi090.stdout:adding op weight read -> 100 2023-11-05T08:32:45.063 INFO:tasks.rados.rados.0.smithi090.stdout:adding op weight delete -> 50 2023-11-05T08:32:45.063 INFO:tasks.rados.rados.0.smithi090.stdout:adding op weight snap_create -> 50 2023-11-05T08:32:45.064 INFO:tasks.rados.rados.0.smithi090.stdout:adding op weight snap_remove -> 50 2023-11-05T08:32:45.064 INFO:tasks.rados.rados.0.smithi090.stdout:adding op weight rollback -> 50 2023-11-05T08:32:45.064 INFO:tasks.rados.rados.0.smithi090.stdout:adding op weight setattr -> 25 2023-11-05T08:32:45.064 INFO:tasks.rados.rados.0.smithi090.stdout:adding op weight rmattr -> 25 2023-11-05T08:32:45.064 INFO:tasks.rados.rados.0.smithi090.stdout:adding op weight copy_from -> 50 2023-11-05T08:32:45.065 INFO:tasks.rados.rados.0.smithi090.stdout:adding op weight append -> 100 2023-11-05T08:32:45.065 INFO:tasks.rados.rados.0.smithi090.stdout:ceph version 18.0.0-6776-g420cd129 (420cd129397301b1521ceb74e1bdbd9b855d2189) reef (dev) 2023-11-05T08:32:45.065 INFO:tasks.rados.rados.0.smithi090.stdout:Configuration: 2023-11-05T08:32:45.065 INFO:tasks.rados.rados.0.smithi090.stdout: Number of operations: 400 2023-11-05T08:32:45.065 INFO:tasks.rados.rados.0.smithi090.stdout: Number of objects: 50 2023-11-05T08:32:45.066 INFO:tasks.rados.rados.0.smithi090.stdout: Max in flight operations: 16 2023-11-05T08:32:45.066 INFO:tasks.rados.rados.0.smithi090.stdout: Object size (in bytes): 4000000 2023-11-05T08:32:45.066 INFO:tasks.rados.rados.0.smithi090.stdout: Write stride min: 400000 2023-11-05T08:32:45.066 INFO:tasks.rados.rados.0.smithi090.stdout: Write stride max: 800000 2023-11-05T08:32:45.069 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2023-11-05T08:32:45.089 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:32:45.085+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:32:45.167 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0: {} 2023-11-05T08:32:45.167 INFO:teuthology.orchestra.run.smithi002.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-11-05T08:32:45.187 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1: {} 2023-11-05T08:32:45.187 INFO:teuthology.orchestra.run.smithi002.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-11-05T08:32:45.190 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:32:45.185+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:32:45.191 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:32:45.191 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2023-11-05T08:32:45.191 INFO:teuthology.orchestra.run.smithi069.stdout: "complaint_time": 30, 2023-11-05T08:32:45.192 INFO:teuthology.orchestra.run.smithi069.stdout: "num_blocked_ops": 0 2023-11-05T08:32:45.192 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:32:45.205 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2023-11-05T08:32:45.206 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2: {} 2023-11-05T08:32:45.206 INFO:teuthology.orchestra.run.smithi002.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-11-05T08:32:45.226 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3: {} 2023-11-05T08:32:45.226 INFO:teuthology.orchestra.run.smithi002.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-11-05T08:32:45.244 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4: {} 2023-11-05T08:32:45.244 INFO:teuthology.orchestra.run.smithi002.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-11-05T08:32:45.266 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5: {} 2023-11-05T08:32:45.266 INFO:teuthology.orchestra.run.smithi002.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-11-05T08:32:45.287 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6: {} 2023-11-05T08:32:45.288 INFO:teuthology.orchestra.run.smithi002.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-11-05T08:32:45.290 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:32:45.287+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:32:45.310 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7: {} 2023-11-05T08:32:45.310 INFO:teuthology.orchestra.run.smithi002.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-11-05T08:32:45.330 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:32:45.330 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 20, 2023-11-05T08:32:45.330 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 600, 2023-11-05T08:32:45.330 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [] 2023-11-05T08:32:45.330 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:32:45.331 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8: {} 2023-11-05T08:32:45.331 INFO:teuthology.orchestra.run.smithi002.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-11-05T08:32:45.344 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2023-11-05T08:32:45.353 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9: {} 2023-11-05T08:32:45.354 INFO:teuthology.orchestra.run.smithi002.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-11-05T08:32:45.372 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10: {} 2023-11-05T08:32:45.373 INFO:teuthology.orchestra.run.smithi002.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-11-05T08:32:45.391 INFO:tasks.ceph.osd.0.smithi002.stderr:2023-11-05T08:32:45.391+0000 7f89eaff4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35665) UID: 0 2023-11-05T08:32:45.396 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11: {} 2023-11-05T08:32:45.397 INFO:teuthology.orchestra.run.smithi002.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-11-05T08:32:45.415 INFO:teuthology.orchestra.run.smithi002.stdout:osd.12: {} 2023-11-05T08:32:45.416 INFO:teuthology.orchestra.run.smithi002.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-11-05T08:32:45.438 INFO:teuthology.orchestra.run.smithi002.stdout:osd.13: {} 2023-11-05T08:32:45.439 INFO:teuthology.orchestra.run.smithi002.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-11-05T08:32:45.461 INFO:teuthology.orchestra.run.smithi002.stdout:osd.14: {} 2023-11-05T08:32:45.461 INFO:teuthology.orchestra.run.smithi002.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-11-05T08:32:45.462 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:45.462 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:45.463 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-05T08:32:45.459+0000 7f9fda467640 0 -- 172.21.15.2:0/2381440970 >> [v2:172.21.15.90:6824/739151808,v1:172.21.15.90:6825/739151808] conn(0x7f9fd4223b50 msgr2=0x7f9fd4225f40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-11-05T08:32:45.475 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2023-11-05T08:32:45.493 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-11-05T08:32:45.489+0000 7f966619d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35976) UID: 0 2023-11-05T08:32:45.594 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:32:45.593+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:32:45.597 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:45.597 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:45.611 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2023-11-05T08:32:45.687 INFO:teuthology.orchestra.run.smithi002.stdout:osd.15: {} 2023-11-05T08:32:45.688 INFO:teuthology.orchestra.run.smithi002.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-11-05T08:32:45.694 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:32:45.693+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:32:45.732 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:45.732 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:45.746 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-11-05T08:32:45.795 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:32:45.793+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:32:45.876 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:45.876 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:45.890 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-11-05T08:32:45.895 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:32:45.892+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:32:45.996 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:32:45.993+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:32:46.006 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:46.007 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:46.012 INFO:teuthology.orchestra.run.smithi002.stderr:set osd.3 primary-affinity to 0.046311 (830352) 2023-11-05T08:32:46.019 INFO:tasks.rados.rados.0.smithi090.stdout:1: write initial oid 1 2023-11-05T08:32:46.019 INFO:tasks.rados.rados.0.smithi090.stdout:1: seq_num 0 ranges {0=32768} 2023-11-05T08:32:46.019 INFO:tasks.rados.rados.0.smithi090.stdout:1: writing smithi09039252-1 from 0 to 32768 tid 1 2023-11-05T08:32:46.019 INFO:tasks.rados.rados.0.smithi090.stdout:2: write initial oid 2 2023-11-05T08:32:46.019 INFO:tasks.rados.rados.0.smithi090.stdout:2: seq_num 1 ranges {0=32768} 2023-11-05T08:32:46.020 INFO:tasks.rados.rados.0.smithi090.stdout:2: writing smithi09039252-2 from 0 to 32768 tid 1 2023-11-05T08:32:46.020 INFO:tasks.rados.rados.0.smithi090.stdout:3: write initial oid 3 2023-11-05T08:32:46.021 INFO:tasks.rados.rados.0.smithi090.stdout:3: seq_num 2 ranges {0=49152} 2023-11-05T08:32:46.021 INFO:tasks.rados.rados.0.smithi090.stdout:3: writing smithi09039252-3 from 0 to 49152 tid 1 2023-11-05T08:32:46.021 INFO:tasks.rados.rados.0.smithi090.stdout:4: write initial oid 4 2023-11-05T08:32:46.021 INFO:tasks.rados.rados.0.smithi090.stdout:4: seq_num 3 ranges {0=65536} 2023-11-05T08:32:46.022 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-11-05T08:32:46.023 INFO:tasks.rados.rados.0.smithi090.stdout:4: writing smithi09039252-4 from 0 to 65536 tid 1 2023-11-05T08:32:46.024 INFO:tasks.rados.rados.0.smithi090.stdout:5: write initial oid 5 2023-11-05T08:32:46.024 INFO:tasks.rados.rados.0.smithi090.stdout:5: seq_num 4 ranges {0=81920} 2023-11-05T08:32:46.024 INFO:tasks.rados.rados.0.smithi090.stdout:5: writing smithi09039252-5 from 0 to 81920 tid 1 2023-11-05T08:32:46.024 INFO:tasks.rados.rados.0.smithi090.stdout:6: write initial oid 6 2023-11-05T08:32:46.025 INFO:tasks.rados.rados.0.smithi090.stdout:6: seq_num 5 ranges {0=98304} 2023-11-05T08:32:46.025 INFO:tasks.rados.rados.0.smithi090.stdout:6: writing smithi09039252-6 from 0 to 98304 tid 1 2023-11-05T08:32:46.025 INFO:tasks.rados.rados.0.smithi090.stdout:7: write initial oid 7 2023-11-05T08:32:46.025 INFO:tasks.rados.rados.0.smithi090.stdout:7: seq_num 6 ranges {0=114688} 2023-11-05T08:32:46.025 INFO:tasks.rados.rados.0.smithi090.stdout:7: writing smithi09039252-7 from 0 to 114688 tid 1 2023-11-05T08:32:46.025 INFO:tasks.rados.rados.0.smithi090.stdout:8: write initial oid 8 2023-11-05T08:32:46.026 INFO:tasks.rados.rados.0.smithi090.stdout:8: seq_num 7 ranges {0=131072} 2023-11-05T08:32:46.026 INFO:tasks.rados.rados.0.smithi090.stdout:8: writing smithi09039252-8 from 0 to 131072 tid 1 2023-11-05T08:32:46.027 INFO:tasks.rados.rados.0.smithi090.stdout:9: write initial oid 9 2023-11-05T08:32:46.027 INFO:tasks.rados.rados.0.smithi090.stdout:9: seq_num 8 ranges {0=147456} 2023-11-05T08:32:46.027 INFO:tasks.rados.rados.0.smithi090.stdout:9: writing smithi09039252-9 from 0 to 147456 tid 1 2023-11-05T08:32:46.027 INFO:tasks.rados.rados.0.smithi090.stdout:10: write initial oid 10 2023-11-05T08:32:46.027 INFO:tasks.rados.rados.0.smithi090.stdout:10: seq_num 9 ranges {0=163840} 2023-11-05T08:32:46.029 INFO:tasks.rados.rados.0.smithi090.stdout:10: writing smithi09039252-10 from 0 to 163840 tid 1 2023-11-05T08:32:46.029 INFO:tasks.rados.rados.0.smithi090.stdout:11: write initial oid 11 2023-11-05T08:32:46.029 INFO:tasks.rados.rados.0.smithi090.stdout:11: seq_num 10 ranges {0=180224} 2023-11-05T08:32:46.029 INFO:tasks.rados.rados.0.smithi090.stdout:11: writing smithi09039252-11 from 0 to 180224 tid 1 2023-11-05T08:32:46.029 INFO:tasks.rados.rados.0.smithi090.stdout:12: write initial oid 12 2023-11-05T08:32:46.030 INFO:tasks.rados.rados.0.smithi090.stdout:12: seq_num 11 ranges {0=196608} 2023-11-05T08:32:46.030 INFO:tasks.rados.rados.0.smithi090.stdout:12: writing smithi09039252-12 from 0 to 196608 tid 1 2023-11-05T08:32:46.031 INFO:tasks.rados.rados.0.smithi090.stdout:13: write initial oid 13 2023-11-05T08:32:46.031 INFO:tasks.rados.rados.0.smithi090.stdout:13: seq_num 12 ranges {0=212992} 2023-11-05T08:32:46.034 INFO:tasks.rados.rados.0.smithi090.stdout:13: writing smithi09039252-13 from 0 to 212992 tid 1 2023-11-05T08:32:46.034 INFO:tasks.rados.rados.0.smithi090.stdout:14: write initial oid 14 2023-11-05T08:32:46.034 INFO:tasks.rados.rados.0.smithi090.stdout:14: seq_num 13 ranges {0=229376} 2023-11-05T08:32:46.035 INFO:tasks.rados.rados.0.smithi090.stdout:14: writing smithi09039252-14 from 0 to 229376 tid 1 2023-11-05T08:32:46.035 INFO:tasks.rados.rados.0.smithi090.stdout:15: write initial oid 15 2023-11-05T08:32:46.035 INFO:tasks.rados.rados.0.smithi090.stdout:15: seq_num 14 ranges {0=245760} 2023-11-05T08:32:46.036 INFO:tasks.rados.rados.0.smithi090.stdout:15: writing smithi09039252-15 from 0 to 245760 tid 1 2023-11-05T08:32:46.036 INFO:tasks.rados.rados.0.smithi090.stdout:1: finishing write tid 1 to smithi09039252-1 2023-11-05T08:32:46.036 INFO:tasks.rados.rados.0.smithi090.stdout:1: finishing write tid 2 to smithi09039252-1 2023-11-05T08:32:46.036 INFO:tasks.rados.rados.0.smithi090.stdout:16: write initial oid 16 2023-11-05T08:32:46.037 INFO:tasks.rados.rados.0.smithi090.stdout:16: seq_num 15 ranges {0=262144} 2023-11-05T08:32:46.037 INFO:tasks.rados.rados.0.smithi090.stdout:16: writing smithi09039252-16 from 0 to 262144 tid 1 2023-11-05T08:32:46.038 INFO:tasks.rados.rados.0.smithi090.stdout:1: finishing write tid 3 to smithi09039252-1 2023-11-05T08:32:46.038 INFO:tasks.rados.rados.0.smithi090.stdout:1: oid 1 updating version 0 to 2 2023-11-05T08:32:46.038 INFO:tasks.rados.rados.0.smithi090.stdout:1: oid 1 version 2 is already newer than 1 2023-11-05T08:32:46.038 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 1 v 2 (ObjNum 0 snap 0 seq_num 0) dirty exists 2023-11-05T08:32:46.039 INFO:tasks.rados.rados.0.smithi090.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2023-11-05T08:32:46.039 INFO:tasks.rados.rados.0.smithi090.stdout:1: done (15 left) 2023-11-05T08:32:46.039 INFO:tasks.rados.rados.0.smithi090.stdout:17: write initial oid 17 2023-11-05T08:32:46.039 INFO:tasks.rados.rados.0.smithi090.stdout:17: seq_num 16 ranges {0=278528} 2023-11-05T08:32:46.040 INFO:tasks.rados.rados.0.smithi090.stdout:17: writing smithi09039252-17 from 0 to 278528 tid 1 2023-11-05T08:32:46.040 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-11-05T08:32:46.042 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-11-05T08:32:46.043 INFO:tasks.rados.rados.0.smithi090.stdout:2: finishing write tid 1 to smithi09039252-2 2023-11-05T08:32:46.044 INFO:tasks.rados.rados.0.smithi090.stdout:10: finishing write tid 1 to smithi09039252-10 2023-11-05T08:32:46.045 INFO:tasks.rados.rados.0.smithi090.stdout:2: finishing write tid 2 to smithi09039252-2 2023-11-05T08:32:46.046 INFO:tasks.rados.rados.0.smithi090.stdout:4: finishing write tid 1 to smithi09039252-4 2023-11-05T08:32:46.048 INFO:tasks.rados.rados.0.smithi090.stdout:2: finishing write tid 3 to smithi09039252-2 2023-11-05T08:32:46.049 INFO:tasks.rados.rados.0.smithi090.stdout:2: oid 2 updating version 0 to 1 2023-11-05T08:32:46.049 INFO:tasks.rados.rados.0.smithi090.stdout:2: oid 2 updating version 1 to 2 2023-11-05T08:32:46.049 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 2 v 2 (ObjNum 1 snap 0 seq_num 1) dirty exists 2023-11-05T08:32:46.049 INFO:tasks.rados.rados.0.smithi090.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2023-11-05T08:32:46.049 INFO:tasks.rados.rados.0.smithi090.stdout:2: done (15 left) 2023-11-05T08:32:46.050 INFO:tasks.rados.rados.0.smithi090.stdout:18: write initial oid 18 2023-11-05T08:32:46.050 INFO:tasks.rados.rados.0.smithi090.stdout:18: seq_num 17 ranges {0=294912} 2023-11-05T08:32:46.050 INFO:tasks.rados.rados.0.smithi090.stdout:18: writing smithi09039252-18 from 0 to 294912 tid 1 2023-11-05T08:32:46.051 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-11-05T08:32:46.051 INFO:tasks.rados.rados.0.smithi090.stdout:7: finishing write tid 1 to smithi09039252-7 2023-11-05T08:32:46.051 INFO:tasks.rados.rados.0.smithi090.stdout:4: finishing write tid 2 to smithi09039252-4 2023-11-05T08:32:46.053 INFO:tasks.rados.rados.0.smithi090.stdout:10: finishing write tid 2 to smithi09039252-10 2023-11-05T08:32:46.058 INFO:tasks.rados.rados.0.smithi090.stdout:6: finishing write tid 1 to smithi09039252-6 2023-11-05T08:32:46.058 INFO:tasks.rados.rados.0.smithi090.stdout:6: finishing write tid 2 to smithi09039252-6 2023-11-05T08:32:46.058 INFO:tasks.rados.rados.0.smithi090.stdout:16: finishing write tid 1 to smithi09039252-16 2023-11-05T08:32:46.059 INFO:tasks.rados.rados.0.smithi090.stdout:4: finishing write tid 3 to smithi09039252-4 2023-11-05T08:32:46.059 INFO:tasks.rados.rados.0.smithi090.stdout:4: oid 4 updating version 0 to 4 2023-11-05T08:32:46.060 INFO:tasks.rados.rados.0.smithi090.stdout:4: oid 4 version 4 is already newer than 3 2023-11-05T08:32:46.060 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 4 v 4 (ObjNum 3 snap 0 seq_num 3) dirty exists 2023-11-05T08:32:46.060 INFO:tasks.rados.rados.0.smithi090.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2023-11-05T08:32:46.060 INFO:tasks.rados.rados.0.smithi090.stdout:17: finishing write tid 1 to smithi09039252-17 2023-11-05T08:32:46.060 INFO:tasks.rados.rados.0.smithi090.stdout:4: done (15 left) 2023-11-05T08:32:46.060 INFO:tasks.rados.rados.0.smithi090.stdout:19: write initial oid 19 2023-11-05T08:32:46.061 INFO:tasks.rados.rados.0.smithi090.stdout:19: seq_num 18 ranges {0=311296} 2023-11-05T08:32:46.062 INFO:tasks.rados.rados.0.smithi090.stdout:19: writing smithi09039252-19 from 0 to 311296 tid 1 2023-11-05T08:32:46.062 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-11-05T08:32:46.062 INFO:tasks.rados.rados.0.smithi090.stdout:17: finishing write tid 2 to smithi09039252-17 2023-11-05T08:32:46.062 INFO:tasks.rados.rados.0.smithi090.stdout:10: finishing write tid 3 to smithi09039252-10 2023-11-05T08:32:46.063 INFO:tasks.rados.rados.0.smithi090.stdout:10: oid 10 updating version 0 to 1 2023-11-05T08:32:46.063 INFO:tasks.rados.rados.0.smithi090.stdout:10: oid 10 updating version 1 to 2 2023-11-05T08:32:46.063 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 10 v 2 (ObjNum 9 snap 0 seq_num 9) dirty exists 2023-11-05T08:32:46.063 INFO:tasks.rados.rados.0.smithi090.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2023-11-05T08:32:46.063 INFO:tasks.rados.rados.0.smithi090.stdout:7: finishing write tid 2 to smithi09039252-7 2023-11-05T08:32:46.064 INFO:tasks.rados.rados.0.smithi090.stdout:16: finishing write tid 2 to smithi09039252-16 2023-11-05T08:32:46.064 INFO:tasks.rados.rados.0.smithi090.stdout:10: done (15 left) 2023-11-05T08:32:46.064 INFO:tasks.rados.rados.0.smithi090.stdout:20: write initial oid 20 2023-11-05T08:32:46.064 INFO:tasks.rados.rados.0.smithi090.stdout:20: seq_num 19 ranges {0=327680} 2023-11-05T08:32:46.065 INFO:tasks.rados.rados.0.smithi090.stdout:20: writing smithi09039252-20 from 0 to 327680 tid 1 2023-11-05T08:32:46.065 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-11-05T08:32:46.065 INFO:tasks.rados.rados.0.smithi090.stdout:6: finishing write tid 3 to smithi09039252-6 2023-11-05T08:32:46.066 INFO:tasks.rados.rados.0.smithi090.stdout:6: oid 6 updating version 0 to 5 2023-11-05T08:32:46.066 INFO:tasks.rados.rados.0.smithi090.stdout:6: oid 6 updating version 5 to 6 2023-11-05T08:32:46.066 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 6 v 6 (ObjNum 5 snap 0 seq_num 5) dirty exists 2023-11-05T08:32:46.066 INFO:tasks.rados.rados.0.smithi090.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2023-11-05T08:32:46.066 INFO:tasks.rados.rados.0.smithi090.stdout:6: done (15 left) 2023-11-05T08:32:46.067 INFO:tasks.rados.rados.0.smithi090.stdout:21: write initial oid 21 2023-11-05T08:32:46.067 INFO:tasks.rados.rados.0.smithi090.stdout:21: seq_num 20 ranges {0=344064} 2023-11-05T08:32:46.068 INFO:tasks.rados.rados.0.smithi090.stdout:21: writing smithi09039252-21 from 0 to 344064 tid 1 2023-11-05T08:32:46.068 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-11-05T08:32:46.068 INFO:tasks.rados.rados.0.smithi090.stdout:16: finishing write tid 3 to smithi09039252-16 2023-11-05T08:32:46.068 INFO:tasks.rados.rados.0.smithi090.stdout:16: oid 16 updating version 0 to 7 2023-11-05T08:32:46.069 INFO:tasks.rados.rados.0.smithi090.stdout:16: oid 16 updating version 7 to 8 2023-11-05T08:32:46.069 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 16 v 8 (ObjNum 15 snap 0 seq_num 15) dirty exists 2023-11-05T08:32:46.069 INFO:tasks.rados.rados.0.smithi090.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2023-11-05T08:32:46.069 INFO:tasks.rados.rados.0.smithi090.stdout:17: finishing write tid 3 to smithi09039252-17 2023-11-05T08:32:46.069 INFO:tasks.rados.rados.0.smithi090.stdout:17: oid 17 updating version 0 to 2 2023-11-05T08:32:46.070 INFO:tasks.rados.rados.0.smithi090.stdout:17: oid 17 version 2 is already newer than 1 2023-11-05T08:32:46.070 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 17 v 2 (ObjNum 16 snap 0 seq_num 16) dirty exists 2023-11-05T08:32:46.070 INFO:tasks.rados.rados.0.smithi090.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2023-11-05T08:32:46.070 INFO:tasks.rados.rados.0.smithi090.stdout:16: done (15 left) 2023-11-05T08:32:46.070 INFO:tasks.rados.rados.0.smithi090.stdout:17: done (14 left) 2023-11-05T08:32:46.071 INFO:tasks.rados.rados.0.smithi090.stdout:22: write initial oid 22 2023-11-05T08:32:46.071 INFO:tasks.rados.rados.0.smithi090.stdout:22: seq_num 21 ranges {0=360448} 2023-11-05T08:32:46.071 INFO:tasks.rados.rados.0.smithi090.stdout:22: writing smithi09039252-22 from 0 to 360448 tid 1 2023-11-05T08:32:46.072 INFO:tasks.rados.rados.0.smithi090.stdout:23: write initial oid 23 2023-11-05T08:32:46.072 INFO:tasks.rados.rados.0.smithi090.stdout:23: seq_num 22 ranges {0=376832} 2023-11-05T08:32:46.074 INFO:tasks.rados.rados.0.smithi090.stdout:23: writing smithi09039252-23 from 0 to 376832 tid 1 2023-11-05T08:32:46.074 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-11-05T08:32:46.074 INFO:tasks.rados.rados.0.smithi090.stdout:3: finishing write tid 1 to smithi09039252-3 2023-11-05T08:32:46.074 INFO:tasks.rados.rados.0.smithi090.stdout:3: finishing write tid 2 to smithi09039252-3 2023-11-05T08:32:46.075 INFO:tasks.rados.rados.0.smithi090.stdout:5: finishing write tid 1 to smithi09039252-5 2023-11-05T08:32:46.075 INFO:tasks.rados.rados.0.smithi090.stdout:13: finishing write tid 1 to smithi09039252-13 2023-11-05T08:32:46.075 INFO:tasks.rados.rados.0.smithi090.stdout:5: finishing write tid 2 to smithi09039252-5 2023-11-05T08:32:46.075 INFO:tasks.rados.rados.0.smithi090.stdout:18: finishing write tid 1 to smithi09039252-18 2023-11-05T08:32:46.075 INFO:tasks.rados.rados.0.smithi090.stdout:11: finishing write tid 1 to smithi09039252-11 2023-11-05T08:32:46.076 INFO:tasks.rados.rados.0.smithi090.stdout:18: finishing write tid 2 to smithi09039252-18 2023-11-05T08:32:46.076 INFO:tasks.rados.rados.0.smithi090.stdout:9: finishing write tid 1 to smithi09039252-9 2023-11-05T08:32:46.076 INFO:tasks.rados.rados.0.smithi090.stdout:7: finishing write tid 3 to smithi09039252-7 2023-11-05T08:32:46.076 INFO:tasks.rados.rados.0.smithi090.stdout:7: oid 7 updating version 0 to 1 2023-11-05T08:32:46.076 INFO:tasks.rados.rados.0.smithi090.stdout:7: oid 7 updating version 1 to 2 2023-11-05T08:32:46.076 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 7 v 2 (ObjNum 6 snap 0 seq_num 6) dirty exists 2023-11-05T08:32:46.077 INFO:tasks.rados.rados.0.smithi090.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2023-11-05T08:32:46.077 INFO:tasks.rados.rados.0.smithi090.stdout:11: finishing write tid 2 to smithi09039252-11 2023-11-05T08:32:46.077 INFO:tasks.rados.rados.0.smithi090.stdout:13: finishing write tid 2 to smithi09039252-13 2023-11-05T08:32:46.077 INFO:tasks.rados.rados.0.smithi090.stdout:18: finishing write tid 3 to smithi09039252-18 2023-11-05T08:32:46.077 INFO:tasks.rados.rados.0.smithi090.stdout:18: oid 18 updating version 0 to 2 2023-11-05T08:32:46.078 INFO:tasks.rados.rados.0.smithi090.stdout:18: oid 18 version 2 is already newer than 1 2023-11-05T08:32:46.078 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 18 v 2 (ObjNum 17 snap 0 seq_num 17) dirty exists 2023-11-05T08:32:46.078 INFO:tasks.rados.rados.0.smithi090.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2023-11-05T08:32:46.078 INFO:tasks.rados.rados.0.smithi090.stdout:8: finishing write tid 1 to smithi09039252-8 2023-11-05T08:32:46.078 INFO:tasks.rados.rados.0.smithi090.stdout:8: finishing write tid 2 to smithi09039252-8 2023-11-05T08:32:46.079 INFO:tasks.rados.rados.0.smithi090.stdout:7: done (15 left) 2023-11-05T08:32:46.079 INFO:tasks.rados.rados.0.smithi090.stdout:18: done (14 left) 2023-11-05T08:32:46.079 INFO:tasks.rados.rados.0.smithi090.stdout:24: write initial oid 24 2023-11-05T08:32:46.079 INFO:tasks.rados.rados.0.smithi090.stdout:24: seq_num 23 ranges {0=393216} 2023-11-05T08:32:46.080 INFO:tasks.rados.rados.0.smithi090.stdout:24: writing smithi09039252-24 from 0 to 393216 tid 1 2023-11-05T08:32:46.080 INFO:tasks.rados.rados.0.smithi090.stdout:25: write initial oid 25 2023-11-05T08:32:46.080 INFO:tasks.rados.rados.0.smithi090.stdout:25: seq_num 24 ranges {0=409600} 2023-11-05T08:32:46.082 INFO:tasks.rados.rados.0.smithi090.stdout:25: writing smithi09039252-25 from 0 to 409600 tid 1 2023-11-05T08:32:46.082 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-11-05T08:32:46.082 INFO:tasks.rados.rados.0.smithi090.stdout:5: finishing write tid 3 to smithi09039252-5 2023-11-05T08:32:46.082 INFO:tasks.rados.rados.0.smithi090.stdout:5: oid 5 updating version 0 to 2 2023-11-05T08:32:46.083 INFO:tasks.rados.rados.0.smithi090.stdout:5: oid 5 version 2 is already newer than 1 2023-11-05T08:32:46.083 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 5 v 2 (ObjNum 4 snap 0 seq_num 4) dirty exists 2023-11-05T08:32:46.083 INFO:tasks.rados.rados.0.smithi090.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2023-11-05T08:32:46.083 INFO:tasks.rados.rados.0.smithi090.stdout:3: finishing write tid 3 to smithi09039252-3 2023-11-05T08:32:46.083 INFO:tasks.rados.rados.0.smithi090.stdout:3: oid 3 updating version 0 to 2 2023-11-05T08:32:46.084 INFO:tasks.rados.rados.0.smithi090.stdout:3: oid 3 version 2 is already newer than 1 2023-11-05T08:32:46.084 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 3 v 2 (ObjNum 2 snap 0 seq_num 2) dirty exists 2023-11-05T08:32:46.084 INFO:tasks.rados.rados.0.smithi090.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2023-11-05T08:32:46.084 INFO:tasks.rados.rados.0.smithi090.stdout:19: finishing write tid 1 to smithi09039252-19 2023-11-05T08:32:46.084 INFO:tasks.rados.rados.0.smithi090.stdout:20: finishing write tid 1 to smithi09039252-20 2023-11-05T08:32:46.085 INFO:tasks.rados.rados.0.smithi090.stdout:12: finishing write tid 1 to smithi09039252-12 2023-11-05T08:32:46.085 INFO:tasks.rados.rados.0.smithi090.stdout:12: finishing write tid 2 to smithi09039252-12 2023-11-05T08:32:46.085 INFO:tasks.rados.rados.0.smithi090.stdout:19: finishing write tid 2 to smithi09039252-19 2023-11-05T08:32:46.085 INFO:tasks.rados.rados.0.smithi090.stdout:11: finishing write tid 3 to smithi09039252-11 2023-11-05T08:32:46.085 INFO:tasks.rados.rados.0.smithi090.stdout:11: oid 11 updating version 0 to 1 2023-11-05T08:32:46.086 INFO:tasks.rados.rados.0.smithi090.stdout:11: oid 11 updating version 1 to 2 2023-11-05T08:32:46.086 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 11 v 2 (ObjNum 10 snap 0 seq_num 10) dirty exists 2023-11-05T08:32:46.086 INFO:tasks.rados.rados.0.smithi090.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2023-11-05T08:32:46.086 INFO:tasks.rados.rados.0.smithi090.stdout:9: finishing write tid 2 to smithi09039252-9 2023-11-05T08:32:46.087 INFO:tasks.rados.rados.0.smithi090.stdout:13: finishing write tid 3 to smithi09039252-13 2023-11-05T08:32:46.087 INFO:tasks.rados.rados.0.smithi090.stdout:13: oid 13 updating version 0 to 4 2023-11-05T08:32:46.087 INFO:tasks.rados.rados.0.smithi090.stdout:13: oid 13 version 4 is already newer than 3 2023-11-05T08:32:46.087 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 13 v 4 (ObjNum 12 snap 0 seq_num 12) dirty exists 2023-11-05T08:32:46.087 INFO:tasks.rados.rados.0.smithi090.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2023-11-05T08:32:46.087 INFO:tasks.rados.rados.0.smithi090.stdout:19: finishing write tid 3 to smithi09039252-19 2023-11-05T08:32:46.088 INFO:tasks.rados.rados.0.smithi090.stdout:19: oid 19 updating version 0 to 1 2023-11-05T08:32:46.088 INFO:tasks.rados.rados.0.smithi090.stdout:19: oid 19 updating version 1 to 2 2023-11-05T08:32:46.088 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 19 v 2 (ObjNum 18 snap 0 seq_num 18) dirty exists 2023-11-05T08:32:46.088 INFO:tasks.rados.rados.0.smithi090.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2023-11-05T08:32:46.089 INFO:tasks.rados.rados.0.smithi090.stdout:3: done (15 left) 2023-11-05T08:32:46.089 INFO:tasks.rados.rados.0.smithi090.stdout:5: done (14 left) 2023-11-05T08:32:46.089 INFO:tasks.rados.rados.0.smithi090.stdout:11: done (13 left) 2023-11-05T08:32:46.089 INFO:tasks.rados.rados.0.smithi090.stdout:13: done (12 left) 2023-11-05T08:32:46.089 INFO:tasks.rados.rados.0.smithi090.stdout:19: done (11 left) 2023-11-05T08:32:46.090 INFO:tasks.rados.rados.0.smithi090.stdout:26: write initial oid 26 2023-11-05T08:32:46.090 INFO:tasks.rados.rados.0.smithi090.stdout:26: seq_num 25 ranges {0=425984} 2023-11-05T08:32:46.091 INFO:tasks.rados.rados.0.smithi090.stdout:26: writing smithi09039252-26 from 0 to 425984 tid 1 2023-11-05T08:32:46.091 INFO:tasks.rados.rados.0.smithi090.stdout:27: write initial oid 27 2023-11-05T08:32:46.092 INFO:tasks.rados.rados.0.smithi090.stdout:27: seq_num 26 ranges {0=442368} 2023-11-05T08:32:46.092 INFO:tasks.rados.rados.0.smithi090.stdout:27: writing smithi09039252-27 from 0 to 442368 tid 1 2023-11-05T08:32:46.092 INFO:tasks.rados.rados.0.smithi090.stdout:28: write initial oid 28 2023-11-05T08:32:46.092 INFO:tasks.rados.rados.0.smithi090.stdout:28: seq_num 27 ranges {0=458752} 2023-11-05T08:32:46.092 INFO:tasks.rados.rados.0.smithi090.stdout:28: writing smithi09039252-28 from 0 to 458752 tid 1 2023-11-05T08:32:46.092 INFO:tasks.rados.rados.0.smithi090.stdout:29: write initial oid 29 2023-11-05T08:32:46.093 INFO:tasks.rados.rados.0.smithi090.stdout:29: seq_num 28 ranges {0=475136} 2023-11-05T08:32:46.094 INFO:tasks.rados.rados.0.smithi090.stdout:29: writing smithi09039252-29 from 0 to 475136 tid 1 2023-11-05T08:32:46.095 INFO:tasks.rados.rados.0.smithi090.stdout:30: write initial oid 30 2023-11-05T08:32:46.095 INFO:tasks.rados.rados.0.smithi090.stdout:30: seq_num 29 ranges {0=491520} 2023-11-05T08:32:46.097 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:32:46.093+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:32:46.098 INFO:tasks.rados.rados.0.smithi090.stdout:30: writing smithi09039252-30 from 0 to 491520 tid 1 2023-11-05T08:32:46.098 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-11-05T08:32:46.099 INFO:tasks.rados.rados.0.smithi090.stdout:23: finishing write tid 1 to smithi09039252-23 2023-11-05T08:32:46.099 INFO:tasks.rados.rados.0.smithi090.stdout:22: finishing write tid 1 to smithi09039252-22 2023-11-05T08:32:46.099 INFO:tasks.rados.rados.0.smithi090.stdout:20: finishing write tid 2 to smithi09039252-20 2023-11-05T08:32:46.099 INFO:tasks.rados.rados.0.smithi090.stdout:22: finishing write tid 2 to smithi09039252-22 2023-11-05T08:32:46.099 INFO:tasks.rados.rados.0.smithi090.stdout:14: finishing write tid 1 to smithi09039252-14 2023-11-05T08:32:46.100 INFO:tasks.rados.rados.0.smithi090.stdout:8: finishing write tid 3 to smithi09039252-8 2023-11-05T08:32:46.100 INFO:tasks.rados.rados.0.smithi090.stdout:8: oid 8 updating version 0 to 1 2023-11-05T08:32:46.100 INFO:tasks.rados.rados.0.smithi090.stdout:8: oid 8 updating version 1 to 2 2023-11-05T08:32:46.100 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 8 v 2 (ObjNum 7 snap 0 seq_num 7) dirty exists 2023-11-05T08:32:46.100 INFO:tasks.rados.rados.0.smithi090.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2023-11-05T08:32:46.101 INFO:tasks.rados.rados.0.smithi090.stdout:23: finishing write tid 2 to smithi09039252-23 2023-11-05T08:32:46.101 INFO:tasks.rados.rados.0.smithi090.stdout:21: finishing write tid 1 to smithi09039252-21 2023-11-05T08:32:46.101 INFO:tasks.rados.rados.0.smithi090.stdout:8: done (15 left) 2023-11-05T08:32:46.101 INFO:tasks.rados.rados.0.smithi090.stdout:31: write initial oid 31 2023-11-05T08:32:46.101 INFO:tasks.rados.rados.0.smithi090.stdout:31: seq_num 30 ranges {0=507904} 2023-11-05T08:32:46.103 INFO:tasks.rados.rados.0.smithi090.stdout:31: writing smithi09039252-31 from 0 to 507904 tid 1 2023-11-05T08:32:46.103 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-11-05T08:32:46.103 INFO:tasks.rados.rados.0.smithi090.stdout:9: finishing write tid 3 to smithi09039252-9 2023-11-05T08:32:46.104 INFO:tasks.rados.rados.0.smithi090.stdout:9: oid 9 updating version 0 to 3 2023-11-05T08:32:46.104 INFO:tasks.rados.rados.0.smithi090.stdout:9: oid 9 updating version 3 to 4 2023-11-05T08:32:46.104 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 9 v 4 (ObjNum 8 snap 0 seq_num 8) dirty exists 2023-11-05T08:32:46.104 INFO:tasks.rados.rados.0.smithi090.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2023-11-05T08:32:46.104 INFO:tasks.rados.rados.0.smithi090.stdout:20: finishing write tid 3 to smithi09039252-20 2023-11-05T08:32:46.105 INFO:tasks.rados.rados.0.smithi090.stdout:20: oid 20 updating version 0 to 3 2023-11-05T08:32:46.105 INFO:tasks.rados.rados.0.smithi090.stdout:20: oid 20 updating version 3 to 4 2023-11-05T08:32:46.105 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 20 v 4 (ObjNum 19 snap 0 seq_num 19) dirty exists 2023-11-05T08:32:46.105 INFO:tasks.rados.rados.0.smithi090.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2023-11-05T08:32:46.105 INFO:tasks.rados.rados.0.smithi090.stdout:23: finishing write tid 3 to smithi09039252-23 2023-11-05T08:32:46.105 INFO:tasks.rados.rados.0.smithi090.stdout:23: oid 23 updating version 0 to 4 2023-11-05T08:32:46.106 INFO:tasks.rados.rados.0.smithi090.stdout:23: oid 23 version 4 is already newer than 3 2023-11-05T08:32:46.106 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 23 v 4 (ObjNum 22 snap 0 seq_num 22) dirty exists 2023-11-05T08:32:46.106 INFO:tasks.rados.rados.0.smithi090.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2023-11-05T08:32:46.106 INFO:tasks.rados.rados.0.smithi090.stdout:22: finishing write tid 3 to smithi09039252-22 2023-11-05T08:32:46.107 INFO:tasks.rados.rados.0.smithi090.stdout:22: oid 22 updating version 0 to 3 2023-11-05T08:32:46.107 INFO:tasks.rados.rados.0.smithi090.stdout:22: oid 22 updating version 3 to 4 2023-11-05T08:32:46.107 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 22 v 4 (ObjNum 21 snap 0 seq_num 21) dirty exists 2023-11-05T08:32:46.107 INFO:tasks.rados.rados.0.smithi090.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2023-11-05T08:32:46.107 INFO:tasks.rados.rados.0.smithi090.stdout:12: finishing write tid 3 to smithi09039252-12 2023-11-05T08:32:46.108 INFO:tasks.rados.rados.0.smithi090.stdout:12: oid 12 updating version 0 to 3 2023-11-05T08:32:46.108 INFO:tasks.rados.rados.0.smithi090.stdout:12: oid 12 updating version 3 to 4 2023-11-05T08:32:46.108 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 12 v 4 (ObjNum 11 snap 0 seq_num 11) dirty exists 2023-11-05T08:32:46.108 INFO:tasks.rados.rados.0.smithi090.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2023-11-05T08:32:46.108 INFO:tasks.rados.rados.0.smithi090.stdout:14: finishing write tid 2 to smithi09039252-14 2023-11-05T08:32:46.109 INFO:tasks.rados.rados.0.smithi090.stdout:21: finishing write tid 2 to smithi09039252-21 2023-11-05T08:32:46.109 INFO:tasks.rados.rados.0.smithi090.stdout:24: finishing write tid 1 to smithi09039252-24 2023-11-05T08:32:46.109 INFO:tasks.rados.rados.0.smithi090.stdout:26: finishing write tid 1 to smithi09039252-26 2023-11-05T08:32:46.109 INFO:tasks.rados.rados.0.smithi090.stdout:26: finishing write tid 2 to smithi09039252-26 2023-11-05T08:32:46.110 INFO:tasks.rados.rados.0.smithi090.stdout:9: done (15 left) 2023-11-05T08:32:46.110 INFO:tasks.rados.rados.0.smithi090.stdout:12: done (14 left) 2023-11-05T08:32:46.110 INFO:tasks.rados.rados.0.smithi090.stdout:20: done (13 left) 2023-11-05T08:32:46.110 INFO:tasks.rados.rados.0.smithi090.stdout:22: done (12 left) 2023-11-05T08:32:46.110 INFO:tasks.rados.rados.0.smithi090.stdout:23: done (11 left) 2023-11-05T08:32:46.111 INFO:tasks.rados.rados.0.smithi090.stdout:32: write initial oid 32 2023-11-05T08:32:46.111 INFO:tasks.rados.rados.0.smithi090.stdout:32: seq_num 31 ranges {0=524288} 2023-11-05T08:32:46.112 INFO:tasks.rados.rados.0.smithi090.stdout:32: writing smithi09039252-32 from 0 to 524288 tid 1 2023-11-05T08:32:46.112 INFO:tasks.rados.rados.0.smithi090.stdout:33: write initial oid 33 2023-11-05T08:32:46.112 INFO:tasks.rados.rados.0.smithi090.stdout:33: seq_num 32 ranges {0=540672} 2023-11-05T08:32:46.112 INFO:tasks.rados.rados.0.smithi090.stdout:33: writing smithi09039252-33 from 0 to 540672 tid 1 2023-11-05T08:32:46.112 INFO:tasks.rados.rados.0.smithi090.stdout:34: write initial oid 34 2023-11-05T08:32:46.113 INFO:tasks.rados.rados.0.smithi090.stdout:34: seq_num 33 ranges {0=557056} 2023-11-05T08:32:46.114 INFO:tasks.rados.rados.0.smithi090.stdout:34: writing smithi09039252-34 from 0 to 557056 tid 1 2023-11-05T08:32:46.114 INFO:tasks.rados.rados.0.smithi090.stdout:35: write initial oid 35 2023-11-05T08:32:46.115 INFO:tasks.rados.rados.0.smithi090.stdout:35: seq_num 34 ranges {0=573440} 2023-11-05T08:32:46.118 INFO:tasks.rados.rados.0.smithi090.stdout:35: writing smithi09039252-35 from 0 to 573440 tid 1 2023-11-05T08:32:46.119 INFO:tasks.rados.rados.0.smithi090.stdout:36: write initial oid 36 2023-11-05T08:32:46.119 INFO:tasks.rados.rados.0.smithi090.stdout:36: seq_num 35 ranges {0=589824} 2023-11-05T08:32:46.123 INFO:tasks.rados.rados.0.smithi090.stdout:36: writing smithi09039252-36 from 0 to 589824 tid 1 2023-11-05T08:32:46.124 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-11-05T08:32:46.124 INFO:tasks.rados.rados.0.smithi090.stdout:21: finishing write tid 3 to smithi09039252-21 2023-11-05T08:32:46.124 INFO:tasks.rados.rados.0.smithi090.stdout:21: oid 21 updating version 0 to 1 2023-11-05T08:32:46.124 INFO:tasks.rados.rados.0.smithi090.stdout:21: oid 21 updating version 1 to 2 2023-11-05T08:32:46.125 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 21 v 2 (ObjNum 20 snap 0 seq_num 20) dirty exists 2023-11-05T08:32:46.125 INFO:tasks.rados.rados.0.smithi090.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2023-11-05T08:32:46.125 INFO:tasks.rados.rados.0.smithi090.stdout:26: finishing write tid 3 to smithi09039252-26 2023-11-05T08:32:46.125 INFO:tasks.rados.rados.0.smithi090.stdout:26: oid 26 updating version 0 to 1 2023-11-05T08:32:46.125 INFO:tasks.rados.rados.0.smithi090.stdout:26: oid 26 updating version 1 to 2 2023-11-05T08:32:46.126 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 26 v 2 (ObjNum 25 snap 0 seq_num 25) dirty exists 2023-11-05T08:32:46.126 INFO:tasks.rados.rados.0.smithi090.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2023-11-05T08:32:46.126 INFO:tasks.rados.rados.0.smithi090.stdout:25: finishing write tid 1 to smithi09039252-25 2023-11-05T08:32:46.126 INFO:tasks.rados.rados.0.smithi090.stdout:24: finishing write tid 2 to smithi09039252-24 2023-11-05T08:32:46.127 INFO:tasks.rados.rados.0.smithi090.stdout:21: done (15 left) 2023-11-05T08:32:46.127 INFO:tasks.rados.rados.0.smithi090.stdout:26: done (14 left) 2023-11-05T08:32:46.127 INFO:tasks.rados.rados.0.smithi090.stdout:37: write initial oid 37 2023-11-05T08:32:46.127 INFO:tasks.rados.rados.0.smithi090.stdout:37: seq_num 36 ranges {0=606208} 2023-11-05T08:32:46.128 INFO:tasks.rados.rados.0.smithi090.stdout:37: writing smithi09039252-37 from 0 to 606208 tid 1 2023-11-05T08:32:46.128 INFO:tasks.rados.rados.0.smithi090.stdout:38: write initial oid 38 2023-11-05T08:32:46.129 INFO:tasks.rados.rados.0.smithi090.stdout:38: seq_num 37 ranges {0=622592} 2023-11-05T08:32:46.132 INFO:tasks.rados.rados.0.smithi090.stdout:38: writing smithi09039252-38 from 0 to 622592 tid 1 2023-11-05T08:32:46.133 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-11-05T08:32:46.133 INFO:tasks.rados.rados.0.smithi090.stdout:14: finishing write tid 3 to smithi09039252-14 2023-11-05T08:32:46.133 INFO:tasks.rados.rados.0.smithi090.stdout:14: oid 14 updating version 0 to 6 2023-11-05T08:32:46.133 INFO:tasks.rados.rados.0.smithi090.stdout:14: oid 14 version 6 is already newer than 5 2023-11-05T08:32:46.134 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 14 v 6 (ObjNum 13 snap 0 seq_num 13) dirty exists 2023-11-05T08:32:46.134 INFO:tasks.rados.rados.0.smithi090.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2023-11-05T08:32:46.134 INFO:tasks.rados.rados.0.smithi090.stdout:25: finishing write tid 2 to smithi09039252-25 2023-11-05T08:32:46.134 INFO:tasks.rados.rados.0.smithi090.stdout:14: done (15 left) 2023-11-05T08:32:46.134 INFO:tasks.rados.rados.0.smithi090.stdout:39: write initial oid 39 2023-11-05T08:32:46.135 INFO:tasks.rados.rados.0.smithi090.stdout:39: seq_num 38 ranges {0=638976} 2023-11-05T08:32:46.137 INFO:tasks.rados.rados.0.smithi090.stdout:39: writing smithi09039252-39 from 0 to 638976 tid 1 2023-11-05T08:32:46.138 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-11-05T08:32:46.138 INFO:tasks.rados.rados.0.smithi090.stdout:24: finishing write tid 3 to smithi09039252-24 2023-11-05T08:32:46.138 INFO:tasks.rados.rados.0.smithi090.stdout:24: oid 24 updating version 0 to 3 2023-11-05T08:32:46.138 INFO:tasks.rados.rados.0.smithi090.stdout:24: oid 24 updating version 3 to 4 2023-11-05T08:32:46.138 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 24 v 4 (ObjNum 23 snap 0 seq_num 23) dirty exists 2023-11-05T08:32:46.139 INFO:tasks.rados.rados.0.smithi090.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2023-11-05T08:32:46.139 INFO:tasks.rados.rados.0.smithi090.stdout:28: finishing write tid 1 to smithi09039252-28 2023-11-05T08:32:46.139 INFO:tasks.rados.rados.0.smithi090.stdout:29: finishing write tid 1 to smithi09039252-29 2023-11-05T08:32:46.139 INFO:tasks.rados.rados.0.smithi090.stdout:29: finishing write tid 2 to smithi09039252-29 2023-11-05T08:32:46.139 INFO:tasks.rados.rados.0.smithi090.stdout:24: done (15 left) 2023-11-05T08:32:46.140 INFO:tasks.rados.rados.0.smithi090.stdout:40: write initial oid 40 2023-11-05T08:32:46.140 INFO:tasks.rados.rados.0.smithi090.stdout:40: seq_num 39 ranges {0=671744} 2023-11-05T08:32:46.143 INFO:tasks.rados.rados.0.smithi090.stdout:40: writing smithi09039252-40 from 0 to 671744 tid 1 2023-11-05T08:32:46.143 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-11-05T08:32:46.143 INFO:tasks.rados.rados.0.smithi090.stdout:30: finishing write tid 1 to smithi09039252-30 2023-11-05T08:32:46.144 INFO:tasks.rados.rados.0.smithi090.stdout:28: finishing write tid 2 to smithi09039252-28 2023-11-05T08:32:46.144 INFO:tasks.rados.rados.0.smithi090.stdout:30: finishing write tid 2 to smithi09039252-30 2023-11-05T08:32:46.144 INFO:tasks.rados.rados.0.smithi090.stdout:27: finishing write tid 1 to smithi09039252-27 2023-11-05T08:32:46.144 INFO:tasks.rados.rados.0.smithi090.stdout:29: finishing write tid 3 to smithi09039252-29 2023-11-05T08:32:46.144 INFO:tasks.rados.rados.0.smithi090.stdout:29: oid 29 updating version 0 to 5 2023-11-05T08:32:46.145 INFO:tasks.rados.rados.0.smithi090.stdout:29: oid 29 updating version 5 to 6 2023-11-05T08:32:46.145 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 29 v 6 (ObjNum 28 snap 0 seq_num 28) dirty exists 2023-11-05T08:32:46.145 INFO:tasks.rados.rados.0.smithi090.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2023-11-05T08:32:46.145 INFO:tasks.rados.rados.0.smithi090.stdout:30: finishing write tid 3 to smithi09039252-30 2023-11-05T08:32:46.145 INFO:tasks.rados.rados.0.smithi090.stdout:30: oid 30 updating version 0 to 3 2023-11-05T08:32:46.146 INFO:tasks.rados.rados.0.smithi090.stdout:30: oid 30 updating version 3 to 4 2023-11-05T08:32:46.146 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 30 v 4 (ObjNum 29 snap 0 seq_num 29) dirty exists 2023-11-05T08:32:46.146 INFO:tasks.rados.rados.0.smithi090.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2023-11-05T08:32:46.146 INFO:tasks.rados.rados.0.smithi090.stdout:25: finishing write tid 3 to smithi09039252-25 2023-11-05T08:32:46.146 INFO:tasks.rados.rados.0.smithi090.stdout:25: oid 25 updating version 0 to 8 2023-11-05T08:32:46.147 INFO:tasks.rados.rados.0.smithi090.stdout:25: oid 25 version 8 is already newer than 7 2023-11-05T08:32:46.147 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 25 v 8 (ObjNum 24 snap 0 seq_num 24) dirty exists 2023-11-05T08:32:46.147 INFO:tasks.rados.rados.0.smithi090.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2023-11-05T08:32:46.147 INFO:tasks.rados.rados.0.smithi090.stdout:28: finishing write tid 3 to smithi09039252-28 2023-11-05T08:32:46.147 INFO:tasks.rados.rados.0.smithi090.stdout:28: oid 28 updating version 0 to 6 2023-11-05T08:32:46.148 INFO:tasks.rados.rados.0.smithi090.stdout:28: oid 28 version 6 is already newer than 5 2023-11-05T08:32:46.148 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 28 v 6 (ObjNum 27 snap 0 seq_num 27) dirty exists 2023-11-05T08:32:46.148 INFO:tasks.rados.rados.0.smithi090.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2023-11-05T08:32:46.148 INFO:tasks.rados.rados.0.smithi090.stdout:27: finishing write tid 2 to smithi09039252-27 2023-11-05T08:32:46.148 INFO:tasks.rados.rados.0.smithi090.stdout:25: done (15 left) 2023-11-05T08:32:46.148 INFO:tasks.rados.rados.0.smithi090.stdout:28: done (14 left) 2023-11-05T08:32:46.149 INFO:tasks.rados.rados.0.smithi090.stdout:29: done (13 left) 2023-11-05T08:32:46.149 INFO:tasks.rados.rados.0.smithi090.stdout:30: done (12 left) 2023-11-05T08:32:46.149 INFO:tasks.rados.rados.0.smithi090.stdout:41: write initial oid 41 2023-11-05T08:32:46.149 INFO:tasks.rados.rados.0.smithi090.stdout:41: seq_num 40 ranges {0=688128} 2023-11-05T08:32:46.150 INFO:tasks.rados.rados.0.smithi090.stdout:41: writing smithi09039252-41 from 0 to 688128 tid 1 2023-11-05T08:32:46.150 INFO:tasks.rados.rados.0.smithi090.stdout:42: write initial oid 42 2023-11-05T08:32:46.150 INFO:tasks.rados.rados.0.smithi090.stdout:42: seq_num 41 ranges {0=704512} 2023-11-05T08:32:46.152 INFO:tasks.rados.rados.0.smithi090.stdout:42: writing smithi09039252-42 from 0 to 704512 tid 1 2023-11-05T08:32:46.153 INFO:tasks.rados.rados.0.smithi090.stdout:43: write initial oid 43 2023-11-05T08:32:46.153 INFO:tasks.rados.rados.0.smithi090.stdout:43: seq_num 42 ranges {0=720896} 2023-11-05T08:32:46.157 INFO:tasks.rados.rados.0.smithi090.stdout:43: writing smithi09039252-43 from 0 to 720896 tid 1 2023-11-05T08:32:46.158 INFO:tasks.rados.rados.0.smithi090.stdout:44: write initial oid 44 2023-11-05T08:32:46.158 INFO:tasks.rados.rados.0.smithi090.stdout:44: seq_num 43 ranges {0=737280} 2023-11-05T08:32:46.162 INFO:tasks.rados.rados.0.smithi090.stdout:44: writing smithi09039252-44 from 0 to 737280 tid 1 2023-11-05T08:32:46.162 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-11-05T08:32:46.163 INFO:tasks.rados.rados.0.smithi090.stdout:32: finishing write tid 1 to smithi09039252-32 2023-11-05T08:32:46.163 INFO:tasks.rados.rados.0.smithi090.stdout:32: finishing write tid 2 to smithi09039252-32 2023-11-05T08:32:46.163 INFO:tasks.rados.rados.0.smithi090.stdout:33: finishing write tid 1 to smithi09039252-33 2023-11-05T08:32:46.163 INFO:tasks.rados.rados.0.smithi090.stdout:27: finishing write tid 3 to smithi09039252-27 2023-11-05T08:32:46.163 INFO:tasks.rados.rados.0.smithi090.stdout:27: oid 27 updating version 0 to 9 2023-11-05T08:32:46.164 INFO:tasks.rados.rados.0.smithi090.stdout:27: oid 27 updating version 9 to 10 2023-11-05T08:32:46.164 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 27 v 10 (ObjNum 26 snap 0 seq_num 26) dirty exists 2023-11-05T08:32:46.164 INFO:tasks.rados.rados.0.smithi090.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2023-11-05T08:32:46.164 INFO:tasks.rados.rados.0.smithi090.stdout:27: done (15 left) 2023-11-05T08:32:46.165 INFO:tasks.rados.rados.0.smithi090.stdout:45: write initial oid 45 2023-11-05T08:32:46.165 INFO:tasks.rados.rados.0.smithi090.stdout:45: seq_num 44 ranges {0=753664} 2023-11-05T08:32:46.167 INFO:tasks.rados.rados.0.smithi090.stdout:45: writing smithi09039252-45 from 0 to 753664 tid 1 2023-11-05T08:32:46.167 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-11-05T08:32:46.167 INFO:tasks.rados.rados.0.smithi090.stdout:33: finishing write tid 2 to smithi09039252-33 2023-11-05T08:32:46.167 INFO:tasks.rados.rados.0.smithi090.stdout:32: finishing write tid 3 to smithi09039252-32 2023-11-05T08:32:46.168 INFO:tasks.rados.rados.0.smithi090.stdout:32: oid 32 updating version 0 to 4 2023-11-05T08:32:46.168 INFO:tasks.rados.rados.0.smithi090.stdout:32: oid 32 version 4 is already newer than 3 2023-11-05T08:32:46.168 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 32 v 4 (ObjNum 31 snap 0 seq_num 31) dirty exists 2023-11-05T08:32:46.168 INFO:tasks.rados.rados.0.smithi090.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2023-11-05T08:32:46.168 INFO:tasks.rados.rados.0.smithi090.stdout:32: done (15 left) 2023-11-05T08:32:46.168 INFO:tasks.rados.rados.0.smithi090.stdout:46: write initial oid 46 2023-11-05T08:32:46.169 INFO:tasks.rados.rados.0.smithi090.stdout:46: seq_num 45 ranges {0=770048} 2023-11-05T08:32:46.172 INFO:tasks.rados.rados.0.smithi090.stdout:46: writing smithi09039252-46 from 0 to 770048 tid 1 2023-11-05T08:32:46.172 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-11-05T08:32:46.173 INFO:tasks.rados.rados.0.smithi090.stdout:33: finishing write tid 3 to smithi09039252-33 2023-11-05T08:32:46.173 INFO:tasks.rados.rados.0.smithi090.stdout:33: oid 33 updating version 0 to 3 2023-11-05T08:32:46.173 INFO:tasks.rados.rados.0.smithi090.stdout:33: oid 33 updating version 3 to 4 2023-11-05T08:32:46.173 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 33 v 4 (ObjNum 32 snap 0 seq_num 32) dirty exists 2023-11-05T08:32:46.174 INFO:tasks.rados.rados.0.smithi090.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2023-11-05T08:32:46.174 INFO:tasks.rados.rados.0.smithi090.stdout:34: finishing write tid 1 to smithi09039252-34 2023-11-05T08:32:46.174 INFO:tasks.rados.rados.0.smithi090.stdout:34: finishing write tid 2 to smithi09039252-34 2023-11-05T08:32:46.174 INFO:tasks.rados.rados.0.smithi090.stdout:34: finishing write tid 3 to smithi09039252-34 2023-11-05T08:32:46.174 INFO:tasks.rados.rados.0.smithi090.stdout:34: oid 34 updating version 0 to 11 2023-11-05T08:32:46.175 INFO:tasks.rados.rados.0.smithi090.stdout:34: oid 34 updating version 11 to 12 2023-11-05T08:32:46.175 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 34 v 12 (ObjNum 33 snap 0 seq_num 33) dirty exists 2023-11-05T08:32:46.175 INFO:tasks.rados.rados.0.smithi090.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2023-11-05T08:32:46.175 INFO:tasks.rados.rados.0.smithi090.stdout:35: finishing write tid 1 to smithi09039252-35 2023-11-05T08:32:46.176 INFO:tasks.rados.rados.0.smithi090.stdout:35: finishing write tid 2 to smithi09039252-35 2023-11-05T08:32:46.176 INFO:tasks.rados.rados.0.smithi090.stdout:33: done (15 left) 2023-11-05T08:32:46.176 INFO:tasks.rados.rados.0.smithi090.stdout:34: done (14 left) 2023-11-05T08:32:46.176 INFO:tasks.rados.rados.0.smithi090.stdout:47: write initial oid 47 2023-11-05T08:32:46.176 INFO:tasks.rados.rados.0.smithi090.stdout:47: seq_num 46 ranges {0=786432} 2023-11-05T08:32:46.177 INFO:tasks.rados.rados.0.smithi090.stdout:47: writing smithi09039252-47 from 0 to 786432 tid 1 2023-11-05T08:32:46.178 INFO:tasks.rados.rados.0.smithi090.stdout:48: write initial oid 48 2023-11-05T08:32:46.178 INFO:tasks.rados.rados.0.smithi090.stdout:48: seq_num 47 ranges {0=425984,425984=376832} 2023-11-05T08:32:46.179 INFO:tasks.rados.rados.0.smithi090.stdout:48: writing smithi09039252-48 from 0 to 425984 tid 1 2023-11-05T08:32:46.182 INFO:tasks.rados.rados.0.smithi090.stdout:48: writing smithi09039252-48 from 425984 to 802816 tid 2 2023-11-05T08:32:46.183 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-11-05T08:32:46.183 INFO:tasks.rados.rados.0.smithi090.stdout:36: finishing write tid 1 to smithi09039252-36 2023-11-05T08:32:46.183 INFO:tasks.rados.rados.0.smithi090.stdout:35: finishing write tid 3 to smithi09039252-35 2023-11-05T08:32:46.183 INFO:tasks.rados.rados.0.smithi090.stdout:35: oid 35 updating version 0 to 8 2023-11-05T08:32:46.183 INFO:tasks.rados.rados.0.smithi090.stdout:35: oid 35 version 8 is already newer than 7 2023-11-05T08:32:46.183 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 35 v 8 (ObjNum 34 snap 0 seq_num 34) dirty exists 2023-11-05T08:32:46.184 INFO:tasks.rados.rados.0.smithi090.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2023-11-05T08:32:46.184 INFO:tasks.rados.rados.0.smithi090.stdout:36: finishing write tid 2 to smithi09039252-36 2023-11-05T08:32:46.184 INFO:tasks.rados.rados.0.smithi090.stdout:36: finishing write tid 3 to smithi09039252-36 2023-11-05T08:32:46.184 INFO:tasks.rados.rados.0.smithi090.stdout:36: oid 36 updating version 0 to 6 2023-11-05T08:32:46.184 INFO:tasks.rados.rados.0.smithi090.stdout:36: oid 36 version 6 is already newer than 5 2023-11-05T08:32:46.185 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 36 v 6 (ObjNum 35 snap 0 seq_num 35) dirty exists 2023-11-05T08:32:46.185 INFO:tasks.rados.rados.0.smithi090.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2023-11-05T08:32:46.185 INFO:tasks.rados.rados.0.smithi090.stdout:35: done (15 left) 2023-11-05T08:32:46.185 INFO:tasks.rados.rados.0.smithi090.stdout:36: done (14 left) 2023-11-05T08:32:46.186 INFO:tasks.rados.rados.0.smithi090.stdout:49: write initial oid 49 2023-11-05T08:32:46.186 INFO:tasks.rados.rados.0.smithi090.stdout:49: seq_num 48 ranges {0=442368,442368=376832} 2023-11-05T08:32:46.186 INFO:tasks.rados.rados.0.smithi090.stdout:49: writing smithi09039252-49 from 0 to 442368 tid 1 2023-11-05T08:32:46.187 INFO:tasks.rados.rados.0.smithi090.stdout:49: writing smithi09039252-49 from 442368 to 819200 tid 2 2023-11-05T08:32:46.187 INFO:tasks.rados.rados.0.smithi090.stdout:50: write initial oid 50 2023-11-05T08:32:46.187 INFO:tasks.rados.rados.0.smithi090.stdout:50: seq_num 49 ranges {0=458752,458752=376832} 2023-11-05T08:32:46.190 INFO:tasks.rados.rados.0.smithi090.stdout:50: writing smithi09039252-50 from 0 to 458752 tid 1 2023-11-05T08:32:46.194 INFO:tasks.rados.rados.0.smithi090.stdout:50: writing smithi09039252-50 from 458752 to 835584 tid 2 2023-11-05T08:32:46.194 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-11-05T08:32:46.194 INFO:tasks.rados.rados.0.smithi090.stdout:37: finishing write tid 1 to smithi09039252-37 2023-11-05T08:32:46.194 INFO:tasks.rados.rados.0.smithi090.stdout:37: finishing write tid 2 to smithi09039252-37 2023-11-05T08:32:46.195 INFO:tasks.rados.rados.0.smithi090.stdout:37: finishing write tid 3 to smithi09039252-37 2023-11-05T08:32:46.195 INFO:tasks.rados.rados.0.smithi090.stdout:37: oid 37 updating version 0 to 7 2023-11-05T08:32:46.195 INFO:tasks.rados.rados.0.smithi090.stdout:37: oid 37 updating version 7 to 8 2023-11-05T08:32:46.195 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 37 v 8 (ObjNum 36 snap 0 seq_num 36) dirty exists 2023-11-05T08:32:46.195 INFO:tasks.rados.rados.0.smithi090.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2023-11-05T08:32:46.196 INFO:tasks.rados.rados.0.smithi090.stdout:38: finishing write tid 1 to smithi09039252-38 2023-11-05T08:32:46.196 INFO:tasks.rados.rados.0.smithi090.stdout:38: finishing write tid 2 to smithi09039252-38 2023-11-05T08:32:46.196 INFO:tasks.rados.rados.0.smithi090.stdout:38: finishing write tid 3 to smithi09039252-38 2023-11-05T08:32:46.196 INFO:tasks.rados.rados.0.smithi090.stdout:38: oid 38 updating version 0 to 8 2023-11-05T08:32:46.197 INFO:tasks.rados.rados.0.smithi090.stdout:38: oid 38 version 8 is already newer than 7 2023-11-05T08:32:46.197 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 38 v 8 (ObjNum 37 snap 0 seq_num 37) dirty exists 2023-11-05T08:32:46.197 INFO:tasks.rados.rados.0.smithi090.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2023-11-05T08:32:46.197 INFO:tasks.rados.rados.0.smithi090.stdout:39: finishing write tid 1 to smithi09039252-39 2023-11-05T08:32:46.197 INFO:tasks.rados.rados.0.smithi090.stdout:39: finishing write tid 2 to smithi09039252-39 2023-11-05T08:32:46.198 INFO:tasks.rados.rados.0.smithi090.stdout:39: finishing write tid 3 to smithi09039252-39 2023-11-05T08:32:46.198 INFO:tasks.rados.rados.0.smithi090.stdout:39: oid 39 updating version 0 to 5 2023-11-05T08:32:46.198 INFO:tasks.rados.rados.0.smithi090.stdout:39: oid 39 updating version 5 to 6 2023-11-05T08:32:46.198 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 39 v 6 (ObjNum 38 snap 0 seq_num 38) dirty exists 2023-11-05T08:32:46.198 INFO:tasks.rados.rados.0.smithi090.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2023-11-05T08:32:46.199 INFO:tasks.rados.rados.0.smithi090.stdout:40: finishing write tid 1 to smithi09039252-40 2023-11-05T08:32:46.199 INFO:tasks.rados.rados.0.smithi090.stdout:40: finishing write tid 2 to smithi09039252-40 2023-11-05T08:32:46.199 INFO:tasks.rados.rados.0.smithi090.stdout:40: finishing write tid 3 to smithi09039252-40 2023-11-05T08:32:46.199 INFO:tasks.rados.rados.0.smithi090.stdout:40: oid 40 updating version 0 to 2 2023-11-05T08:32:46.199 INFO:tasks.rados.rados.0.smithi090.stdout:40: oid 40 version 2 is already newer than 1 2023-11-05T08:32:46.200 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 40 v 2 (ObjNum 39 snap 0 seq_num 39) dirty exists 2023-11-05T08:32:46.200 INFO:tasks.rados.rados.0.smithi090.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2023-11-05T08:32:46.200 INFO:tasks.rados.rados.0.smithi090.stdout:41: finishing write tid 1 to smithi09039252-41 2023-11-05T08:32:46.200 INFO:tasks.rados.rados.0.smithi090.stdout:41: finishing write tid 2 to smithi09039252-41 2023-11-05T08:32:46.200 INFO:tasks.rados.rados.0.smithi090.stdout:41: finishing write tid 3 to smithi09039252-41 2023-11-05T08:32:46.201 INFO:tasks.rados.rados.0.smithi090.stdout:41: oid 41 updating version 0 to 9 2023-11-05T08:32:46.201 INFO:tasks.rados.rados.0.smithi090.stdout:41: oid 41 updating version 9 to 10 2023-11-05T08:32:46.201 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 41 v 10 (ObjNum 40 snap 0 seq_num 40) dirty exists 2023-11-05T08:32:46.201 INFO:tasks.rados.rados.0.smithi090.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2023-11-05T08:32:46.201 INFO:tasks.rados.rados.0.smithi090.stdout:42: finishing write tid 1 to smithi09039252-42 2023-11-05T08:32:46.202 INFO:tasks.rados.rados.0.smithi090.stdout:42: finishing write tid 2 to smithi09039252-42 2023-11-05T08:32:46.202 INFO:tasks.rados.rados.0.smithi090.stdout:42: finishing write tid 3 to smithi09039252-42 2023-11-05T08:32:46.202 INFO:tasks.rados.rados.0.smithi090.stdout:42: oid 42 updating version 0 to 6 2023-11-05T08:32:46.202 INFO:tasks.rados.rados.0.smithi090.stdout:42: oid 42 version 6 is already newer than 5 2023-11-05T08:32:46.202 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 42 v 6 (ObjNum 41 snap 0 seq_num 41) dirty exists 2023-11-05T08:32:46.203 INFO:tasks.rados.rados.0.smithi090.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2023-11-05T08:32:46.203 INFO:tasks.rados.rados.0.smithi090.stdout:44: finishing write tid 1 to smithi09039252-44 2023-11-05T08:32:46.203 INFO:tasks.rados.rados.0.smithi090.stdout:44: finishing write tid 2 to smithi09039252-44 2023-11-05T08:32:46.203 INFO:tasks.rados.rados.0.smithi090.stdout:44: finishing write tid 3 to smithi09039252-44 2023-11-05T08:32:46.203 INFO:tasks.rados.rados.0.smithi090.stdout:44: oid 44 updating version 0 to 10 2023-11-05T08:32:46.204 INFO:tasks.rados.rados.0.smithi090.stdout:44: oid 44 version 10 is already newer than 9 2023-11-05T08:32:46.204 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 44 v 10 (ObjNum 43 snap 0 seq_num 43) dirty exists 2023-11-05T08:32:46.204 INFO:tasks.rados.rados.0.smithi090.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2023-11-05T08:32:46.204 INFO:tasks.rados.rados.0.smithi090.stdout:45: finishing write tid 1 to smithi09039252-45 2023-11-05T08:32:46.204 INFO:tasks.rados.rados.0.smithi090.stdout:45: finishing write tid 2 to smithi09039252-45 2023-11-05T08:32:46.205 INFO:tasks.rados.rados.0.smithi090.stdout:46: finishing write tid 1 to smithi09039252-46 2023-11-05T08:32:46.205 INFO:tasks.rados.rados.0.smithi090.stdout:45: finishing write tid 3 to smithi09039252-45 2023-11-05T08:32:46.205 INFO:tasks.rados.rados.0.smithi090.stdout:45: oid 45 updating version 0 to 12 2023-11-05T08:32:46.205 INFO:tasks.rados.rados.0.smithi090.stdout:45: oid 45 version 12 is already newer than 11 2023-11-05T08:32:46.205 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 45 v 12 (ObjNum 44 snap 0 seq_num 44) dirty exists 2023-11-05T08:32:46.206 INFO:tasks.rados.rados.0.smithi090.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2023-11-05T08:32:46.206 INFO:tasks.rados.rados.0.smithi090.stdout:46: finishing write tid 2 to smithi09039252-46 2023-11-05T08:32:46.206 INFO:tasks.rados.rados.0.smithi090.stdout:46: finishing write tid 3 to smithi09039252-46 2023-11-05T08:32:46.206 INFO:tasks.rados.rados.0.smithi090.stdout:46: oid 46 updating version 0 to 8 2023-11-05T08:32:46.206 INFO:tasks.rados.rados.0.smithi090.stdout:46: oid 46 version 8 is already newer than 7 2023-11-05T08:32:46.207 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 46 v 8 (ObjNum 45 snap 0 seq_num 45) dirty exists 2023-11-05T08:32:46.207 INFO:tasks.rados.rados.0.smithi090.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2023-11-05T08:32:46.207 INFO:tasks.rados.rados.0.smithi090.stdout:47: finishing write tid 1 to smithi09039252-47 2023-11-05T08:32:46.207 INFO:tasks.rados.rados.0.smithi090.stdout:47: finishing write tid 2 to smithi09039252-47 2023-11-05T08:32:46.207 INFO:tasks.rados.rados.0.smithi090.stdout:47: finishing write tid 3 to smithi09039252-47 2023-11-05T08:32:46.207 INFO:tasks.rados.rados.0.smithi090.stdout:47: oid 47 updating version 0 to 5 2023-11-05T08:32:46.208 INFO:tasks.rados.rados.0.smithi090.stdout:47: oid 47 updating version 5 to 6 2023-11-05T08:32:46.208 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 47 v 6 (ObjNum 46 snap 0 seq_num 46) dirty exists 2023-11-05T08:32:46.208 INFO:tasks.rados.rados.0.smithi090.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2023-11-05T08:32:46.208 INFO:tasks.rados.rados.0.smithi090.stdout:48: finishing write tid 1 to smithi09039252-48 2023-11-05T08:32:46.208 INFO:tasks.rados.rados.0.smithi090.stdout:48: finishing write tid 2 to smithi09039252-48 2023-11-05T08:32:46.209 INFO:tasks.rados.rados.0.smithi090.stdout:48: finishing write tid 3 to smithi09039252-48 2023-11-05T08:32:46.209 INFO:tasks.rados.rados.0.smithi090.stdout:48: finishing write tid 4 to smithi09039252-48 2023-11-05T08:32:46.209 INFO:tasks.rados.rados.0.smithi090.stdout:48: oid 48 updating version 0 to 7 2023-11-05T08:32:46.209 INFO:tasks.rados.rados.0.smithi090.stdout:48: oid 48 version 7 is already newer than 5 2023-11-05T08:32:46.209 INFO:tasks.rados.rados.0.smithi090.stdout:48: oid 48 version 7 is already newer than 6 2023-11-05T08:32:46.210 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 48 v 7 (ObjNum 47 snap 0 seq_num 47) dirty exists 2023-11-05T08:32:46.210 INFO:tasks.rados.rados.0.smithi090.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2023-11-05T08:32:46.210 INFO:tasks.rados.rados.0.smithi090.stdout:49: finishing write tid 1 to smithi09039252-49 2023-11-05T08:32:46.210 INFO:tasks.rados.rados.0.smithi090.stdout:37: done (15 left) 2023-11-05T08:32:46.210 INFO:tasks.rados.rados.0.smithi090.stdout:38: done (14 left) 2023-11-05T08:32:46.211 INFO:tasks.rados.rados.0.smithi090.stdout:39: done (13 left) 2023-11-05T08:32:46.211 INFO:tasks.rados.rados.0.smithi090.stdout:40: done (12 left) 2023-11-05T08:32:46.211 INFO:tasks.rados.rados.0.smithi090.stdout:41: done (11 left) 2023-11-05T08:32:46.211 INFO:tasks.rados.rados.0.smithi090.stdout:42: done (10 left) 2023-11-05T08:32:46.211 INFO:tasks.rados.rados.0.smithi090.stdout:44: done (9 left) 2023-11-05T08:32:46.212 INFO:tasks.rados.rados.0.smithi090.stdout:45: done (8 left) 2023-11-05T08:32:46.212 INFO:tasks.rados.rados.0.smithi090.stdout:46: done (7 left) 2023-11-05T08:32:46.212 INFO:tasks.rados.rados.0.smithi090.stdout:47: done (6 left) 2023-11-05T08:32:46.212 INFO:tasks.rados.rados.0.smithi090.stdout:48: done (5 left) 2023-11-05T08:32:46.213 INFO:tasks.rados.rados.0.smithi090.stdout:51: copy_from oid 2 from oid 36 current snap is 0 2023-11-05T08:32:46.213 INFO:tasks.rados.rados.0.smithi090.stdout:52: rmattr oid 42 current snap is 0 2023-11-05T08:32:46.213 INFO:tasks.rados.rados.0.smithi090.stdout:52: done (6 left) 2023-11-05T08:32:46.213 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 13 current snap is 0 2023-11-05T08:32:46.213 INFO:tasks.rados.rados.0.smithi090.stdout:53: seq_num 50 ranges {212992=475136,688128=376832} 2023-11-05T08:32:46.215 INFO:tasks.rados.rados.0.smithi090.stdout:53: writing smithi09039252-13 from 212992 to 688128 tid 1 2023-11-05T08:32:46.215 INFO:tasks.rados.rados.0.smithi090.stdout:53: writing smithi09039252-13 from 688128 to 1064960 tid 2 2023-11-05T08:32:46.215 INFO:tasks.rados.rados.0.smithi090.stdout:54: read oid 5 snap -1 2023-11-05T08:32:46.215 INFO:tasks.rados.rados.0.smithi090.stdout:54: expect (ObjNum 4 snap 0 seq_num 4) 2023-11-05T08:32:46.215 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 47 current snap is 0 2023-11-05T08:32:46.216 INFO:tasks.rados.rados.0.smithi090.stdout:49: finishing write tid 2 to smithi09039252-49 2023-11-05T08:32:46.216 INFO:tasks.rados.rados.0.smithi090.stdout:55: seq_num 51 ranges {786432=491520,1277952=376832} 2023-11-05T08:32:46.216 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:46.216 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:46.218 INFO:tasks.rados.rados.0.smithi090.stdout:55: writing smithi09039252-47 from 786432 to 1277952 tid 1 2023-11-05T08:32:46.218 INFO:tasks.rados.rados.0.smithi090.stdout:55: writing smithi09039252-47 from 1277952 to 1654784 tid 2 2023-11-05T08:32:46.219 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 25 current snap is 0 2023-11-05T08:32:46.219 INFO:tasks.rados.rados.0.smithi090.stdout:56: seq_num 52 ranges {409600=507904,917504=376832} 2023-11-05T08:32:46.219 INFO:tasks.rados.rados.0.smithi090.stdout:56: writing smithi09039252-25 from 409600 to 917504 tid 1 2023-11-05T08:32:46.219 INFO:tasks.rados.rados.0.smithi090.stdout:56: writing smithi09039252-25 from 917504 to 1294336 tid 2 2023-11-05T08:32:46.220 INFO:tasks.rados.rados.0.smithi090.stdout:57: read oid 4 snap -1 2023-11-05T08:32:46.220 INFO:tasks.rados.rados.0.smithi090.stdout:57: expect (ObjNum 3 snap 0 seq_num 3) 2023-11-05T08:32:46.220 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 46 current snap is 0 2023-11-05T08:32:46.220 INFO:tasks.rados.rados.0.smithi090.stdout:49: finishing write tid 3 to smithi09039252-49 2023-11-05T08:32:46.221 INFO:tasks.rados.rados.0.smithi090.stdout:50: finishing write tid 1 to smithi09039252-50 2023-11-05T08:32:46.221 INFO:tasks.rados.rados.0.smithi090.stdout:50: finishing write tid 2 to smithi09039252-50 2023-11-05T08:32:46.221 INFO:tasks.rados.rados.0.smithi090.stdout:50: finishing write tid 3 to smithi09039252-50 2023-11-05T08:32:46.221 INFO:tasks.rados.rados.0.smithi090.stdout:50: finishing write tid 4 to smithi09039252-50 2023-11-05T08:32:46.222 INFO:tasks.rados.rados.0.smithi090.stdout:50: oid 50 updating version 0 to 3 2023-11-05T08:32:46.222 INFO:tasks.rados.rados.0.smithi090.stdout:50: oid 50 updating version 3 to 4 2023-11-05T08:32:46.222 INFO:tasks.rados.rados.0.smithi090.stdout:50: oid 50 updating version 4 to 5 2023-11-05T08:32:46.222 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 50 v 5 (ObjNum 49 snap 0 seq_num 49) dirty exists 2023-11-05T08:32:46.222 INFO:tasks.rados.rados.0.smithi090.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2023-11-05T08:32:46.223 INFO:tasks.rados.rados.0.smithi090.stdout:58: seq_num 53 ranges {770048=524288,1294336=376832} 2023-11-05T08:32:46.223 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:32:46.215+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:32:46.225 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-11-05T08:32:46.225 INFO:tasks.rados.rados.0.smithi090.stdout:58: writing smithi09039252-46 from 770048 to 1294336 tid 1 2023-11-05T08:32:46.228 INFO:tasks.rados.rados.0.smithi090.stdout:58: writing smithi09039252-46 from 1294336 to 1671168 tid 2 2023-11-05T08:32:46.228 INFO:tasks.rados.rados.0.smithi090.stdout:50: done (11 left) 2023-11-05T08:32:46.228 INFO:tasks.rados.rados.0.smithi090.stdout:59: read oid 33 snap -1 2023-11-05T08:32:46.229 INFO:tasks.rados.rados.0.smithi090.stdout:59: expect (ObjNum 32 snap 0 seq_num 32) 2023-11-05T08:32:46.229 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 11 current snap is 0 2023-11-05T08:32:46.229 INFO:tasks.rados.rados.0.smithi090.stdout:60: seq_num 54 ranges {180224=540672,720896=376832} 2023-11-05T08:32:46.233 INFO:tasks.rados.rados.0.smithi090.stdout:60: writing smithi09039252-11 from 180224 to 720896 tid 1 2023-11-05T08:32:46.238 INFO:tasks.rados.rados.0.smithi090.stdout:60: writing smithi09039252-11 from 720896 to 1097728 tid 2 2023-11-05T08:32:46.238 INFO:tasks.rados.rados.0.smithi090.stdout:61: rollback oid 23 current snap is 0 2023-11-05T08:32:46.238 INFO:tasks.rados.rados.0.smithi090.stdout:61: done (13 left) 2023-11-05T08:32:46.238 INFO:tasks.rados.rados.0.smithi090.stdout:62: copy_from oid 9 from oid 7 current snap is 0 2023-11-05T08:32:46.238 INFO:tasks.rados.rados.0.smithi090.stdout:55: finishing write tid 1 to smithi09039252-47 2023-11-05T08:32:46.239 INFO:tasks.rados.rados.0.smithi090.stdout:55: finishing write tid 2 to smithi09039252-47 2023-11-05T08:32:46.239 INFO:tasks.rados.rados.0.smithi090.stdout:55: finishing write tid 3 to smithi09039252-47 2023-11-05T08:32:46.239 INFO:tasks.rados.rados.0.smithi090.stdout:55: finishing write tid 4 to smithi09039252-47 2023-11-05T08:32:46.239 INFO:tasks.rados.rados.0.smithi090.stdout:55: oid 47 updating version 0 to 7 2023-11-05T08:32:46.239 INFO:tasks.rados.rados.0.smithi090.stdout:55: oid 47 updating version 7 to 9 2023-11-05T08:32:46.240 INFO:tasks.rados.rados.0.smithi090.stdout:55: oid 47 version 9 is already newer than 8 2023-11-05T08:32:46.240 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 47 v 9 (ObjNum 51 snap 0 seq_num 51) dirty exists 2023-11-05T08:32:46.240 INFO:tasks.rados.rados.0.smithi090.stdout:55: left oid 47 (ObjNum 51 snap 0 seq_num 51) 2023-11-05T08:32:46.240 INFO:tasks.rados.rados.0.smithi090.stdout:49: finishing write tid 4 to smithi09039252-49 2023-11-05T08:32:46.240 INFO:tasks.rados.rados.0.smithi090.stdout:49: oid 49 updating version 0 to 13 2023-11-05T08:32:46.241 INFO:tasks.rados.rados.0.smithi090.stdout:49: oid 49 updating version 13 to 14 2023-11-05T08:32:46.241 INFO:tasks.rados.rados.0.smithi090.stdout:49: oid 49 updating version 14 to 15 2023-11-05T08:32:46.241 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 49 v 15 (ObjNum 48 snap 0 seq_num 48) dirty exists 2023-11-05T08:32:46.241 INFO:tasks.rados.rados.0.smithi090.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2023-11-05T08:32:46.241 INFO:tasks.rados.rados.0.smithi090.stdout:57: expect (ObjNum 3 snap 0 seq_num 3) 2023-11-05T08:32:46.243 INFO:tasks.rados.rados.0.smithi090.stdout:56: finishing write tid 1 to smithi09039252-25 2023-11-05T08:32:46.243 INFO:tasks.rados.rados.0.smithi090.stdout:54: expect (ObjNum 4 snap 0 seq_num 4) 2023-11-05T08:32:46.243 INFO:tasks.rados.rados.0.smithi090.stdout:53: finishing write tid 1 to smithi09039252-13 2023-11-05T08:32:46.243 INFO:tasks.rados.rados.0.smithi090.stdout:53: finishing write tid 2 to smithi09039252-13 2023-11-05T08:32:46.244 INFO:tasks.rados.rados.0.smithi090.stdout:56: finishing write tid 2 to smithi09039252-25 2023-11-05T08:32:46.244 INFO:tasks.rados.rados.0.smithi090.stdout:49: done (13 left) 2023-11-05T08:32:46.244 INFO:tasks.rados.rados.0.smithi090.stdout:54: done (12 left) 2023-11-05T08:32:46.244 INFO:tasks.rados.rados.0.smithi090.stdout:55: done (11 left) 2023-11-05T08:32:46.244 INFO:tasks.rados.rados.0.smithi090.stdout:57: done (10 left) 2023-11-05T08:32:46.245 INFO:tasks.rados.rados.0.smithi090.stdout:63: rmattr oid 49 current snap is 0 2023-11-05T08:32:46.245 INFO:tasks.rados.rados.0.smithi090.stdout:63: done (10 left) 2023-11-05T08:32:46.245 INFO:tasks.rados.rados.0.smithi090.stdout:64: copy_from oid 5 from oid 41 current snap is 0 2023-11-05T08:32:46.245 INFO:tasks.rados.rados.0.smithi090.stdout:53: finishing write tid 3 to smithi09039252-13 2023-11-05T08:32:46.245 INFO:tasks.rados.rados.0.smithi090.stdout:53: finishing write tid 4 to smithi09039252-13 2023-11-05T08:32:46.245 INFO:tasks.rados.rados.0.smithi090.stdout:53: oid 13 updating version 0 to 17 2023-11-05T08:32:46.246 INFO:tasks.rados.rados.0.smithi090.stdout:53: oid 13 updating version 17 to 18 2023-11-05T08:32:46.246 INFO:tasks.rados.rados.0.smithi090.stdout:53: oid 13 version 18 is already newer than 16 2023-11-05T08:32:46.246 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 13 v 18 (ObjNum 50 snap 0 seq_num 50) dirty exists 2023-11-05T08:32:46.246 INFO:tasks.rados.rados.0.smithi090.stdout:53: left oid 13 (ObjNum 50 snap 0 seq_num 50) 2023-11-05T08:32:46.246 INFO:tasks.rados.rados.0.smithi090.stdout:51: finishing copy_from racing read to smithi09039252-2 2023-11-05T08:32:46.247 INFO:tasks.rados.rados.0.smithi090.stdout:56: finishing write tid 3 to smithi09039252-25 2023-11-05T08:32:46.247 INFO:tasks.rados.rados.0.smithi090.stdout:56: finishing write tid 4 to smithi09039252-25 2023-11-05T08:32:46.247 INFO:tasks.rados.rados.0.smithi090.stdout:56: oid 25 updating version 0 to 14 2023-11-05T08:32:46.247 INFO:tasks.rados.rados.0.smithi090.stdout:56: oid 25 updating version 14 to 15 2023-11-05T08:32:46.247 INFO:tasks.rados.rados.0.smithi090.stdout:56: oid 25 version 15 is already newer than 13 2023-11-05T08:32:46.248 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 25 v 15 (ObjNum 52 snap 0 seq_num 52) dirty exists 2023-11-05T08:32:46.248 INFO:tasks.rados.rados.0.smithi090.stdout:56: left oid 25 (ObjNum 52 snap 0 seq_num 52) 2023-11-05T08:32:46.248 INFO:tasks.rados.rados.0.smithi090.stdout:58: finishing write tid 1 to smithi09039252-46 2023-11-05T08:32:46.248 INFO:tasks.rados.rados.0.smithi090.stdout:53: done (10 left) 2023-11-05T08:32:46.248 INFO:tasks.rados.rados.0.smithi090.stdout:56: done (9 left) 2023-11-05T08:32:46.249 INFO:tasks.rados.rados.0.smithi090.stdout:65: delete oid 20 current snap is 0 2023-11-05T08:32:46.249 INFO:tasks.rados.rados.0.smithi090.stdout:59: expect (ObjNum 32 snap 0 seq_num 32) 2023-11-05T08:32:46.255 INFO:tasks.rados.rados.0.smithi090.stdout:58: finishing write tid 2 to smithi09039252-46 2023-11-05T08:32:46.256 INFO:tasks.rados.rados.0.smithi090.stdout:51: finishing copy_from to smithi09039252-2 2023-11-05T08:32:46.256 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 2 v 6 (ObjNum 35 snap 0 seq_num 35) dirty exists 2023-11-05T08:32:46.256 INFO:tasks.rados.rados.0.smithi090.stdout:51: done (9 left) 2023-11-05T08:32:46.256 INFO:tasks.rados.rados.0.smithi090.stdout:59: done (8 left) 2023-11-05T08:32:46.256 INFO:tasks.rados.rados.0.smithi090.stdout:65: done (7 left) 2023-11-05T08:32:46.256 INFO:tasks.rados.rados.0.smithi090.stdout:66: snap_create 2023-11-05T08:32:46.257 INFO:tasks.rados.rados.0.smithi090.stdout:58: finishing write tid 3 to smithi09039252-46 2023-11-05T08:32:46.257 INFO:tasks.rados.rados.0.smithi090.stdout:60: finishing write tid 1 to smithi09039252-11 2023-11-05T08:32:46.257 INFO:tasks.rados.rados.0.smithi090.stdout:60: finishing write tid 2 to smithi09039252-11 2023-11-05T08:32:46.257 INFO:tasks.rados.rados.0.smithi090.stdout:60: finishing write tid 3 to smithi09039252-11 2023-11-05T08:32:46.257 INFO:tasks.rados.rados.0.smithi090.stdout:60: finishing write tid 4 to smithi09039252-11 2023-11-05T08:32:46.258 INFO:tasks.rados.rados.0.smithi090.stdout:60: oid 11 updating version 0 to 9 2023-11-05T08:32:46.258 INFO:tasks.rados.rados.0.smithi090.stdout:60: oid 11 updating version 9 to 11 2023-11-05T08:32:46.258 INFO:tasks.rados.rados.0.smithi090.stdout:60: oid 11 version 11 is already newer than 10 2023-11-05T08:32:46.258 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 11 v 11 (ObjNum 54 snap 0 seq_num 54) dirty exists 2023-11-05T08:32:46.258 INFO:tasks.rados.rados.0.smithi090.stdout:60: left oid 11 (ObjNum 54 snap 0 seq_num 54) 2023-11-05T08:32:46.258 INFO:tasks.rados.rados.0.smithi090.stdout:64: finishing copy_from racing read to smithi09039252-5 2023-11-05T08:32:46.259 INFO:tasks.rados.rados.0.smithi090.stdout:64: finishing copy_from to smithi09039252-5 2023-11-05T08:32:46.259 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 5 v 10 (ObjNum 40 snap 0 seq_num 40) dirty exists 2023-11-05T08:32:46.259 INFO:tasks.rados.rados.0.smithi090.stdout:58: finishing write tid 4 to smithi09039252-46 2023-11-05T08:32:46.259 INFO:tasks.rados.rados.0.smithi090.stdout:58: oid 46 updating version 0 to 9 2023-11-05T08:32:46.259 INFO:tasks.rados.rados.0.smithi090.stdout:58: oid 46 updating version 9 to 11 2023-11-05T08:32:46.260 INFO:tasks.rados.rados.0.smithi090.stdout:58: oid 46 version 11 is already newer than 10 2023-11-05T08:32:46.260 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 46 v 11 (ObjNum 53 snap 0 seq_num 53) dirty exists 2023-11-05T08:32:46.260 INFO:tasks.rados.rados.0.smithi090.stdout:58: left oid 46 (ObjNum 53 snap 0 seq_num 53) 2023-11-05T08:32:46.260 INFO:tasks.rados.rados.0.smithi090.stdout:62: finishing copy_from racing read to smithi09039252-9 2023-11-05T08:32:46.260 INFO:tasks.rados.rados.0.smithi090.stdout:62: finishing copy_from to smithi09039252-9 2023-11-05T08:32:46.261 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 9 v 2 (ObjNum 6 snap 0 seq_num 6) dirty exists 2023-11-05T08:32:46.314 INFO:tasks.rados.rados.0.smithi090.stdout:31: finishing write tid 1 to smithi09039252-31 2023-11-05T08:32:46.315 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:32:46.313+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:32:46.316 INFO:tasks.rados.rados.0.smithi090.stdout:31: finishing write tid 2 to smithi09039252-31 2023-11-05T08:32:46.319 INFO:tasks.rados.rados.0.smithi090.stdout:31: finishing write tid 3 to smithi09039252-31 2023-11-05T08:32:46.319 INFO:tasks.rados.rados.0.smithi090.stdout:31: oid 31 updating version 0 to 1 2023-11-05T08:32:46.319 INFO:tasks.rados.rados.0.smithi090.stdout:31: oid 31 updating version 1 to 2 2023-11-05T08:32:46.319 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 31 v 2 (ObjNum 30 snap 0 seq_num 30) dirty exists 2023-11-05T08:32:46.320 INFO:tasks.rados.rados.0.smithi090.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2023-11-05T08:32:46.341 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:46.341 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:46.354 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-11-05T08:32:46.416 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:32:46.415+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:32:46.516 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:46.517 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:46.517 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:32:46.513+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:32:46.530 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-11-05T08:32:46.618 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:32:46.616+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:32:46.649 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:46.649 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:46.663 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2023-11-05T08:32:46.719 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:32:46.717+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:32:46.785 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:46.785 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:46.799 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2023-11-05T08:32:46.820 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:32:46.816+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:32:46.914 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:46.915 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:46.920 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:32:46.917+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:32:46.929 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2023-11-05T08:32:47.015 INFO:teuthology.orchestra.run.smithi002.stderr:noscrub is unset 2023-11-05T08:32:47.021 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:32:47.016+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:32:47.023 INFO:tasks.rados.rados.0.smithi090.stdout:31: done (7 left) 2023-11-05T08:32:47.023 INFO:tasks.rados.rados.0.smithi090.stdout:58: done (6 left) 2023-11-05T08:32:47.023 INFO:tasks.rados.rados.0.smithi090.stdout:60: done (5 left) 2023-11-05T08:32:47.024 INFO:tasks.rados.rados.0.smithi090.stdout:62: done (4 left) 2023-11-05T08:32:47.024 INFO:tasks.rados.rados.0.smithi090.stdout:64: done (3 left) 2023-11-05T08:32:47.024 INFO:tasks.rados.rados.0.smithi090.stdout:66: done (2 left) 2023-11-05T08:32:47.024 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 20 current snap is 1 2023-11-05T08:32:47.024 INFO:tasks.rados.rados.0.smithi090.stdout:67: seq_num 56 ranges {0=573440,573440=376832} 2023-11-05T08:32:47.026 INFO:tasks.rados.rados.0.smithi090.stdout:67: writing smithi09039252-20 from 0 to 573440 tid 1 2023-11-05T08:32:47.029 INFO:tasks.rados.rados.0.smithi090.stdout:67: writing smithi09039252-20 from 573440 to 950272 tid 2 2023-11-05T08:32:47.029 INFO:tasks.rados.rados.0.smithi090.stdout:68: read oid 9 snap -1 2023-11-05T08:32:47.029 INFO:tasks.rados.rados.0.smithi090.stdout:68: expect (ObjNum 6 snap 0 seq_num 6) 2023-11-05T08:32:47.029 INFO:tasks.rados.rados.0.smithi090.stdout:69: delete oid 31 current snap is 1 2023-11-05T08:32:47.042 INFO:tasks.rados.rados.0.smithi090.stdout:68: expect (ObjNum 6 snap 0 seq_num 6) 2023-11-05T08:32:47.047 INFO:tasks.rados.rados.0.smithi090.stdout:67: finishing write tid 1 to smithi09039252-20 2023-11-05T08:32:47.047 INFO:tasks.rados.rados.0.smithi090.stdout:68: done (4 left) 2023-11-05T08:32:47.048 INFO:tasks.rados.rados.0.smithi090.stdout:69: done (3 left) 2023-11-05T08:32:47.048 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 12 current snap is 1 2023-11-05T08:32:47.048 INFO:tasks.rados.rados.0.smithi090.stdout:70: seq_num 58 ranges {196608=606208,802816=376832} 2023-11-05T08:32:47.054 INFO:tasks.rados.rados.0.smithi090.stdout:70: writing smithi09039252-12 from 196608 to 802816 tid 1 2023-11-05T08:32:47.057 INFO:tasks.rados.rados.0.smithi090.stdout:70: writing smithi09039252-12 from 802816 to 1179648 tid 2 2023-11-05T08:32:47.057 INFO:tasks.rados.rados.0.smithi090.stdout:71: read oid 36 snap -1 2023-11-05T08:32:47.057 INFO:tasks.rados.rados.0.smithi090.stdout:71: expect (ObjNum 35 snap 0 seq_num 35) 2023-11-05T08:32:47.058 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 40 current snap is 1 2023-11-05T08:32:47.058 INFO:tasks.rados.rados.0.smithi090.stdout:67: finishing write tid 2 to smithi09039252-20 2023-11-05T08:32:47.058 INFO:tasks.rados.rados.0.smithi090.stdout:67: finishing write tid 3 to smithi09039252-20 2023-11-05T08:32:47.058 INFO:tasks.rados.rados.0.smithi090.stdout:72: seq_num 59 ranges {671744=622592,1294336=376832} 2023-11-05T08:32:47.069 INFO:tasks.rados.rados.0.smithi090.stdout:72: writing smithi09039252-40 from 671744 to 1294336 tid 1 2023-11-05T08:32:47.073 INFO:tasks.rados.rados.0.smithi090.stdout:72: writing smithi09039252-40 from 1294336 to 1671168 tid 2 2023-11-05T08:32:47.073 INFO:tasks.rados.rados.0.smithi090.stdout:73: setattr oid 28 current snap is 1 2023-11-05T08:32:47.078 INFO:tasks.rados.rados.0.smithi090.stdout:67: finishing write tid 4 to smithi09039252-20 2023-11-05T08:32:47.078 INFO:tasks.rados.rados.0.smithi090.stdout:67: oid 20 updating version 0 to 11 2023-11-05T08:32:47.079 INFO:tasks.rados.rados.0.smithi090.stdout:67: oid 20 updating version 11 to 12 2023-11-05T08:32:47.079 INFO:tasks.rados.rados.0.smithi090.stdout:67: oid 20 version 12 is already newer than 10 2023-11-05T08:32:47.079 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 20 v 12 (ObjNum 56 snap 1 seq_num 56) dirty exists 2023-11-05T08:32:47.079 INFO:tasks.rados.rados.0.smithi090.stdout:67: left oid 20 (ObjNum 56 snap 1 seq_num 56) 2023-11-05T08:32:47.079 INFO:tasks.rados.rados.0.smithi090.stdout:71: expect (ObjNum 35 snap 0 seq_num 35) 2023-11-05T08:32:47.107 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:47.107 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:47.113 INFO:tasks.rados.rados.0.smithi090.stdout:70: finishing write tid 1 to smithi09039252-12 2023-11-05T08:32:47.113 INFO:tasks.rados.rados.0.smithi090.stdout:70: finishing write tid 2 to smithi09039252-12 2023-11-05T08:32:47.113 INFO:tasks.rados.rados.0.smithi090.stdout:15: finishing write tid 1 to smithi09039252-15 2023-11-05T08:32:47.114 INFO:tasks.rados.rados.0.smithi090.stdout:70: finishing write tid 3 to smithi09039252-12 2023-11-05T08:32:47.114 INFO:tasks.rados.rados.0.smithi090.stdout:15: finishing write tid 2 to smithi09039252-15 2023-11-05T08:32:47.114 INFO:tasks.rados.rados.0.smithi090.stdout:43: finishing write tid 1 to smithi09039252-43 2023-11-05T08:32:47.114 INFO:tasks.rados.rados.0.smithi090.stdout:70: finishing write tid 4 to smithi09039252-12 2023-11-05T08:32:47.114 INFO:tasks.rados.rados.0.smithi090.stdout:70: oid 12 updating version 0 to 18 2023-11-05T08:32:47.115 INFO:tasks.rados.rados.0.smithi090.stdout:70: oid 12 updating version 18 to 19 2023-11-05T08:32:47.115 INFO:tasks.rados.rados.0.smithi090.stdout:70: oid 12 version 19 is already newer than 17 2023-11-05T08:32:47.115 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 12 v 19 (ObjNum 58 snap 1 seq_num 58) dirty exists 2023-11-05T08:32:47.115 INFO:tasks.rados.rados.0.smithi090.stdout:70: left oid 12 (ObjNum 58 snap 1 seq_num 58) 2023-11-05T08:32:47.115 INFO:tasks.rados.rados.0.smithi090.stdout:15: finishing write tid 3 to smithi09039252-15 2023-11-05T08:32:47.116 INFO:tasks.rados.rados.0.smithi090.stdout:15: oid 15 updating version 0 to 2 2023-11-05T08:32:47.116 INFO:tasks.rados.rados.0.smithi090.stdout:15: oid 15 version 2 is already newer than 1 2023-11-05T08:32:47.116 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 15 v 2 (ObjNum 14 snap 0 seq_num 14) dirty exists 2023-11-05T08:32:47.116 INFO:tasks.rados.rados.0.smithi090.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2023-11-05T08:32:47.116 INFO:tasks.rados.rados.0.smithi090.stdout:43: finishing write tid 2 to smithi09039252-43 2023-11-05T08:32:47.116 INFO:tasks.rados.rados.0.smithi090.stdout:72: finishing write tid 1 to smithi09039252-40 2023-11-05T08:32:47.117 INFO:tasks.rados.rados.0.smithi090.stdout:72: finishing write tid 2 to smithi09039252-40 2023-11-05T08:32:47.117 INFO:tasks.rados.rados.0.smithi090.stdout:72: finishing write tid 3 to smithi09039252-40 2023-11-05T08:32:47.117 INFO:tasks.rados.rados.0.smithi090.stdout:72: finishing write tid 4 to smithi09039252-40 2023-11-05T08:32:47.117 INFO:tasks.rados.rados.0.smithi090.stdout:72: oid 40 updating version 0 to 6 2023-11-05T08:32:47.117 INFO:tasks.rados.rados.0.smithi090.stdout:72: oid 40 version 6 is already newer than 4 2023-11-05T08:32:47.118 INFO:tasks.rados.rados.0.smithi090.stdout:72: oid 40 version 6 is already newer than 5 2023-11-05T08:32:47.118 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 40 v 6 (ObjNum 59 snap 1 seq_num 59) dirty exists 2023-11-05T08:32:47.118 INFO:tasks.rados.rados.0.smithi090.stdout:72: left oid 40 (ObjNum 59 snap 1 seq_num 59) 2023-11-05T08:32:47.118 INFO:tasks.rados.rados.0.smithi090.stdout:43: finishing write tid 3 to smithi09039252-43 2023-11-05T08:32:47.119 INFO:tasks.rados.rados.0.smithi090.stdout:43: oid 43 updating version 0 to 4 2023-11-05T08:32:47.119 INFO:tasks.rados.rados.0.smithi090.stdout:43: oid 43 version 4 is already newer than 3 2023-11-05T08:32:47.119 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 43 v 4 (ObjNum 42 snap 0 seq_num 42) dirty exists 2023-11-05T08:32:47.119 INFO:tasks.rados.rados.0.smithi090.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2023-11-05T08:32:47.119 INFO:tasks.rados.rados.0.smithi090.stdout:15: done (6 left) 2023-11-05T08:32:47.120 INFO:tasks.rados.rados.0.smithi090.stdout:43: done (5 left) 2023-11-05T08:32:47.120 INFO:tasks.rados.rados.0.smithi090.stdout:67: done (4 left) 2023-11-05T08:32:47.120 INFO:tasks.rados.rados.0.smithi090.stdout:70: done (3 left) 2023-11-05T08:32:47.120 INFO:tasks.rados.rados.0.smithi090.stdout:71: done (2 left) 2023-11-05T08:32:47.120 INFO:tasks.rados.rados.0.smithi090.stdout:72: done (1 left) 2023-11-05T08:32:47.121 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 23 current snap is 1 2023-11-05T08:32:47.121 INFO:tasks.rados.rados.0.smithi090.stdout:74: seq_num 60 ranges {376832=638976,1015808=376832} 2023-11-05T08:32:47.122 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2023-11-05T08:32:47.123 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:32:47.117+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:32:47.123 INFO:tasks.rados.rados.0.smithi090.stdout:74: writing smithi09039252-23 from 376832 to 1015808 tid 1 2023-11-05T08:32:47.127 INFO:tasks.rados.rados.0.smithi090.stdout:74: writing smithi09039252-23 from 1015808 to 1392640 tid 2 2023-11-05T08:32:47.127 INFO:tasks.rados.rados.0.smithi090.stdout:75: copy_from oid 31 from oid 40 current snap is 1 2023-11-05T08:32:47.128 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 28 v 13 (ObjNum 27 snap 0 seq_num 27) dirty exists 2023-11-05T08:32:47.128 INFO:tasks.rados.rados.0.smithi090.stdout:73: done (2 left) 2023-11-05T08:32:47.128 INFO:tasks.rados.rados.0.smithi090.stdout:76: snap_create 2023-11-05T08:32:47.134 INFO:tasks.rados.rados.0.smithi090.stdout:74: finishing write tid 1 to smithi09039252-23 2023-11-05T08:32:47.134 INFO:tasks.rados.rados.0.smithi090.stdout:74: finishing write tid 2 to smithi09039252-23 2023-11-05T08:32:47.134 INFO:tasks.rados.rados.0.smithi090.stdout:74: finishing write tid 3 to smithi09039252-23 2023-11-05T08:32:47.135 INFO:tasks.rados.rados.0.smithi090.stdout:74: finishing write tid 4 to smithi09039252-23 2023-11-05T08:32:47.135 INFO:tasks.rados.rados.0.smithi090.stdout:74: oid 23 updating version 0 to 8 2023-11-05T08:32:47.136 INFO:tasks.rados.rados.0.smithi090.stdout:74: oid 23 version 8 is already newer than 7 2023-11-05T08:32:47.136 INFO:tasks.rados.rados.0.smithi090.stdout:74: oid 23 version 8 is already newer than 6 2023-11-05T08:32:47.136 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 23 v 8 (ObjNum 60 snap 1 seq_num 60) dirty exists 2023-11-05T08:32:47.136 INFO:tasks.rados.rados.0.smithi090.stdout:74: left oid 23 (ObjNum 60 snap 1 seq_num 60) 2023-11-05T08:32:47.143 INFO:tasks.rados.rados.0.smithi090.stdout:75: finishing copy_from racing read to smithi09039252-31 2023-11-05T08:32:47.147 INFO:tasks.rados.rados.0.smithi090.stdout:75: finishing copy_from to smithi09039252-31 2023-11-05T08:32:47.148 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 31 v 6 (ObjNum 59 snap 1 seq_num 59) dirty exists 2023-11-05T08:32:47.223 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:32:47.220+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:32:47.252 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:47.252 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:47.266 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2023-11-05T08:32:47.324 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:32:47.324+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:32:47.425 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:32:47.421+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:32:47.432 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:47.433 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:47.447 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2023-11-05T08:32:47.526 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:32:47.523+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:32:47.562 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:47.562 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:47.576 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-11-05T08:32:47.627 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:32:47.627+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:32:47.713 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:47.714 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:47.718 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-05T08:32:47.727 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:32:47.725+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:32:47.730 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-11-05T08:32:47.828 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:32:47.827+0000 7fdc5cd58640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:32:47.875 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0: {} 2023-11-05T08:32:47.876 INFO:teuthology.orchestra.run.smithi002.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-11-05T08:32:47.910 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1: {} 2023-11-05T08:32:47.910 INFO:teuthology.orchestra.run.smithi002.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-11-05T08:32:47.918 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:32:47.918 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [], 2023-11-05T08:32:47.918 INFO:teuthology.orchestra.run.smithi002.stdout: "complaint_time": 30, 2023-11-05T08:32:47.918 INFO:teuthology.orchestra.run.smithi002.stdout: "num_blocked_ops": 0 2023-11-05T08:32:47.919 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:32:47.929 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:32:47.925+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:32:47.938 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-11-05T08:32:47.943 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2: {} 2023-11-05T08:32:47.943 INFO:teuthology.orchestra.run.smithi002.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-11-05T08:32:47.963 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3: {} 2023-11-05T08:32:47.963 INFO:teuthology.orchestra.run.smithi002.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-11-05T08:32:48.023 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4: {} 2023-11-05T08:32:48.023 INFO:teuthology.orchestra.run.smithi002.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-11-05T08:32:48.030 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-11-05T08:32:48.029+0000 7f966619d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35976) UID: 0 2023-11-05T08:32:48.050 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5: {} 2023-11-05T08:32:48.050 INFO:teuthology.orchestra.run.smithi002.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-11-05T08:32:48.051 INFO:tasks.rados.rados.0.smithi090.stdout:74: done (2 left) 2023-11-05T08:32:48.052 INFO:tasks.rados.rados.0.smithi090.stdout:75: done (1 left) 2023-11-05T08:32:48.052 INFO:tasks.rados.rados.0.smithi090.stdout:76: done (0 left) 2023-11-05T08:32:48.052 INFO:tasks.rados.rados.0.smithi090.stdout:77: rollback oid 21 current snap is 2 2023-11-05T08:32:48.052 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 21 to 1 2023-11-05T08:32:48.052 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 47 current snap is 2 2023-11-05T08:32:48.052 INFO:tasks.rados.rados.0.smithi090.stdout:78: seq_num 61 ranges {1654784=655360,2310144=376832} 2023-11-05T08:32:48.075 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:32:48.076 INFO:teuthology.orchestra.run.smithi002.stdout: "size": 20, 2023-11-05T08:32:48.076 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 600, 2023-11-05T08:32:48.076 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [] 2023-11-05T08:32:48.076 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:32:48.077 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6: {} 2023-11-05T08:32:48.078 INFO:teuthology.orchestra.run.smithi002.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-11-05T08:32:48.081 INFO:tasks.rados.rados.0.smithi090.stdout:78: writing smithi09039252-47 from 1654784 to 2310144 tid 1 2023-11-05T08:32:48.087 INFO:tasks.rados.rados.0.smithi090.stdout:78: writing smithi09039252-47 from 2310144 to 2686976 tid 2 2023-11-05T08:32:48.087 INFO:tasks.rados.rados.0.smithi090.stdout:79: rollback oid 31 current snap is 2 2023-11-05T08:32:48.087 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 31 to 1 2023-11-05T08:32:48.087 INFO:tasks.rados.rados.0.smithi090.stdout:77: finishing rollback tid 0 to smithi09039252-21 2023-11-05T08:32:48.087 INFO:tasks.rados.rados.0.smithi090.stdout:77: finishing rollback tid 1 to smithi09039252-21 2023-11-05T08:32:48.088 INFO:tasks.rados.rados.0.smithi090.stdout:77: finishing rollback tid 2 to smithi09039252-21 2023-11-05T08:32:48.088 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 21 v 10 (ObjNum 20 snap 0 seq_num 20) dirty exists 2023-11-05T08:32:48.088 INFO:tasks.rados.rados.0.smithi090.stdout:77: done (2 left) 2023-11-05T08:32:48.088 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 46 current snap is 2 2023-11-05T08:32:48.088 INFO:tasks.rados.rados.0.smithi090.stdout:80: seq_num 62 ranges {1671168=671744,2342912=376832} 2023-11-05T08:32:48.093 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-11-05T08:32:48.112 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7: {} 2023-11-05T08:32:48.113 INFO:teuthology.orchestra.run.smithi002.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-11-05T08:32:48.115 INFO:tasks.rados.rados.0.smithi090.stdout:80: writing smithi09039252-46 from 1671168 to 2342912 tid 1 2023-11-05T08:32:48.118 INFO:tasks.rados.rados.0.smithi090.stdout:80: writing smithi09039252-46 from 2342912 to 2719744 tid 2 2023-11-05T08:32:48.119 INFO:tasks.rados.rados.0.smithi090.stdout:81: snap_remove snap 0 2023-11-05T08:32:48.130 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:32:48.129+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:32:48.135 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8: {} 2023-11-05T08:32:48.136 INFO:teuthology.orchestra.run.smithi002.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-11-05T08:32:48.159 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9: {} 2023-11-05T08:32:48.159 INFO:teuthology.orchestra.run.smithi002.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-11-05T08:32:48.182 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10: {} 2023-11-05T08:32:48.182 INFO:teuthology.orchestra.run.smithi002.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-11-05T08:32:48.206 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11: {} 2023-11-05T08:32:48.207 INFO:teuthology.orchestra.run.smithi002.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-11-05T08:32:48.227 INFO:teuthology.orchestra.run.smithi002.stdout:osd.12: {} 2023-11-05T08:32:48.228 INFO:teuthology.orchestra.run.smithi002.stderr:osd.12: osd_enable_op_tracker = 'true' 2023-11-05T08:32:48.232 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:32:48.229+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:32:48.233 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:32:48.234 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [], 2023-11-05T08:32:48.234 INFO:teuthology.orchestra.run.smithi002.stdout: "num_ops": 0 2023-11-05T08:32:48.234 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:32:48.248 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-11-05T08:32:48.249 INFO:teuthology.orchestra.run.smithi002.stdout:osd.13: {} 2023-11-05T08:32:48.250 INFO:teuthology.orchestra.run.smithi002.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-11-05T08:32:48.267 INFO:teuthology.orchestra.run.smithi002.stdout:osd.14: {} 2023-11-05T08:32:48.268 INFO:teuthology.orchestra.run.smithi002.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-11-05T08:32:48.288 INFO:teuthology.orchestra.run.smithi002.stdout:osd.15: {} 2023-11-05T08:32:48.288 INFO:teuthology.orchestra.run.smithi002.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-11-05T08:32:48.333 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:32:48.329+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:32:48.392 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:32:48.393 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [], 2023-11-05T08:32:48.393 INFO:teuthology.orchestra.run.smithi002.stdout: "complaint_time": 30, 2023-11-05T08:32:48.393 INFO:teuthology.orchestra.run.smithi002.stdout: "num_blocked_ops": 0 2023-11-05T08:32:48.393 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:32:48.409 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-11-05T08:32:48.434 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-11-05T08:32:48.429+0000 7f966619d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35976) UID: 0 2023-11-05T08:32:48.535 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:32:48.535+0000 7fdc5cd58640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:32:48.536 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:32:48.536 INFO:teuthology.orchestra.run.smithi002.stdout: "size": 20, 2023-11-05T08:32:48.536 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 600, 2023-11-05T08:32:48.536 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [ 2023-11-05T08:32:48.536 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.537 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.3s2 35/30 ECSubWrite(tid=179, reqid=client.4691.0:218, at_version=35'8, trim_to=0'0, roll_forward_to=33'6))", 2023-11-05T08:32:48.537 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:48.056203+0000", 2023-11-05T08:32:48.537 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 0.47981526899999999, 2023-11-05T08:32:48.537 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0023062460000000001, 2023-11-05T08:32:48.538 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:32:48.538 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:32:48.538 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:32:48.538 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.538 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:32:48.539 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.056203+0000", 2023-11-05T08:32:48.539 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:32:48.539 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.539 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.539 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:32:48.540 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.056203+0000", 2023-11-05T08:32:48.540 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:32:48.540 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.540 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.540 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:32:48.541 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.056204+0000", 2023-11-05T08:32:48.541 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.79e-07 2023-11-05T08:32:48.541 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.541 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.541 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:32:48.542 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.056237+0000", 2023-11-05T08:32:48.542 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.3176999999999998e-05 2023-11-05T08:32:48.542 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.542 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.542 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:32:48.543 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.056259+0000", 2023-11-05T08:32:48.543 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.1753999999999999e-05 2023-11-05T08:32:48.543 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.543 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.544 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:32:48.544 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.056272+0000", 2023-11-05T08:32:48.544 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.3186e-05 2023-11-05T08:32:48.544 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.544 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.545 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:32:48.545 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.056365+0000", 2023-11-05T08:32:48.545 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.2944999999999999e-05 2023-11-05T08:32:48.545 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.545 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.545 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:32:48.546 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.056374+0000", 2023-11-05T08:32:48.546 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.5249999999999994e-06 2023-11-05T08:32:48.546 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.546 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.546 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:32:48.547 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.058474+0000", 2023-11-05T08:32:48.547 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0020994960000000002 2023-11-05T08:32:48.547 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.547 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.547 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:32:48.547 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.058510+0000", 2023-11-05T08:32:48.548 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.5883999999999997e-05 2023-11-05T08:32:48.548 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:32:48.548 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:32:48.548 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:32:48.548 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.549 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.549 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.3s2 35/30 ECSubWrite(tid=180, reqid=client.4691.0:219, at_version=35'9, trim_to=0'0, roll_forward_to=33'6))", 2023-11-05T08:32:48.549 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:48.058669+0000", 2023-11-05T08:32:48.549 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 0.47734939100000001, 2023-11-05T08:32:48.549 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001356381, 2023-11-05T08:32:48.550 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:32:48.550 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:32:48.550 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:32:48.550 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.550 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:32:48.551 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.058669+0000", 2023-11-05T08:32:48.551 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:32:48.551 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.551 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.551 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:32:48.551 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.058669+0000", 2023-11-05T08:32:48.552 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:32:48.552 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.552 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.552 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:32:48.552 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.058670+0000", 2023-11-05T08:32:48.553 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.8000000000000005e-07 2023-11-05T08:32:48.553 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.553 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.553 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:32:48.553 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.058690+0000", 2023-11-05T08:32:48.553 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.0173000000000002e-05 2023-11-05T08:32:48.554 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.554 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.554 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:32:48.554 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.058715+0000", 2023-11-05T08:32:48.554 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.5080999999999999e-05 2023-11-05T08:32:48.555 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.555 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.555 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:32:48.555 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.058727+0000", 2023-11-05T08:32:48.555 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.1635e-05 2023-11-05T08:32:48.555 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.556 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.556 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:32:48.556 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.058785+0000", 2023-11-05T08:32:48.556 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.8498999999999997e-05 2023-11-05T08:32:48.556 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.557 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.557 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:32:48.557 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.058794+0000", 2023-11-05T08:32:48.557 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.9879999999999993e-06 2023-11-05T08:32:48.557 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.558 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.558 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:32:48.558 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.059997+0000", 2023-11-05T08:32:48.558 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001202966 2023-11-05T08:32:48.558 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.559 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.559 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:32:48.559 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.060026+0000", 2023-11-05T08:32:48.559 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.8259000000000002e-05 2023-11-05T08:32:48.559 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:32:48.560 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:32:48.560 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:32:48.560 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.560 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.560 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.3s2 35/30 ECSubWrite(tid=181, reqid=client.4691.0:220, at_version=35'10, trim_to=0'0, roll_forward_to=33'6))", 2023-11-05T08:32:48.561 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:48.060826+0000", 2023-11-05T08:32:48.561 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 0.47519251400000001, 2023-11-05T08:32:48.561 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001051354, 2023-11-05T08:32:48.561 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:32:48.561 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:32:48.562 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:32:48.562 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.562 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:32:48.562 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.060826+0000", 2023-11-05T08:32:48.562 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:32:48.562 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.563 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.563 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:32:48.563 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.060826+0000", 2023-11-05T08:32:48.563 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:32:48.563 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.564 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.564 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:32:48.564 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.060827+0000", 2023-11-05T08:32:48.564 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.9100000000000003e-07 2023-11-05T08:32:48.564 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.565 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.565 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:32:48.565 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.060844+0000", 2023-11-05T08:32:48.565 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.7229999999999999e-05 2023-11-05T08:32:48.565 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.565 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.566 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:32:48.566 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.060860+0000", 2023-11-05T08:32:48.566 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.5951999999999999e-05 2023-11-05T08:32:48.566 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.566 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.567 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:32:48.567 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.060869+0000", 2023-11-05T08:32:48.567 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.9959999999999999e-06 2023-11-05T08:32:48.567 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.567 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.567 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:32:48.568 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.060930+0000", 2023-11-05T08:32:48.568 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.1013e-05 2023-11-05T08:32:48.568 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.568 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.568 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:32:48.568 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.060939+0000", 2023-11-05T08:32:48.569 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.8070000000000006e-06 2023-11-05T08:32:48.569 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.569 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.569 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:32:48.569 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.061812+0000", 2023-11-05T08:32:48.569 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00087319199999999996 2023-11-05T08:32:48.570 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.570 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.570 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:32:48.570 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.061877+0000", 2023-11-05T08:32:48.570 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.5473000000000001e-05 2023-11-05T08:32:48.571 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:32:48.571 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:32:48.571 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:32:48.571 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.571 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.571 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.3s2 35/30 ECSubWrite(tid=182, reqid=client.4691.0:220, at_version=0'0, trim_to=0'0, roll_forward_to=35'10))", 2023-11-05T08:32:48.572 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:48.066378+0000", 2023-11-05T08:32:48.572 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 0.46964104299999998, 2023-11-05T08:32:48.572 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0013351890000000001, 2023-11-05T08:32:48.572 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:32:48.572 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:32:48.573 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:32:48.573 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.573 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:32:48.573 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.066378+0000", 2023-11-05T08:32:48.573 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:32:48.574 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.574 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.574 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:32:48.574 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.066378+0000", 2023-11-05T08:32:48.574 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:32:48.575 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.575 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.575 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:32:48.575 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.066378+0000", 2023-11-05T08:32:48.575 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.7599999999999997e-07 2023-11-05T08:32:48.575 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.576 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.576 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:32:48.576 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.066391+0000", 2023-11-05T08:32:48.576 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.2289999999999999e-05 2023-11-05T08:32:48.576 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.577 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.577 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:32:48.577 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.066407+0000", 2023-11-05T08:32:48.577 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.6812000000000001e-05 2023-11-05T08:32:48.577 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.578 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.578 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:32:48.578 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.066417+0000", 2023-11-05T08:32:48.578 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.7869999999999996e-06 2023-11-05T08:32:48.578 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.579 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.579 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:32:48.579 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.066562+0000", 2023-11-05T08:32:48.579 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00014468700000000001 2023-11-05T08:32:48.579 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.579 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.580 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:32:48.580 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.066579+0000", 2023-11-05T08:32:48.580 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.6942000000000001e-05 2023-11-05T08:32:48.580 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.580 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.581 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:32:48.581 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.067645+0000", 2023-11-05T08:32:48.581 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001066158 2023-11-05T08:32:48.581 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.581 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.581 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:32:48.582 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.067713+0000", 2023-11-05T08:32:48.582 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.7836999999999994e-05 2023-11-05T08:32:48.582 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:32:48.582 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:32:48.582 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:32:48.582 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.583 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.583 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.5s2 35/30 ECSubWrite(tid=20, reqid=client.4691.0:221, at_version=35'11, trim_to=0'0, roll_forward_to=33'9))", 2023-11-05T08:32:48.583 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:48.086738+0000", 2023-11-05T08:32:48.583 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 0.44928025399999999, 2023-11-05T08:32:48.583 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0065328219999999998, 2023-11-05T08:32:48.584 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:32:48.584 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:32:48.584 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:32:48.584 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.584 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:32:48.584 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.086738+0000", 2023-11-05T08:32:48.585 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:32:48.585 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.585 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.585 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:32:48.585 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.086738+0000", 2023-11-05T08:32:48.586 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:32:48.586 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.586 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.586 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:32:48.586 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.086739+0000", 2023-11-05T08:32:48.586 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.22e-07 2023-11-05T08:32:48.587 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.587 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.587 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:32:48.587 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.087297+0000", 2023-11-05T08:32:48.587 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00055798899999999995 2023-11-05T08:32:48.588 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.588 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.588 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:32:48.588 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.087338+0000", 2023-11-05T08:32:48.588 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.1171000000000003e-05 2023-11-05T08:32:48.588 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.589 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.589 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:32:48.589 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.087364+0000", 2023-11-05T08:32:48.589 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.5826999999999999e-05 2023-11-05T08:32:48.589 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.589 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.590 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:32:48.590 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.087572+0000", 2023-11-05T08:32:48.590 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000208574 2023-11-05T08:32:48.590 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.591 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.591 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:32:48.591 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.087598+0000", 2023-11-05T08:32:48.591 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.5901000000000002e-05 2023-11-05T08:32:48.591 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.592 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.592 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:32:48.592 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.093224+0000", 2023-11-05T08:32:48.592 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0056260210000000001 2023-11-05T08:32:48.592 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.592 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.593 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:32:48.593 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.093271+0000", 2023-11-05T08:32:48.593 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.7117000000000003e-05 2023-11-05T08:32:48.593 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:32:48.593 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:32:48.594 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:32:48.594 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.594 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.594 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.5s2 35/30 ECSubWrite(tid=21, reqid=client.4691.0:222, at_version=35'12, trim_to=0'0, roll_forward_to=33'9))", 2023-11-05T08:32:48.594 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:48.091862+0000", 2023-11-05T08:32:48.595 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 0.444156781, 2023-11-05T08:32:48.595 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.003284333, 2023-11-05T08:32:48.595 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:32:48.595 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:32:48.595 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:32:48.596 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.596 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:32:48.596 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.091862+0000", 2023-11-05T08:32:48.596 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:32:48.596 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.596 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.597 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:32:48.597 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.091862+0000", 2023-11-05T08:32:48.597 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:32:48.597 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.597 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.597 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:32:48.598 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.091863+0000", 2023-11-05T08:32:48.598 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.237e-06 2023-11-05T08:32:48.598 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.598 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.598 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:32:48.599 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.092010+0000", 2023-11-05T08:32:48.599 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00014685699999999999 2023-11-05T08:32:48.599 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.599 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.599 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:32:48.600 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.092034+0000", 2023-11-05T08:32:48.600 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.4596e-05 2023-11-05T08:32:48.600 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.600 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.600 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:32:48.600 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.092047+0000", 2023-11-05T08:32:48.601 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.2841e-05 2023-11-05T08:32:48.601 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.601 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.601 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:32:48.601 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.092166+0000", 2023-11-05T08:32:48.602 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000118317 2023-11-05T08:32:48.602 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.602 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.602 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:32:48.602 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.092182+0000", 2023-11-05T08:32:48.603 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.5852999999999999e-05 2023-11-05T08:32:48.603 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.603 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.603 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:32:48.603 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.095105+0000", 2023-11-05T08:32:48.603 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00292312 2023-11-05T08:32:48.604 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.604 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.604 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:32:48.604 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.095146+0000", 2023-11-05T08:32:48.604 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.1511999999999998e-05 2023-11-05T08:32:48.605 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:32:48.605 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:32:48.605 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:32:48.605 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.605 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.606 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.5s2 35/30 ECSubWrite(tid=22, reqid=client.4691.0:223, at_version=35'13, trim_to=0'0, roll_forward_to=33'9))", 2023-11-05T08:32:48.606 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:48.097383+0000", 2023-11-05T08:32:48.606 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 0.43863586199999999, 2023-11-05T08:32:48.606 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00092105799999999999, 2023-11-05T08:32:48.606 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:32:48.607 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:32:48.607 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:32:48.607 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.607 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:32:48.607 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.097383+0000", 2023-11-05T08:32:48.607 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:32:48.608 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.608 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.608 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:32:48.608 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.097383+0000", 2023-11-05T08:32:48.608 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:32:48.608 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.609 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.609 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:32:48.609 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.097384+0000", 2023-11-05T08:32:48.609 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.5999999999999991e-07 2023-11-05T08:32:48.609 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.610 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.610 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:32:48.610 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.097407+0000", 2023-11-05T08:32:48.610 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.2904e-05 2023-11-05T08:32:48.610 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.610 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.611 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:32:48.611 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.097426+0000", 2023-11-05T08:32:48.611 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.9568e-05 2023-11-05T08:32:48.611 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.611 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.612 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:32:48.612 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.097438+0000", 2023-11-05T08:32:48.612 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.2105e-05 2023-11-05T08:32:48.612 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.612 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.612 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:32:48.613 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.097504+0000", 2023-11-05T08:32:48.613 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.5689000000000001e-05 2023-11-05T08:32:48.613 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.613 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.613 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:32:48.613 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.097510+0000", 2023-11-05T08:32:48.614 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.003e-06 2023-11-05T08:32:48.614 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.614 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.614 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:32:48.614 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.098277+0000", 2023-11-05T08:32:48.615 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00076719600000000005 2023-11-05T08:32:48.615 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.615 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.615 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:32:48.615 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.098304+0000", 2023-11-05T08:32:48.616 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.6633000000000001e-05 2023-11-05T08:32:48.616 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:32:48.616 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:32:48.616 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:32:48.616 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.616 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.617 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.5s2 35/30 ECSubWrite(tid=23, reqid=client.4691.0:223, at_version=0'0, trim_to=0'0, roll_forward_to=35'13))", 2023-11-05T08:32:48.617 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:48.103053+0000", 2023-11-05T08:32:48.617 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 0.432965088, 2023-11-05T08:32:48.617 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00096376299999999997, 2023-11-05T08:32:48.617 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:32:48.617 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:32:48.618 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:32:48.618 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.618 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:32:48.618 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.103053+0000", 2023-11-05T08:32:48.618 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:32:48.619 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.619 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.619 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:32:48.619 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.103053+0000", 2023-11-05T08:32:48.619 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:32:48.619 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.620 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.620 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:32:48.620 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.103054+0000", 2023-11-05T08:32:48.620 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.5099999999999998e-07 2023-11-05T08:32:48.620 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.620 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.621 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:32:48.621 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.103087+0000", 2023-11-05T08:32:48.621 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.2258000000000002e-05 2023-11-05T08:32:48.621 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.621 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.622 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:32:48.622 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.103110+0000", 2023-11-05T08:32:48.622 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.3198999999999999e-05 2023-11-05T08:32:48.622 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.622 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.623 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:32:48.623 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.103121+0000", 2023-11-05T08:32:48.623 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.132e-05 2023-11-05T08:32:48.623 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.623 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.623 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:32:48.624 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.103292+0000", 2023-11-05T08:32:48.624 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00017078500000000001 2023-11-05T08:32:48.624 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.624 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.624 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:32:48.625 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.103301+0000", 2023-11-05T08:32:48.625 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.3940000000000002e-06 2023-11-05T08:32:48.625 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.625 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.625 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:32:48.625 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.103972+0000", 2023-11-05T08:32:48.626 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00067038899999999997 2023-11-05T08:32:48.626 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.626 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.626 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:32:48.626 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.104017+0000", 2023-11-05T08:32:48.626 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.5566999999999999e-05 2023-11-05T08:32:48.627 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:32:48.627 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:32:48.627 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:32:48.627 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.627 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.628 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpRead(3.5s2 35/30 ECSubRead(tid=24, to_read={3:a32feb2c:::smithi09039252-47:head=0,4096,0}, subchunks={3:a32feb2c:::smithi09039252-47:head=[0,1]}, attrs_to_read=))", 2023-11-05T08:32:48.628 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:48.105500+0000", 2023-11-05T08:32:48.628 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 0.43051868700000001, 2023-11-05T08:32:48.628 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00025701100000000002, 2023-11-05T08:32:48.628 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:32:48.628 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:32:48.629 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:32:48.629 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.629 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:32:48.629 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.105500+0000", 2023-11-05T08:32:48.629 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:32:48.629 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.630 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.630 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:32:48.630 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.105500+0000", 2023-11-05T08:32:48.630 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:32:48.630 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.631 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.631 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:32:48.631 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.105501+0000", 2023-11-05T08:32:48.631 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.8599999999999998e-07 2023-11-05T08:32:48.631 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.632 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.632 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:32:48.632 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.105508+0000", 2023-11-05T08:32:48.632 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.4449999999999999e-06 2023-11-05T08:32:48.632 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.633 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.633 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:32:48.633 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.105521+0000", 2023-11-05T08:32:48.633 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.3234000000000001e-05 2023-11-05T08:32:48.633 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.634 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.634 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:32:48.634 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.105530+0000", 2023-11-05T08:32:48.634 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.2e-06 2023-11-05T08:32:48.634 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.635 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.635 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:32:48.635 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.105621+0000", 2023-11-05T08:32:48.635 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.0722000000000003e-05 2023-11-05T08:32:48.635 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:48.636 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:48.636 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:32:48.636 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.105757+0000", 2023-11-05T08:32:48.636 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00013572399999999999 2023-11-05T08:32:48.636 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:32:48.637 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:32:48.637 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:32:48.637 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:32:48.637 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:32:48.637 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:32:48.639 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2023-11-05T08:32:48.639 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:32:48.639+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:32:48.740 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:32:48.737+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:32:48.813 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:32:48.814 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [], 2023-11-05T08:32:48.814 INFO:teuthology.orchestra.run.smithi002.stdout: "num_ops": 0 2023-11-05T08:32:48.814 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:32:48.828 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2023-11-05T08:32:48.840 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:32:48.835+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:32:48.941 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:32:48.937+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:32:48.954 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:32:48.954 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [], 2023-11-05T08:32:48.954 INFO:teuthology.orchestra.run.smithi002.stdout: "complaint_time": 30, 2023-11-05T08:32:48.954 INFO:teuthology.orchestra.run.smithi002.stdout: "num_blocked_ops": 0 2023-11-05T08:32:48.955 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:32:48.969 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2023-11-05T08:32:49.042 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:32:49.039+0000 7fdc5cd58640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:32:49.049 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-11-05T08:32:49.058 INFO:tasks.rados.rados.0.smithi090.stdout:81: done (3 left) 2023-11-05T08:32:49.059 INFO:tasks.rados.rados.0.smithi090.stdout:82: rollback oid 23 current snap is 2 2023-11-05T08:32:49.059 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 23 to 1 2023-11-05T08:32:49.059 INFO:tasks.rados.rados.0.smithi090.stdout:78: finishing write tid 1 to smithi09039252-47 2023-11-05T08:32:49.059 INFO:tasks.rados.rados.0.smithi090.stdout:78: finishing write tid 2 to smithi09039252-47 2023-11-05T08:32:49.059 INFO:tasks.rados.rados.0.smithi090.stdout:79: finishing rollback tid 0 to smithi09039252-31 2023-11-05T08:32:49.060 INFO:tasks.rados.rados.0.smithi090.stdout:79: finishing rollback tid 1 to smithi09039252-31 2023-11-05T08:32:49.060 INFO:tasks.rados.rados.0.smithi090.stdout:78: finishing write tid 3 to smithi09039252-47 2023-11-05T08:32:49.060 INFO:tasks.rados.rados.0.smithi090.stdout:79: finishing rollback tid 2 to smithi09039252-31 2023-11-05T08:32:49.060 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 31 v 9 (ObjNum 59 snap 1 seq_num 59) dirty exists 2023-11-05T08:32:49.060 INFO:tasks.rados.rados.0.smithi090.stdout:78: finishing write tid 4 to smithi09039252-47 2023-11-05T08:32:49.060 INFO:tasks.rados.rados.0.smithi090.stdout:78: oid 47 updating version 0 to 11 2023-11-05T08:32:49.061 INFO:tasks.rados.rados.0.smithi090.stdout:78: oid 47 updating version 11 to 12 2023-11-05T08:32:49.061 INFO:tasks.rados.rados.0.smithi090.stdout:78: oid 47 updating version 12 to 13 2023-11-05T08:32:49.061 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 47 v 13 (ObjNum 61 snap 2 seq_num 61) dirty exists 2023-11-05T08:32:49.061 INFO:tasks.rados.rados.0.smithi090.stdout:78: left oid 47 (ObjNum 61 snap 2 seq_num 61) 2023-11-05T08:32:49.061 INFO:tasks.rados.rados.0.smithi090.stdout:80: finishing write tid 1 to smithi09039252-46 2023-11-05T08:32:49.062 INFO:tasks.rados.rados.0.smithi090.stdout:80: finishing write tid 2 to smithi09039252-46 2023-11-05T08:32:49.062 INFO:tasks.rados.rados.0.smithi090.stdout:80: finishing write tid 3 to smithi09039252-46 2023-11-05T08:32:49.062 INFO:tasks.rados.rados.0.smithi090.stdout:80: finishing write tid 4 to smithi09039252-46 2023-11-05T08:32:49.062 INFO:tasks.rados.rados.0.smithi090.stdout:80: oid 46 updating version 0 to 14 2023-11-05T08:32:49.062 INFO:tasks.rados.rados.0.smithi090.stdout:80: oid 46 version 14 is already newer than 13 2023-11-05T08:32:49.063 INFO:tasks.rados.rados.0.smithi090.stdout:80: oid 46 updating version 14 to 15 2023-11-05T08:32:49.063 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 46 v 15 (ObjNum 62 snap 2 seq_num 62) dirty exists 2023-11-05T08:32:49.063 INFO:tasks.rados.rados.0.smithi090.stdout:80: left oid 46 (ObjNum 62 snap 2 seq_num 62) 2023-11-05T08:32:49.063 INFO:tasks.rados.rados.0.smithi090.stdout:78: done (3 left) 2023-11-05T08:32:49.063 INFO:tasks.rados.rados.0.smithi090.stdout:79: done (2 left) 2023-11-05T08:32:49.064 INFO:tasks.rados.rados.0.smithi090.stdout:80: done (1 left) 2023-11-05T08:32:49.064 INFO:tasks.rados.rados.0.smithi090.stdout:83: read oid 31 snap -1 2023-11-05T08:32:49.064 INFO:tasks.rados.rados.0.smithi090.stdout:83: expect (ObjNum 59 snap 1 seq_num 59) 2023-11-05T08:32:49.064 INFO:tasks.rados.rados.0.smithi090.stdout:84: snap_create 2023-11-05T08:32:49.076 INFO:tasks.ceph.osd.0.smithi002.stderr:2023-11-05T08:32:49.067+0000 7f89d053b640 -1 snap_mapper.get_next_objects_to_trimNew Clone-Objects were added to Snap 2 after trimming was started 2023-11-05T08:32:49.076 INFO:tasks.ceph.osd.0.smithi002.stderr:2023-11-05T08:32:49.067+0000 7f89d053b640 -1 snap_mapper.0x7f89d0537750 2023-11-05T08:32:49.078 INFO:tasks.rados.rados.0.smithi090.stdout:83: expect (ObjNum 59 snap 1 seq_num 59) 2023-11-05T08:32:49.079 INFO:tasks.ceph.osd.0.smithi002.stderr:./src/osd/SnapMapper.cc: In function 'int SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector*)' thread 7f89d053b640 time 2023-11-05T08:32:49.072277+0000 2023-11-05T08:32:49.079 INFO:tasks.ceph.osd.0.smithi002.stderr:./src/osd/SnapMapper.cc: 650: ceph_abort_msg("Clone-Objects were added to a trimmed snap") 2023-11-05T08:32:49.080 INFO:tasks.ceph.osd.0.smithi002.stderr: ceph version 18.0.0-6776-g420cd129 (420cd129397301b1521ceb74e1bdbd9b855d2189) reef (dev) 2023-11-05T08:32:49.080 INFO:tasks.ceph.osd.0.smithi002.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x55d2449f39dc] 2023-11-05T08:32:49.080 INFO:tasks.ceph.osd.0.smithi002.stderr: 2: (SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector >*)+0x4dd) [0x55d244cd7c9d] 2023-11-05T08:32:49.080 INFO:tasks.ceph.osd.0.smithi002.stderr: 3: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x55d244c422c2] 2023-11-05T08:32:49.080 INFO:tasks.ceph.osd.0.smithi002.stderr: 4: ceph-osd(+0xd7e721) [0x55d244a9d721] 2023-11-05T08:32:49.081 INFO:tasks.ceph.osd.0.smithi002.stderr: 5: ceph-osd(+0x1695cbb) [0x55d2453b4cbb] 2023-11-05T08:32:49.081 INFO:tasks.ceph.osd.0.smithi002.stderr: 6: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x55d244bdf288] 2023-11-05T08:32:49.081 INFO:tasks.ceph.osd.0.smithi002.stderr: 7: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x55d244d159df] 2023-11-05T08:32:49.081 INFO:tasks.ceph.osd.0.smithi002.stderr: 8: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55d244b33d23] 2023-11-05T08:32:49.081 INFO:tasks.ceph.osd.0.smithi002.stderr: 9: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x55d244fe5053] 2023-11-05T08:32:49.082 INFO:tasks.ceph.osd.0.smithi002.stderr: 10: ceph-osd(+0x12c6584) [0x55d244fe5584] 2023-11-05T08:32:49.082 INFO:tasks.ceph.osd.0.smithi002.stderr: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f89eec52b43] 2023-11-05T08:32:49.082 INFO:tasks.ceph.osd.0.smithi002.stderr: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f89eece4a00] 2023-11-05T08:32:49.082 INFO:tasks.ceph.osd.0.smithi002.stderr:*** Caught signal (Aborted) ** 2023-11-05T08:32:49.082 INFO:tasks.ceph.osd.0.smithi002.stderr: in thread 7f89d053b640 thread_name:tp_osd_tp 2023-11-05T08:32:49.083 INFO:tasks.ceph.osd.0.smithi002.stderr:2023-11-05T08:32:49.079+0000 7f89d053b640 -1 ./src/osd/SnapMapper.cc: In function 'int SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector*)' thread 7f89d053b640 time 2023-11-05T08:32:49.072277+0000 2023-11-05T08:32:49.083 INFO:tasks.ceph.osd.0.smithi002.stderr:./src/osd/SnapMapper.cc: 650: ceph_abort_msg("Clone-Objects were added to a trimmed snap") 2023-11-05T08:32:49.083 INFO:tasks.ceph.osd.0.smithi002.stderr: 2023-11-05T08:32:49.083 INFO:tasks.ceph.osd.0.smithi002.stderr: ceph version 18.0.0-6776-g420cd129 (420cd129397301b1521ceb74e1bdbd9b855d2189) reef (dev) 2023-11-05T08:32:49.084 INFO:tasks.ceph.osd.0.smithi002.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x55d2449f39dc] 2023-11-05T08:32:49.084 INFO:tasks.ceph.osd.0.smithi002.stderr: 2: (SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector >*)+0x4dd) [0x55d244cd7c9d] 2023-11-05T08:32:49.084 INFO:tasks.ceph.osd.0.smithi002.stderr: 3: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x55d244c422c2] 2023-11-05T08:32:49.084 INFO:tasks.ceph.osd.0.smithi002.stderr: 4: ceph-osd(+0xd7e721) [0x55d244a9d721] 2023-11-05T08:32:49.084 INFO:tasks.ceph.osd.0.smithi002.stderr: 5: ceph-osd(+0x1695cbb) [0x55d2453b4cbb] 2023-11-05T08:32:49.085 INFO:tasks.ceph.osd.0.smithi002.stderr: 6: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x55d244bdf288] 2023-11-05T08:32:49.085 INFO:tasks.ceph.osd.0.smithi002.stderr: 7: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x55d244d159df] 2023-11-05T08:32:49.085 INFO:tasks.ceph.osd.0.smithi002.stderr: 8: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55d244b33d23] 2023-11-05T08:32:49.085 INFO:tasks.ceph.osd.0.smithi002.stderr: 9: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x55d244fe5053] 2023-11-05T08:32:49.085 INFO:tasks.ceph.osd.0.smithi002.stderr: 10: ceph-osd(+0x12c6584) [0x55d244fe5584] 2023-11-05T08:32:49.086 INFO:tasks.ceph.osd.0.smithi002.stderr: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f89eec52b43] 2023-11-05T08:32:49.086 INFO:tasks.ceph.osd.0.smithi002.stderr: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f89eece4a00] 2023-11-05T08:32:49.086 INFO:tasks.ceph.osd.0.smithi002.stderr: 2023-11-05T08:32:49.087 INFO:tasks.ceph.osd.0.smithi002.stderr: ceph version 18.0.0-6776-g420cd129 (420cd129397301b1521ceb74e1bdbd9b855d2189) reef (dev) 2023-11-05T08:32:49.087 INFO:tasks.ceph.osd.0.smithi002.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f89eec00520] 2023-11-05T08:32:49.087 INFO:tasks.ceph.osd.0.smithi002.stderr: 2: pthread_kill() 2023-11-05T08:32:49.087 INFO:tasks.ceph.osd.0.smithi002.stderr: 3: raise() 2023-11-05T08:32:49.087 INFO:tasks.ceph.osd.0.smithi002.stderr: 4: abort() 2023-11-05T08:32:49.088 INFO:tasks.ceph.osd.0.smithi002.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x55d2449f3a91] 2023-11-05T08:32:49.088 INFO:tasks.ceph.osd.0.smithi002.stderr: 6: (SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector >*)+0x4dd) [0x55d244cd7c9d] 2023-11-05T08:32:49.088 INFO:tasks.ceph.osd.0.smithi002.stderr: 7: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x55d244c422c2] 2023-11-05T08:32:49.088 INFO:tasks.ceph.osd.0.smithi002.stderr: 8: ceph-osd(+0xd7e721) [0x55d244a9d721] 2023-11-05T08:32:49.088 INFO:tasks.ceph.osd.0.smithi002.stderr: 9: ceph-osd(+0x1695cbb) [0x55d2453b4cbb] 2023-11-05T08:32:49.088 INFO:tasks.ceph.osd.0.smithi002.stderr: 10: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x55d244bdf288] 2023-11-05T08:32:49.089 INFO:tasks.ceph.osd.0.smithi002.stderr: 11: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x55d244d159df] 2023-11-05T08:32:49.089 INFO:tasks.ceph.osd.0.smithi002.stderr: 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55d244b33d23] 2023-11-05T08:32:49.089 INFO:tasks.ceph.osd.0.smithi002.stderr: 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x55d244fe5053] 2023-11-05T08:32:49.089 INFO:tasks.ceph.osd.0.smithi002.stderr: 14: ceph-osd(+0x12c6584) [0x55d244fe5584] 2023-11-05T08:32:49.089 INFO:tasks.ceph.osd.0.smithi002.stderr: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f89eec52b43] 2023-11-05T08:32:49.089 INFO:tasks.ceph.osd.0.smithi002.stderr: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f89eece4a00] 2023-11-05T08:32:49.090 INFO:tasks.ceph.osd.0.smithi002.stderr:2023-11-05T08:32:49.083+0000 7f89d053b640 -1 *** Caught signal (Aborted) ** 2023-11-05T08:32:49.090 INFO:tasks.ceph.osd.0.smithi002.stderr: in thread 7f89d053b640 thread_name:tp_osd_tp 2023-11-05T08:32:49.090 INFO:tasks.ceph.osd.0.smithi002.stderr: 2023-11-05T08:32:49.090 INFO:tasks.ceph.osd.0.smithi002.stderr: ceph version 18.0.0-6776-g420cd129 (420cd129397301b1521ceb74e1bdbd9b855d2189) reef (dev) 2023-11-05T08:32:49.091 INFO:tasks.ceph.osd.0.smithi002.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f89eec00520] 2023-11-05T08:32:49.091 INFO:tasks.ceph.osd.0.smithi002.stderr: 2: pthread_kill() 2023-11-05T08:32:49.091 INFO:tasks.ceph.osd.0.smithi002.stderr: 3: raise() 2023-11-05T08:32:49.091 INFO:tasks.ceph.osd.0.smithi002.stderr: 4: abort() 2023-11-05T08:32:49.091 INFO:tasks.ceph.osd.0.smithi002.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x55d2449f3a91] 2023-11-05T08:32:49.091 INFO:tasks.ceph.osd.0.smithi002.stderr: 6: (SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector >*)+0x4dd) [0x55d244cd7c9d] 2023-11-05T08:32:49.092 INFO:tasks.ceph.osd.0.smithi002.stderr: 7: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x55d244c422c2] 2023-11-05T08:32:49.092 INFO:tasks.ceph.osd.0.smithi002.stderr: 8: ceph-osd(+0xd7e721) [0x55d244a9d721] 2023-11-05T08:32:49.092 INFO:tasks.ceph.osd.0.smithi002.stderr: 9: ceph-osd(+0x1695cbb) [0x55d2453b4cbb] 2023-11-05T08:32:49.092 INFO:tasks.ceph.osd.0.smithi002.stderr: 10: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x55d244bdf288] 2023-11-05T08:32:49.092 INFO:tasks.ceph.osd.0.smithi002.stderr: 11: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x55d244d159df] 2023-11-05T08:32:49.093 INFO:tasks.ceph.osd.0.smithi002.stderr: 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55d244b33d23] 2023-11-05T08:32:49.093 INFO:tasks.ceph.osd.0.smithi002.stderr: 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x55d244fe5053] 2023-11-05T08:32:49.093 INFO:tasks.ceph.osd.0.smithi002.stderr: 14: ceph-osd(+0x12c6584) [0x55d244fe5584] 2023-11-05T08:32:49.093 INFO:tasks.ceph.osd.0.smithi002.stderr: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f89eec52b43] 2023-11-05T08:32:49.093 INFO:tasks.ceph.osd.0.smithi002.stderr: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f89eece4a00] 2023-11-05T08:32:49.093 INFO:tasks.ceph.osd.0.smithi002.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-11-05T08:32:49.094 INFO:tasks.ceph.osd.0.smithi002.stderr: 2023-11-05T08:32:49.107 INFO:tasks.ceph.osd.0.smithi002.stderr: -1442> 2023-11-05T08:32:49.067+0000 7f89d053b640 -1 snap_mapper.get_next_objects_to_trimNew Clone-Objects were added to Snap 2 after trimming was started 2023-11-05T08:32:49.107 INFO:tasks.ceph.osd.0.smithi002.stderr: -1441> 2023-11-05T08:32:49.067+0000 7f89d053b640 -1 snap_mapper.0x7f89d0537750 2023-11-05T08:32:49.108 INFO:tasks.ceph.osd.0.smithi002.stderr: -613> 2023-11-05T08:32:49.079+0000 7f89d053b640 -1 ./src/osd/SnapMapper.cc: In function 'int SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector*)' thread 7f89d053b640 time 2023-11-05T08:32:49.072277+0000 2023-11-05T08:32:49.108 INFO:tasks.ceph.osd.0.smithi002.stderr:./src/osd/SnapMapper.cc: 650: ceph_abort_msg("Clone-Objects were added to a trimmed snap") 2023-11-05T08:32:49.109 INFO:tasks.ceph.osd.0.smithi002.stderr: 2023-11-05T08:32:49.109 INFO:tasks.ceph.osd.0.smithi002.stderr: ceph version 18.0.0-6776-g420cd129 (420cd129397301b1521ceb74e1bdbd9b855d2189) reef (dev) 2023-11-05T08:32:49.109 INFO:tasks.ceph.osd.0.smithi002.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x55d2449f39dc] 2023-11-05T08:32:49.109 INFO:tasks.ceph.osd.0.smithi002.stderr: 2: (SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector >*)+0x4dd) [0x55d244cd7c9d] 2023-11-05T08:32:49.109 INFO:tasks.ceph.osd.0.smithi002.stderr: 3: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x55d244c422c2] 2023-11-05T08:32:49.109 INFO:tasks.ceph.osd.0.smithi002.stderr: 4: ceph-osd(+0xd7e721) [0x55d244a9d721] 2023-11-05T08:32:49.110 INFO:tasks.ceph.osd.0.smithi002.stderr: 5: ceph-osd(+0x1695cbb) [0x55d2453b4cbb] 2023-11-05T08:32:49.110 INFO:tasks.ceph.osd.0.smithi002.stderr: 6: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x55d244bdf288] 2023-11-05T08:32:49.110 INFO:tasks.ceph.osd.0.smithi002.stderr: 7: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x55d244d159df] 2023-11-05T08:32:49.110 INFO:tasks.ceph.osd.0.smithi002.stderr: 8: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55d244b33d23] 2023-11-05T08:32:49.110 INFO:tasks.ceph.osd.0.smithi002.stderr: 9: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x55d244fe5053] 2023-11-05T08:32:49.111 INFO:tasks.ceph.osd.0.smithi002.stderr: 10: ceph-osd(+0x12c6584) [0x55d244fe5584] 2023-11-05T08:32:49.111 INFO:tasks.ceph.osd.0.smithi002.stderr: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f89eec52b43] 2023-11-05T08:32:49.111 INFO:tasks.ceph.osd.0.smithi002.stderr: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f89eece4a00] 2023-11-05T08:32:49.111 INFO:tasks.ceph.osd.0.smithi002.stderr: 2023-11-05T08:32:49.111 INFO:tasks.ceph.osd.0.smithi002.stderr: -275> 2023-11-05T08:32:49.083+0000 7f89d053b640 -1 *** Caught signal (Aborted) ** 2023-11-05T08:32:49.111 INFO:tasks.ceph.osd.0.smithi002.stderr: in thread 7f89d053b640 thread_name:tp_osd_tp 2023-11-05T08:32:49.112 INFO:tasks.ceph.osd.0.smithi002.stderr: 2023-11-05T08:32:49.112 INFO:tasks.ceph.osd.0.smithi002.stderr: ceph version 18.0.0-6776-g420cd129 (420cd129397301b1521ceb74e1bdbd9b855d2189) reef (dev) 2023-11-05T08:32:49.112 INFO:tasks.ceph.osd.0.smithi002.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f89eec00520] 2023-11-05T08:32:49.112 INFO:tasks.ceph.osd.0.smithi002.stderr: 2: pthread_kill() 2023-11-05T08:32:49.112 INFO:tasks.ceph.osd.0.smithi002.stderr: 3: raise() 2023-11-05T08:32:49.112 INFO:tasks.ceph.osd.0.smithi002.stderr: 4: abort() 2023-11-05T08:32:49.113 INFO:tasks.ceph.osd.0.smithi002.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x55d2449f3a91] 2023-11-05T08:32:49.113 INFO:tasks.ceph.osd.0.smithi002.stderr: 6: (SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector >*)+0x4dd) [0x55d244cd7c9d] 2023-11-05T08:32:49.113 INFO:tasks.ceph.osd.0.smithi002.stderr: 7: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x55d244c422c2] 2023-11-05T08:32:49.113 INFO:tasks.ceph.osd.0.smithi002.stderr: 8: ceph-osd(+0xd7e721) [0x55d244a9d721] 2023-11-05T08:32:49.113 INFO:tasks.ceph.osd.0.smithi002.stderr: 9: ceph-osd(+0x1695cbb) [0x55d2453b4cbb] 2023-11-05T08:32:49.114 INFO:tasks.ceph.osd.0.smithi002.stderr: 10: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x55d244bdf288] 2023-11-05T08:32:49.114 INFO:tasks.ceph.osd.0.smithi002.stderr: 11: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x55d244d159df] 2023-11-05T08:32:49.114 INFO:tasks.ceph.osd.0.smithi002.stderr: 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55d244b33d23] 2023-11-05T08:32:49.114 INFO:tasks.ceph.osd.0.smithi002.stderr: 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x55d244fe5053] 2023-11-05T08:32:49.114 INFO:tasks.ceph.osd.0.smithi002.stderr: 14: ceph-osd(+0x12c6584) [0x55d244fe5584] 2023-11-05T08:32:49.115 INFO:tasks.ceph.osd.0.smithi002.stderr: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f89eec52b43] 2023-11-05T08:32:49.115 INFO:tasks.ceph.osd.0.smithi002.stderr: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f89eece4a00] 2023-11-05T08:32:49.115 INFO:tasks.ceph.osd.0.smithi002.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-11-05T08:32:49.115 INFO:tasks.ceph.osd.0.smithi002.stderr: 2023-11-05T08:32:49.116 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:32:49.116 INFO:teuthology.orchestra.run.smithi002.stdout: "size": 20, 2023-11-05T08:32:49.116 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 600, 2023-11-05T08:32:49.116 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [] 2023-11-05T08:32:49.116 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:32:49.118 INFO:tasks.ceph.osd.0.smithi002.stderr: -1911> 2023-11-05T08:32:49.067+0000 7f89d053b640 -1 snap_mapper.get_next_objects_to_trimNew Clone-Objects were added to Snap 2 after trimming was started 2023-11-05T08:32:49.118 INFO:tasks.ceph.osd.0.smithi002.stderr: -1910> 2023-11-05T08:32:49.067+0000 7f89d053b640 -1 snap_mapper.0x7f89d0537750 2023-11-05T08:32:49.118 INFO:tasks.ceph.osd.0.smithi002.stderr: -1082> 2023-11-05T08:32:49.079+0000 7f89d053b640 -1 ./src/osd/SnapMapper.cc: In function 'int SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector*)' thread 7f89d053b640 time 2023-11-05T08:32:49.072277+0000 2023-11-05T08:32:49.119 INFO:tasks.ceph.osd.0.smithi002.stderr:./src/osd/SnapMapper.cc: 650: ceph_abort_msg("Clone-Objects were added to a trimmed snap") 2023-11-05T08:32:49.119 INFO:tasks.ceph.osd.0.smithi002.stderr: 2023-11-05T08:32:49.119 INFO:tasks.ceph.osd.0.smithi002.stderr: ceph version 18.0.0-6776-g420cd129 (420cd129397301b1521ceb74e1bdbd9b855d2189) reef (dev) 2023-11-05T08:32:49.119 INFO:tasks.ceph.osd.0.smithi002.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x55d2449f39dc] 2023-11-05T08:32:49.119 INFO:tasks.ceph.osd.0.smithi002.stderr: 2: (SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector >*)+0x4dd) [0x55d244cd7c9d] 2023-11-05T08:32:49.120 INFO:tasks.ceph.osd.0.smithi002.stderr: 3: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x55d244c422c2] 2023-11-05T08:32:49.120 INFO:tasks.ceph.osd.0.smithi002.stderr: 4: ceph-osd(+0xd7e721) [0x55d244a9d721] 2023-11-05T08:32:49.120 INFO:tasks.ceph.osd.0.smithi002.stderr: 5: ceph-osd(+0x1695cbb) [0x55d2453b4cbb] 2023-11-05T08:32:49.120 INFO:tasks.ceph.osd.0.smithi002.stderr: 6: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x55d244bdf288] 2023-11-05T08:32:49.120 INFO:tasks.ceph.osd.0.smithi002.stderr: 7: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x55d244d159df] 2023-11-05T08:32:49.120 INFO:tasks.ceph.osd.0.smithi002.stderr: 8: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55d244b33d23] 2023-11-05T08:32:49.121 INFO:tasks.ceph.osd.0.smithi002.stderr: 9: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x55d244fe5053] 2023-11-05T08:32:49.121 INFO:tasks.ceph.osd.0.smithi002.stderr: 10: ceph-osd(+0x12c6584) [0x55d244fe5584] 2023-11-05T08:32:49.121 INFO:tasks.ceph.osd.0.smithi002.stderr: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f89eec52b43] 2023-11-05T08:32:49.121 INFO:tasks.ceph.osd.0.smithi002.stderr: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f89eece4a00] 2023-11-05T08:32:49.121 INFO:tasks.ceph.osd.0.smithi002.stderr: 2023-11-05T08:32:49.122 INFO:tasks.ceph.osd.0.smithi002.stderr: -744> 2023-11-05T08:32:49.083+0000 7f89d053b640 -1 *** Caught signal (Aborted) ** 2023-11-05T08:32:49.122 INFO:tasks.ceph.osd.0.smithi002.stderr: in thread 7f89d053b640 thread_name:tp_osd_tp 2023-11-05T08:32:49.122 INFO:tasks.ceph.osd.0.smithi002.stderr: 2023-11-05T08:32:49.122 INFO:tasks.ceph.osd.0.smithi002.stderr: ceph version 18.0.0-6776-g420cd129 (420cd129397301b1521ceb74e1bdbd9b855d2189) reef (dev) 2023-11-05T08:32:49.122 INFO:tasks.ceph.osd.0.smithi002.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f89eec00520] 2023-11-05T08:32:49.123 INFO:tasks.ceph.osd.0.smithi002.stderr: 2: pthread_kill() 2023-11-05T08:32:49.123 INFO:tasks.ceph.osd.0.smithi002.stderr: 3: raise() 2023-11-05T08:32:49.123 INFO:tasks.ceph.osd.0.smithi002.stderr: 4: abort() 2023-11-05T08:32:49.123 INFO:tasks.ceph.osd.0.smithi002.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x55d2449f3a91] 2023-11-05T08:32:49.123 INFO:tasks.ceph.osd.0.smithi002.stderr: 6: (SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector >*)+0x4dd) [0x55d244cd7c9d] 2023-11-05T08:32:49.123 INFO:tasks.ceph.osd.0.smithi002.stderr: 7: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x55d244c422c2] 2023-11-05T08:32:49.124 INFO:tasks.ceph.osd.0.smithi002.stderr: 8: ceph-osd(+0xd7e721) [0x55d244a9d721] 2023-11-05T08:32:49.124 INFO:tasks.ceph.osd.0.smithi002.stderr: 9: ceph-osd(+0x1695cbb) [0x55d2453b4cbb] 2023-11-05T08:32:49.124 INFO:tasks.ceph.osd.0.smithi002.stderr: 10: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x55d244bdf288] 2023-11-05T08:32:49.124 INFO:tasks.ceph.osd.0.smithi002.stderr: 11: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x55d244d159df] 2023-11-05T08:32:49.124 INFO:tasks.ceph.osd.0.smithi002.stderr: 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55d244b33d23] 2023-11-05T08:32:49.124 INFO:tasks.ceph.osd.0.smithi002.stderr: 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x55d244fe5053] 2023-11-05T08:32:49.125 INFO:tasks.ceph.osd.0.smithi002.stderr: 14: ceph-osd(+0x12c6584) [0x55d244fe5584] 2023-11-05T08:32:49.125 INFO:tasks.ceph.osd.0.smithi002.stderr: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f89eec52b43] 2023-11-05T08:32:49.125 INFO:tasks.ceph.osd.0.smithi002.stderr: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f89eece4a00] 2023-11-05T08:32:49.125 INFO:tasks.ceph.osd.0.smithi002.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-11-05T08:32:49.125 INFO:tasks.ceph.osd.0.smithi002.stderr: 2023-11-05T08:32:49.133 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2023-11-05T08:32:49.142 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:32:49.141+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:32:49.188 INFO:tasks.rados.rados.0.smithi090.stdout:82: finishing rollback tid 0 to smithi09039252-23 2023-11-05T08:32:49.189 INFO:tasks.rados.rados.0.smithi090.stdout:82: finishing rollback tid 1 to smithi09039252-23 2023-11-05T08:32:49.189 INFO:tasks.rados.rados.0.smithi090.stdout:82: finishing rollback tid 2 to smithi09039252-23 2023-11-05T08:32:49.189 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 23 v 12 (ObjNum 60 snap 1 seq_num 60) dirty exists 2023-11-05T08:32:49.243 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:32:49.241+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:32:49.295 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:32:49.296 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [], 2023-11-05T08:32:49.296 INFO:teuthology.orchestra.run.smithi002.stdout: "num_ops": 0 2023-11-05T08:32:49.296 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:32:49.310 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2023-11-05T08:32:49.355 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:32:49.355+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:32:49.444 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:32:49.444 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [], 2023-11-05T08:32:49.445 INFO:teuthology.orchestra.run.smithi002.stdout: "complaint_time": 30, 2023-11-05T08:32:49.445 INFO:teuthology.orchestra.run.smithi002.stdout: "num_blocked_ops": 0 2023-11-05T08:32:49.445 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:32:49.445 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:32:49.441+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:32:49.459 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2023-11-05T08:32:49.546 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:32:49.540+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:32:49.634 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:32:49.634 INFO:teuthology.orchestra.run.smithi002.stdout: "size": 20, 2023-11-05T08:32:49.635 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 600, 2023-11-05T08:32:49.635 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [ 2023-11-05T08:32:49.635 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:49.635 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es6 36/30 ECSubWrite(tid=33, reqid=osd.2.0:33, at_version=36'21, trim_to=0'0, roll_forward_to=34'19))", 2023-11-05T08:32:49.635 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:49.082925+0000", 2023-11-05T08:32:49.636 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 0.55244563599999996, 2023-11-05T08:32:49.636 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0014075069999999999, 2023-11-05T08:32:49.636 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:32:49.636 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:32:49.636 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:32:49.637 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:49.637 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:32:49.637 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.082925+0000", 2023-11-05T08:32:49.637 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:32:49.637 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:49.638 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:49.638 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:32:49.638 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.082925+0000", 2023-11-05T08:32:49.638 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:32:49.638 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:49.639 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:49.639 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:32:49.639 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.082925+0000", 2023-11-05T08:32:49.639 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.5400000000000002e-07 2023-11-05T08:32:49.639 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:49.640 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:49.640 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:32:49.640 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.082947+0000", 2023-11-05T08:32:49.640 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.2376999999999999e-05 2023-11-05T08:32:49.640 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:49.641 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:49.641 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:32:49.641 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.082961+0000", 2023-11-05T08:32:49.641 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.3910999999999999e-05 2023-11-05T08:32:49.641 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:49.641 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:49.642 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:32:49.642 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.082972+0000", 2023-11-05T08:32:49.642 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0732e-05 2023-11-05T08:32:49.642 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:49.642 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:49.643 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:32:49.643 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.083069+0000", 2023-11-05T08:32:49.643 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.6788999999999998e-05 2023-11-05T08:32:49.643 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:49.643 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:49.644 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:32:49.644 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.083083+0000", 2023-11-05T08:32:49.644 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4564e-05 2023-11-05T08:32:49.644 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:49.644 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:49.645 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:32:49.645 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.084287+0000", 2023-11-05T08:32:49.645 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001203771 2023-11-05T08:32:49.645 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:49.645 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:49.646 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:32:49.646 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.084332+0000", 2023-11-05T08:32:49.646 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.5108999999999998e-05 2023-11-05T08:32:49.646 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:32:49.646 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:32:49.647 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:32:49.647 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:49.647 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:49.647 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es6 36/30 ECSubWrite(tid=34, reqid=osd.2.0:33, at_version=0'0, trim_to=0'0, roll_forward_to=36'21))", 2023-11-05T08:32:49.647 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:49.089741+0000", 2023-11-05T08:32:49.648 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 0.54562994899999995, 2023-11-05T08:32:49.648 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0017028290000000001, 2023-11-05T08:32:49.648 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:32:49.648 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:32:49.648 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:32:49.649 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:49.649 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:32:49.649 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.089741+0000", 2023-11-05T08:32:49.649 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:32:49.649 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:49.650 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:49.650 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:32:49.650 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.089741+0000", 2023-11-05T08:32:49.650 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:32:49.650 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:49.651 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:49.651 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:32:49.651 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.089741+0000", 2023-11-05T08:32:49.651 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.6700000000000002e-07 2023-11-05T08:32:49.651 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:49.652 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:49.652 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:32:49.652 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.089772+0000", 2023-11-05T08:32:49.652 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.01e-05 2023-11-05T08:32:49.652 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:49.653 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:49.653 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:32:49.653 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.089792+0000", 2023-11-05T08:32:49.653 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.0593000000000001e-05 2023-11-05T08:32:49.653 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:49.653 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:49.654 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:32:49.654 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.089804+0000", 2023-11-05T08:32:49.654 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.1657999999999999e-05 2023-11-05T08:32:49.654 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:49.654 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:49.655 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:32:49.655 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.089890+0000", 2023-11-05T08:32:49.655 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.6456e-05 2023-11-05T08:32:49.655 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:49.655 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:49.655 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:32:49.656 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.089906+0000", 2023-11-05T08:32:49.656 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.5608999999999999e-05 2023-11-05T08:32:49.656 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:49.656 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:49.656 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:32:49.656 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.091383+0000", 2023-11-05T08:32:49.657 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001477396 2023-11-05T08:32:49.657 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:32:49.657 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:32:49.657 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:32:49.657 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.091443+0000", 2023-11-05T08:32:49.658 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.0050000000000003e-05 2023-11-05T08:32:49.658 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:32:49.658 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:32:49.658 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:32:49.658 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:32:49.658 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:32:49.659 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:32:49.660 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-11-05T08:32:49.661 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:32:49.656+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:32:49.761 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:32:49.759+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:32:49.795 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:32:49.795 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2023-11-05T08:32:49.795 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 0 2023-11-05T08:32:49.795 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:32:49.798 INFO:tasks.ceph.osd.0.smithi002.stderr:daemon-helper: command crashed with signal 6 2023-11-05T08:32:49.809 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-11-05T08:32:49.861 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:32:49.856+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:32:49.932 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:32:49.932 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2023-11-05T08:32:49.933 INFO:teuthology.orchestra.run.smithi053.stdout: "complaint_time": 30, 2023-11-05T08:32:49.933 INFO:teuthology.orchestra.run.smithi053.stdout: "num_blocked_ops": 0 2023-11-05T08:32:49.933 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:32:49.947 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-11-05T08:32:49.994 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:32:49.993+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:32:50.063 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:32:50.059+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:32:50.067 INFO:teuthology.orchestra.run.smithi002.stderr:nodeep-scrub is unset 2023-11-05T08:32:50.075 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:32:50.075 INFO:teuthology.orchestra.run.smithi053.stdout: "size": 20, 2023-11-05T08:32:50.075 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 600, 2023-11-05T08:32:50.075 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2023-11-05T08:32:50.075 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.076 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.6s2 35/30 ECSubWrite(tid=8, reqid=client.4691.0:225, at_version=35'7, trim_to=0'0, roll_forward_to=34'5))", 2023-11-05T08:32:50.076 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:48.092578+0000", 2023-11-05T08:32:50.076 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 1.978864299, 2023-11-05T08:32:50.076 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0056698920000000002, 2023-11-05T08:32:50.077 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:50.077 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:50.077 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:50.077 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.077 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:50.077 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.092578+0000", 2023-11-05T08:32:50.078 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:50.078 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.078 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.078 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:50.079 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.092578+0000", 2023-11-05T08:32:50.079 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:50.079 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.079 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.079 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:50.079 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.092579+0000", 2023-11-05T08:32:50.080 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.5000000000000002e-07 2023-11-05T08:32:50.080 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.080 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.080 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:50.080 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.092639+0000", 2023-11-05T08:32:50.081 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0288000000000003e-05 2023-11-05T08:32:50.081 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.081 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.081 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:50.081 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.092675+0000", 2023-11-05T08:32:50.081 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.6087999999999997e-05 2023-11-05T08:32:50.082 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.082 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.082 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:50.082 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.092700+0000", 2023-11-05T08:32:50.082 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5205999999999999e-05 2023-11-05T08:32:50.083 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.083 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.083 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:50.083 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.092915+0000", 2023-11-05T08:32:50.083 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00021489600000000001 2023-11-05T08:32:50.084 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.084 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.084 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:50.084 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.092932+0000", 2023-11-05T08:32:50.084 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6362999999999999e-05 2023-11-05T08:32:50.085 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.085 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.085 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:50.085 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.098160+0000", 2023-11-05T08:32:50.085 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0052284539999999996 2023-11-05T08:32:50.086 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.086 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.086 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:50.086 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.098248+0000", 2023-11-05T08:32:50.086 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.7847000000000001e-05 2023-11-05T08:32:50.087 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:50.087 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:50.087 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:50.087 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.087 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.088 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.6s2 35/30 ECSubWrite(tid=9, reqid=client.4691.0:226, at_version=35'8, trim_to=0'0, roll_forward_to=34'5))", 2023-11-05T08:32:50.088 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:48.093891+0000", 2023-11-05T08:32:50.088 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 1.97755112, 2023-11-05T08:32:50.088 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0051649729999999998, 2023-11-05T08:32:50.088 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:50.089 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:50.089 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:50.089 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.089 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:50.089 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.093891+0000", 2023-11-05T08:32:50.090 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:50.090 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.090 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.090 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:50.090 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.093891+0000", 2023-11-05T08:32:50.091 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:50.091 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.091 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.091 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:50.091 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.093893+0000", 2023-11-05T08:32:50.092 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5859999999999999e-06 2023-11-05T08:32:50.092 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.093 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.093 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:50.093 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.093943+0000", 2023-11-05T08:32:50.093 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.0028999999999998e-05 2023-11-05T08:32:50.093 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.094 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.094 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:50.094 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.093976+0000", 2023-11-05T08:32:50.094 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2869e-05 2023-11-05T08:32:50.094 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.095 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.095 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:50.095 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.094001+0000", 2023-11-05T08:32:50.095 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5678999999999999e-05 2023-11-05T08:32:50.095 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.095 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.096 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:50.096 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.095043+0000", 2023-11-05T08:32:50.096 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001041978 2023-11-05T08:32:50.096 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.096 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.097 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:50.097 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.095067+0000", 2023-11-05T08:32:50.097 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4076000000000001e-05 2023-11-05T08:32:50.097 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.097 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.098 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:50.098 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.098980+0000", 2023-11-05T08:32:50.098 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0039126609999999996 2023-11-05T08:32:50.098 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.098 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.098 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:50.099 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.099056+0000", 2023-11-05T08:32:50.099 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.6094999999999996e-05 2023-11-05T08:32:50.099 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:50.099 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:50.099 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:50.100 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.100 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.100 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.6s2 35/30 ECSubWrite(tid=10, reqid=client.4691.0:227, at_version=35'9, trim_to=0'0, roll_forward_to=34'5))", 2023-11-05T08:32:50.100 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:48.094938+0000", 2023-11-05T08:32:50.100 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 1.976503833, 2023-11-05T08:32:50.100 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0052729760000000004, 2023-11-05T08:32:50.101 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:50.101 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:50.101 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:50.101 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.101 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:50.101 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.094938+0000", 2023-11-05T08:32:50.102 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:50.102 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.102 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.102 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:50.102 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.094938+0000", 2023-11-05T08:32:50.102 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:50.103 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.103 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.103 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:50.103 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.094940+0000", 2023-11-05T08:32:50.103 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.68e-06 2023-11-05T08:32:50.103 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.104 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.104 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:50.104 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.094984+0000", 2023-11-05T08:32:50.104 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3458999999999998e-05 2023-11-05T08:32:50.104 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.105 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.105 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:50.105 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.095047+0000", 2023-11-05T08:32:50.105 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3324999999999994e-05 2023-11-05T08:32:50.105 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.106 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.106 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:50.106 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.095072+0000", 2023-11-05T08:32:50.106 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4919e-05 2023-11-05T08:32:50.106 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.106 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.107 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:50.107 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.097031+0000", 2023-11-05T08:32:50.107 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0019587739999999999 2023-11-05T08:32:50.107 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.107 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.107 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:50.108 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.097057+0000", 2023-11-05T08:32:50.108 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6142000000000001e-05 2023-11-05T08:32:50.108 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.108 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.108 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:50.108 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.100141+0000", 2023-11-05T08:32:50.109 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0030844599999999998 2023-11-05T08:32:50.109 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.109 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.109 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:50.109 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.100211+0000", 2023-11-05T08:32:50.109 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.0216999999999995e-05 2023-11-05T08:32:50.110 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:50.110 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:50.110 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:50.110 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.110 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.111 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.6s2 35/30 ECSubWrite(tid=11, reqid=client.4691.0:227, at_version=0'0, trim_to=0'0, roll_forward_to=35'9))", 2023-11-05T08:32:50.111 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:48.101494+0000", 2023-11-05T08:32:50.111 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 1.9699484570000001, 2023-11-05T08:32:50.111 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00236718, 2023-11-05T08:32:50.111 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:50.111 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:50.112 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:50.112 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.112 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:50.112 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.101494+0000", 2023-11-05T08:32:50.112 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:50.113 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.113 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.113 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:50.113 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.101494+0000", 2023-11-05T08:32:50.113 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:50.113 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.114 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.114 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:50.114 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.101496+0000", 2023-11-05T08:32:50.114 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.813e-06 2023-11-05T08:32:50.115 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.115 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.115 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:50.117 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.101517+0000", 2023-11-05T08:32:50.117 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1798999999999999e-05 2023-11-05T08:32:50.117 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.118 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.118 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:50.118 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.101546+0000", 2023-11-05T08:32:50.118 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8503000000000001e-05 2023-11-05T08:32:50.118 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.119 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.119 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:50.119 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.101561+0000", 2023-11-05T08:32:50.119 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.541e-05 2023-11-05T08:32:50.119 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.120 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.120 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:50.120 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.101719+0000", 2023-11-05T08:32:50.120 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00015761099999999999 2023-11-05T08:32:50.121 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.121 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.121 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:50.121 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.101756+0000", 2023-11-05T08:32:50.121 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.7005000000000002e-05 2023-11-05T08:32:50.122 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.122 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.122 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:50.122 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.103768+0000", 2023-11-05T08:32:50.123 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0020118319999999999 2023-11-05T08:32:50.123 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.123 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.123 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:50.123 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.103861+0000", 2023-11-05T08:32:50.124 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.3207000000000004e-05 2023-11-05T08:32:50.124 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:50.124 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:50.124 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:50.124 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.125 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.125 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpRead(3.6s2 36/30 ECSubRead(tid=12, to_read={3:6df625fb:::smithi09039252-31:head=0,417792,0}, subchunks={3:6df625fb:::smithi09039252-31:head=[0,1]}, attrs_to_read=))", 2023-11-05T08:32:50.125 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:49.062144+0000", 2023-11-05T08:32:50.125 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 1.0092978560000001, 2023-11-05T08:32:50.125 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0016005240000000001, 2023-11-05T08:32:50.126 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:50.126 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:50.126 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:50.126 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.126 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:50.126 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.062144+0000", 2023-11-05T08:32:50.127 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:50.127 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.127 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.127 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:50.127 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.062144+0000", 2023-11-05T08:32:50.128 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:50.128 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.128 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.128 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:50.128 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.062145+0000", 2023-11-05T08:32:50.128 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8299999999999997e-07 2023-11-05T08:32:50.129 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.129 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.129 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:50.129 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.062162+0000", 2023-11-05T08:32:50.129 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6843000000000001e-05 2023-11-05T08:32:50.129 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.130 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.130 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:50.130 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.062199+0000", 2023-11-05T08:32:50.130 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.7104999999999998e-05 2023-11-05T08:32:50.130 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.130 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.131 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:50.131 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.062227+0000", 2023-11-05T08:32:50.131 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7529e-05 2023-11-05T08:32:50.131 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.131 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.132 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:50.132 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.062449+0000", 2023-11-05T08:32:50.132 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00022217299999999999 2023-11-05T08:32:50.132 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.132 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.132 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:50.133 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.063745+0000", 2023-11-05T08:32:50.133 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001296291 2023-11-05T08:32:50.133 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:50.133 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:50.133 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:50.133 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.134 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.134 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpRead(3.6s2 36/30 ECSubRead(tid=13, to_read={3:6df625fb:::smithi09039252-31:head=0,417792,0}, subchunks={3:6df625fb:::smithi09039252-31:head=[0,1]}, attrs_to_read=))", 2023-11-05T08:32:50.134 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:49.062459+0000", 2023-11-05T08:32:50.134 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 1.008982837, 2023-11-05T08:32:50.134 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0019064749999999999, 2023-11-05T08:32:50.135 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:50.135 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:50.135 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:50.135 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.135 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:50.135 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.062459+0000", 2023-11-05T08:32:50.136 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:50.136 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.136 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.136 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:50.137 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.062459+0000", 2023-11-05T08:32:50.137 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:50.137 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.137 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.137 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:50.138 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.062477+0000", 2023-11-05T08:32:50.138 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7059999999999999e-05 2023-11-05T08:32:50.138 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.138 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.138 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:50.139 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.062497+0000", 2023-11-05T08:32:50.139 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.0018999999999999e-05 2023-11-05T08:32:50.139 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.139 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.139 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:50.140 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.062533+0000", 2023-11-05T08:32:50.140 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.6909000000000002e-05 2023-11-05T08:32:50.140 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.140 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.140 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:50.141 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.062562+0000", 2023-11-05T08:32:50.141 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8745999999999999e-05 2023-11-05T08:32:50.141 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.141 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.141 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:50.141 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.063943+0000", 2023-11-05T08:32:50.142 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0013805550000000001 2023-11-05T08:32:50.142 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.142 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.142 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:50.142 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.064366+0000", 2023-11-05T08:32:50.143 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00042318599999999998 2023-11-05T08:32:50.143 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:50.143 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:50.143 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:50.143 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.144 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.144 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpRead(3.6s2 36/30 ECSubRead(tid=14, to_read={3:6df625fb:::smithi09039252-31:head=0,417792,0}, subchunks={3:6df625fb:::smithi09039252-31:head=[0,1]}, attrs_to_read=))", 2023-11-05T08:32:50.144 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:49.063515+0000", 2023-11-05T08:32:50.144 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 1.0079269609999999, 2023-11-05T08:32:50.145 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0015363270000000001, 2023-11-05T08:32:50.145 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:50.145 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:50.145 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:50.145 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.146 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:50.146 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.063515+0000", 2023-11-05T08:32:50.146 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:50.146 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.147 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.147 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:50.147 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.063515+0000", 2023-11-05T08:32:50.147 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:50.147 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.148 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.148 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:50.148 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.063517+0000", 2023-11-05T08:32:50.148 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.807e-06 2023-11-05T08:32:50.148 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.149 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.149 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:50.149 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.063539+0000", 2023-11-05T08:32:50.149 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1942000000000001e-05 2023-11-05T08:32:50.149 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.149 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.150 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:50.150 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.063576+0000", 2023-11-05T08:32:50.150 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.6918999999999997e-05 2023-11-05T08:32:50.150 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.150 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.151 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:50.151 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.063603+0000", 2023-11-05T08:32:50.151 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6959000000000001e-05 2023-11-05T08:32:50.151 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.151 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.151 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:50.152 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.064468+0000", 2023-11-05T08:32:50.152 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00086499899999999998 2023-11-05T08:32:50.152 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.152 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.152 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:50.153 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.065052+0000", 2023-11-05T08:32:50.153 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00058370099999999995 2023-11-05T08:32:50.153 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:50.153 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:50.153 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:50.154 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.154 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.154 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.4s3 36/30 ECSubWrite(tid=10, reqid=osd.8.0:10, at_version=36'8, trim_to=0'0, roll_forward_to=34'6))", 2023-11-05T08:32:50.154 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:49.077632+0000", 2023-11-05T08:32:50.154 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.99381066600000001, 2023-11-05T08:32:50.155 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0045392519999999997, 2023-11-05T08:32:50.155 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:50.155 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:50.155 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:50.155 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.156 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:50.156 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.077632+0000", 2023-11-05T08:32:50.156 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:50.156 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.157 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.157 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:50.157 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.077632+0000", 2023-11-05T08:32:50.157 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:50.157 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.158 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.158 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:50.158 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.077632+0000", 2023-11-05T08:32:50.158 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.5600000000000004e-07 2023-11-05T08:32:50.159 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.159 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.159 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:50.159 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.077687+0000", 2023-11-05T08:32:50.159 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.4378e-05 2023-11-05T08:32:50.160 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.160 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.160 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:50.160 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.077720+0000", 2023-11-05T08:32:50.160 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3386000000000002e-05 2023-11-05T08:32:50.161 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.161 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.161 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:50.161 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.077759+0000", 2023-11-05T08:32:50.161 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8965999999999999e-05 2023-11-05T08:32:50.162 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.162 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.162 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:50.162 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.077926+0000", 2023-11-05T08:32:50.162 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000166847 2023-11-05T08:32:50.162 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.163 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.163 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:50.163 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.077951+0000", 2023-11-05T08:32:50.163 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4983000000000001e-05 2023-11-05T08:32:50.163 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.163 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.164 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:50.164 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.082094+0000", 2023-11-05T08:32:50.164 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0041433049999999999 2023-11-05T08:32:50.164 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.164 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.165 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:50.165 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.082171+0000", 2023-11-05T08:32:50.165 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.6531000000000003e-05 2023-11-05T08:32:50.165 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:50.165 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:50.165 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:50.166 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.166 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.166 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.8s3 36/30 ECSubWrite(tid=23, reqid=osd.5.0:23, at_version=36'15, trim_to=0'0, roll_forward_to=34'13))", 2023-11-05T08:32:50.166 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:49.078476+0000", 2023-11-05T08:32:50.166 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.99296678699999996, 2023-11-05T08:32:50.166 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0057340960000000002, 2023-11-05T08:32:50.167 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:50.167 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:50.167 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:50.167 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.167 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:50.167 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.078476+0000", 2023-11-05T08:32:50.168 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:50.168 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.168 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.168 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:50.168 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.078476+0000", 2023-11-05T08:32:50.168 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:50.169 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.169 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.169 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:50.169 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.078476+0000", 2023-11-05T08:32:50.169 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1299999999999998e-07 2023-11-05T08:32:50.170 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.170 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.170 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:50.170 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.078542+0000", 2023-11-05T08:32:50.170 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5121000000000002e-05 2023-11-05T08:32:50.170 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.171 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.171 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:50.171 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.078572+0000", 2023-11-05T08:32:50.171 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0812000000000002e-05 2023-11-05T08:32:50.171 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.172 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.172 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:50.172 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.078599+0000", 2023-11-05T08:32:50.172 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6653e-05 2023-11-05T08:32:50.172 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.172 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.173 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:50.173 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.078773+0000", 2023-11-05T08:32:50.173 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00017369099999999999 2023-11-05T08:32:50.173 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.173 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.174 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:50.174 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.078798+0000", 2023-11-05T08:32:50.174 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5153000000000001e-05 2023-11-05T08:32:50.174 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.174 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.175 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:50.175 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.084110+0000", 2023-11-05T08:32:50.175 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0053123190000000002 2023-11-05T08:32:50.175 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.175 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.175 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:50.176 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.084210+0000", 2023-11-05T08:32:50.176 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.9433999999999997e-05 2023-11-05T08:32:50.176 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:50.176 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:50.176 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:50.177 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.177 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.177 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es2 36/30 ECSubWrite(tid=33, reqid=osd.2.0:33, at_version=36'21, trim_to=0'0, roll_forward_to=34'19))", 2023-11-05T08:32:50.177 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:49.081011+0000", 2023-11-05T08:32:50.177 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.99043097599999996, 2023-11-05T08:32:50.178 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0040836860000000004, 2023-11-05T08:32:50.178 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:50.178 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:50.178 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:50.178 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.179 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:50.179 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.081011+0000", 2023-11-05T08:32:50.179 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:50.179 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.179 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.180 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:50.180 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.081011+0000", 2023-11-05T08:32:50.180 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:50.180 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.180 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.181 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:50.181 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.081012+0000", 2023-11-05T08:32:50.181 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.043e-06 2023-11-05T08:32:50.181 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.181 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.182 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:50.182 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.081065+0000", 2023-11-05T08:32:50.182 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3111e-05 2023-11-05T08:32:50.182 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.182 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.183 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:50.183 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.081090+0000", 2023-11-05T08:32:50.183 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4286000000000001e-05 2023-11-05T08:32:50.183 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.183 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.183 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:50.184 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.081111+0000", 2023-11-05T08:32:50.184 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1673000000000001e-05 2023-11-05T08:32:50.184 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.184 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.184 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:50.185 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.081297+0000", 2023-11-05T08:32:50.185 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00018547999999999999 2023-11-05T08:32:50.185 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.185 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.185 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:50.185 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.081322+0000", 2023-11-05T08:32:50.186 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5096000000000002e-05 2023-11-05T08:32:50.186 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.186 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.186 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:50.186 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.085031+0000", 2023-11-05T08:32:50.186 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0037094189999999998 2023-11-05T08:32:50.187 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.187 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.187 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:50.187 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.085095+0000", 2023-11-05T08:32:50.187 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3577999999999993e-05 2023-11-05T08:32:50.187 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:50.188 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:50.188 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:50.188 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.188 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.188 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.4s3 36/30 ECSubWrite(tid=11, reqid=osd.8.0:10, at_version=0'0, trim_to=0'0, roll_forward_to=36'8))", 2023-11-05T08:32:50.189 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:49.085928+0000", 2023-11-05T08:32:50.189 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.98551393099999995, 2023-11-05T08:32:50.189 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0023162009999999999, 2023-11-05T08:32:50.189 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:50.189 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:50.189 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:50.190 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.190 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:50.190 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.085928+0000", 2023-11-05T08:32:50.190 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:50.190 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.190 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.191 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:50.191 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.085928+0000", 2023-11-05T08:32:50.191 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:50.191 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.191 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.191 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:50.192 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.085930+0000", 2023-11-05T08:32:50.192 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5799999999999999e-06 2023-11-05T08:32:50.192 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.192 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.192 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:50.192 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.085955+0000", 2023-11-05T08:32:50.193 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4803e-05 2023-11-05T08:32:50.193 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.193 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.193 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:50.193 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.085982+0000", 2023-11-05T08:32:50.194 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7569e-05 2023-11-05T08:32:50.194 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.194 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.194 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:50.194 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.086001+0000", 2023-11-05T08:32:50.195 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.8875000000000001e-05 2023-11-05T08:32:50.195 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.195 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.195 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:50.196 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.086116+0000", 2023-11-05T08:32:50.196 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000114877 2023-11-05T08:32:50.196 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.196 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.196 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:50.196 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.086131+0000", 2023-11-05T08:32:50.197 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4496e-05 2023-11-05T08:32:50.197 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.197 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.197 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:50.198 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.088143+0000", 2023-11-05T08:32:50.198 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002012874 2023-11-05T08:32:50.198 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.198 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.198 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:50.199 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.088245+0000", 2023-11-05T08:32:50.199 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000101127 2023-11-05T08:32:50.199 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:50.199 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:50.199 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:50.200 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.200 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.200 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.8s3 36/30 ECSubWrite(tid=24, reqid=osd.5.0:23, at_version=0'0, trim_to=0'0, roll_forward_to=36'15))", 2023-11-05T08:32:50.200 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:49.087898+0000", 2023-11-05T08:32:50.200 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.98354397199999999, 2023-11-05T08:32:50.201 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0026013619999999999, 2023-11-05T08:32:50.201 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:50.201 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:50.201 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:50.201 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.201 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:50.202 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.087898+0000", 2023-11-05T08:32:50.202 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:50.202 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.202 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.202 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:50.203 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.087898+0000", 2023-11-05T08:32:50.203 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:50.203 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.203 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.203 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:50.203 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.087901+0000", 2023-11-05T08:32:50.204 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4360000000000001e-06 2023-11-05T08:32:50.204 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.204 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.204 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:50.204 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.087928+0000", 2023-11-05T08:32:50.205 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6846e-05 2023-11-05T08:32:50.205 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.205 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.205 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:50.205 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.087962+0000", 2023-11-05T08:32:50.206 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4289000000000003e-05 2023-11-05T08:32:50.206 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.206 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.206 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:50.206 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.087985+0000", 2023-11-05T08:32:50.207 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3419999999999999e-05 2023-11-05T08:32:50.207 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.207 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.207 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:50.208 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.088132+0000", 2023-11-05T08:32:50.208 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000146835 2023-11-05T08:32:50.208 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.208 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.208 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:50.209 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.088146+0000", 2023-11-05T08:32:50.209 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3626e-05 2023-11-05T08:32:50.209 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.209 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.209 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:50.209 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.090377+0000", 2023-11-05T08:32:50.210 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0022309539999999998 2023-11-05T08:32:50.210 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.210 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.210 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:50.210 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.090500+0000", 2023-11-05T08:32:50.211 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012295599999999999 2023-11-05T08:32:50.211 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:50.211 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:50.211 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:50.211 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.212 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.212 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es2 36/30 ECSubWrite(tid=34, reqid=osd.2.0:33, at_version=0'0, trim_to=0'0, roll_forward_to=36'21))", 2023-11-05T08:32:50.212 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:49.088858+0000", 2023-11-05T08:32:50.212 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.982583863, 2023-11-05T08:32:50.212 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.003812438, 2023-11-05T08:32:50.212 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:50.213 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:50.213 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:50.213 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.213 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:50.214 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.088858+0000", 2023-11-05T08:32:50.214 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:50.214 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.214 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.214 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:50.215 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.088858+0000", 2023-11-05T08:32:50.215 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:50.215 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.215 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.215 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:50.216 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.088860+0000", 2023-11-05T08:32:50.216 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.443e-06 2023-11-05T08:32:50.216 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.216 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.216 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:50.217 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.088888+0000", 2023-11-05T08:32:50.217 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7855999999999999e-05 2023-11-05T08:32:50.217 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.217 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.217 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:50.218 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.088920+0000", 2023-11-05T08:32:50.218 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2645000000000003e-05 2023-11-05T08:32:50.218 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.218 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.218 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:50.219 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.088942+0000", 2023-11-05T08:32:50.219 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1982999999999999e-05 2023-11-05T08:32:50.219 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.219 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.219 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:50.219 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.089102+0000", 2023-11-05T08:32:50.220 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00015976099999999999 2023-11-05T08:32:50.220 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.220 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.220 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:50.220 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.089127+0000", 2023-11-05T08:32:50.220 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4482999999999999e-05 2023-11-05T08:32:50.221 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.221 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.221 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:50.221 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.092565+0000", 2023-11-05T08:32:50.221 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0034379739999999999 2023-11-05T08:32:50.222 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:50.222 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:50.222 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:50.222 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.092671+0000", 2023-11-05T08:32:50.222 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010629299999999999 2023-11-05T08:32:50.222 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:50.223 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:50.223 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:50.223 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:50.223 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:50.223 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:32:50.224 INFO:tasks.rados.rados.0.smithi090.stdout:82: done (2 left) 2023-11-05T08:32:50.224 INFO:tasks.rados.rados.0.smithi090.stdout:83: done (1 left) 2023-11-05T08:32:50.224 INFO:tasks.rados.rados.0.smithi090.stdout:84: done (0 left) 2023-11-05T08:32:50.224 INFO:tasks.rados.rados.0.smithi090.stdout:85: delete oid 50 current snap is 3 2023-11-05T08:32:50.226 INFO:tasks.rados.rados.0.smithi090.stdout:85: done (0 left) 2023-11-05T08:32:50.226 INFO:tasks.rados.rados.0.smithi090.stdout:86: rollback oid 17 current snap is 3 2023-11-05T08:32:50.227 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 17 to 1 2023-11-05T08:32:50.227 INFO:tasks.rados.rados.0.smithi090.stdout:87: rollback oid 30 current snap is 3 2023-11-05T08:32:50.227 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 30 to 2 2023-11-05T08:32:50.227 INFO:tasks.rados.rados.0.smithi090.stdout:88: read oid 49 snap -1 2023-11-05T08:32:50.227 INFO:tasks.rados.rados.0.smithi090.stdout:88: expect (ObjNum 48 snap 0 seq_num 48) 2023-11-05T08:32:50.228 INFO:tasks.rados.rados.0.smithi090.stdout:89: snap_create 2023-11-05T08:32:50.228 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:32:50.220+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:32:50.229 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-11-05T08:32:50.312 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-05T08:32:50.327 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:32:50.324+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:32:50.400 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:32:50.400 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2023-11-05T08:32:50.400 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 0 2023-11-05T08:32:50.400 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:32:50.414 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-11-05T08:32:50.427 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:32:50.423+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:32:50.458 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-11-05T08:32:50.476 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1: {} 2023-11-05T08:32:50.477 INFO:teuthology.orchestra.run.smithi002.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-11-05T08:32:50.497 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2: {} 2023-11-05T08:32:50.497 INFO:teuthology.orchestra.run.smithi002.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-11-05T08:32:50.517 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3: {} 2023-11-05T08:32:50.518 INFO:teuthology.orchestra.run.smithi002.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-11-05T08:32:50.528 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:32:50.525+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:32:50.533 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:32:50.533 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2023-11-05T08:32:50.533 INFO:teuthology.orchestra.run.smithi053.stdout: "complaint_time": 30, 2023-11-05T08:32:50.533 INFO:teuthology.orchestra.run.smithi053.stdout: "num_blocked_ops": 0 2023-11-05T08:32:50.534 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:32:50.535 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4: {} 2023-11-05T08:32:50.536 INFO:teuthology.orchestra.run.smithi002.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-11-05T08:32:50.548 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-11-05T08:32:50.553 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5: {} 2023-11-05T08:32:50.554 INFO:teuthology.orchestra.run.smithi002.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-11-05T08:32:50.575 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6: {} 2023-11-05T08:32:50.575 INFO:teuthology.orchestra.run.smithi002.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-11-05T08:32:50.595 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7: {} 2023-11-05T08:32:50.596 INFO:teuthology.orchestra.run.smithi002.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-11-05T08:32:50.615 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8: {} 2023-11-05T08:32:50.615 INFO:teuthology.orchestra.run.smithi002.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-11-05T08:32:50.628 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:32:50.627+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:32:50.635 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9: {} 2023-11-05T08:32:50.635 INFO:teuthology.orchestra.run.smithi002.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-11-05T08:32:50.657 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10: {} 2023-11-05T08:32:50.657 INFO:teuthology.orchestra.run.smithi002.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-11-05T08:32:50.680 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11: {} 2023-11-05T08:32:50.680 INFO:teuthology.orchestra.run.smithi002.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-11-05T08:32:50.700 INFO:teuthology.orchestra.run.smithi002.stdout:osd.12: {} 2023-11-05T08:32:50.700 INFO:teuthology.orchestra.run.smithi002.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-11-05T08:32:50.722 INFO:teuthology.orchestra.run.smithi002.stdout:osd.13: {} 2023-11-05T08:32:50.723 INFO:teuthology.orchestra.run.smithi002.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-11-05T08:32:50.723 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:50.724 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:50.730 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:32:50.729+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:32:50.738 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2023-11-05T08:32:50.743 INFO:teuthology.orchestra.run.smithi002.stdout:osd.14: {} 2023-11-05T08:32:50.743 INFO:teuthology.orchestra.run.smithi002.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-11-05T08:32:50.763 INFO:teuthology.orchestra.run.smithi002.stdout:osd.15: {} 2023-11-05T08:32:50.764 INFO:teuthology.orchestra.run.smithi002.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-11-05T08:32:50.769 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-05T08:32:50.769 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-05T08:32:50.831 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:32:50.827+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:32:50.915 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:50.915 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:50.929 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2023-11-05T08:32:50.931 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:32:50.929+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:32:51.031 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-05T08:32:51.056 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2023-11-05T08:32:51.056 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-11-05T08:32:51.057 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2023-11-05T08:32:51.057 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-05T08:32:51.058 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:51.058 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:51.071 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2023-11-05T08:32:51.083 INFO:tasks.rados.rados.0.smithi090.stdout:89: done (3 left) 2023-11-05T08:32:51.084 INFO:tasks.rados.rados.0.smithi090.stdout:90: snap_create 2023-11-05T08:32:51.110 INFO:tasks.rados.rados.0.smithi090.stdout:86: finishing rollback tid 0 to smithi09039252-17 2023-11-05T08:32:51.114 INFO:tasks.rados.rados.0.smithi090.stdout:86: finishing rollback tid 1 to smithi09039252-17 2023-11-05T08:32:51.114 INFO:tasks.rados.rados.0.smithi090.stdout:87: finishing rollback tid 0 to smithi09039252-30 2023-11-05T08:32:51.114 INFO:tasks.rados.rados.0.smithi090.stdout:86: finishing rollback tid 2 to smithi09039252-17 2023-11-05T08:32:51.114 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 17 v 8 (ObjNum 16 snap 0 seq_num 16) dirty exists 2023-11-05T08:32:51.116 INFO:tasks.rados.rados.0.smithi090.stdout:87: finishing rollback tid 1 to smithi09039252-30 2023-11-05T08:32:51.117 INFO:tasks.rados.rados.0.smithi090.stdout:87: finishing rollback tid 2 to smithi09039252-30 2023-11-05T08:32:51.117 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 30 v 21 (ObjNum 29 snap 0 seq_num 29) dirty exists 2023-11-05T08:32:51.120 INFO:tasks.rados.rados.0.smithi090.stdout:88: expect (ObjNum 48 snap 0 seq_num 48) 2023-11-05T08:32:51.134 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:32:51.133+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:32:51.198 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:51.198 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:51.213 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2023-11-05T08:32:51.235 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:32:51.235+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:32:51.336 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:32:51.335+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:32:51.380 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:32:51.381 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-11-05T08:32:51.382 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:51.382 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:51.397 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2023-11-05T08:32:51.399 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2023-11-05T08:32:51.079250+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30851616,"num_objects":57,"num_object_clones":5,"num_object_copies":389,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":270,"num_read_kb":4263,"num_write":359,"num_write_kb":33588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":300,"ondisk_log_size":300,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":88,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":483948,"kb_used_data":51608,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509465492,"statfs":{"total":1546188226560,"available":1545692663808,"internally_reserved":0,"allocated":52846592,"data_stored":51692343,"data_compressed":362923,"data_compressed_allocated":872448,"data_compressed_original":2637824,"omap_allocated":25505,"internal_metadata":442473567},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29753344,"num_objects":54,"num_object_clones":5,"num_object_copies":378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":132,"num_read_kb":4144,"num_write":154,"num_write_kb":29632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.061198"},"pg_stats":[{"pgid":"3.f","version":"33'11","reported_seq":36,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-05T08:32:47.061773+0000","last_change":"2023-11-05T08:32:44.110396+0000","last_active":"2023-11-05T08:32:47.061773+0000","last_peered":"2023-11-05T08:32:47.061773+0000","last_clean":"2023-11-05T08:32:47.061773+0000","last_became_active":"2023-11-05T08:32:44.107669+0000","last_became_peered":"2023-11-05T08:32:44.107669+0000","last_unstale":"2023-11-05T08:32:47.061773+0000","last_undegraded":"2023-11-05T08:32:47.061773+0000","last_fullsized":"2023-11-05T08:32:47.061773+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:48.298843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3227648,"num_objects":4,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":9,"num_read_kb":5,"num_write":11,"num_write_kb":3152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,14,13,8,10,0],"acting":[11,2,14,13,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"34'19","reported_seq":46,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-05T08:32:47.080645+0000","last_change":"2023-11-05T08:32:44.950003+0000","last_active":"2023-11-05T08:32:47.080645+0000","last_peered":"2023-11-05T08:32:47.080645+0000","last_clean":"2023-11-05T08:32:47.080645+0000","last_became_active":"2023-11-05T08:32:44.949185+0000","last_became_peered":"2023-11-05T08:32:44.949185+0000","last_unstale":"2023-11-05T08:32:47.080645+0000","last_undegraded":"2023-11-05T08:32:47.080645+0000","last_fullsized":"2023-11-05T08:32:47.080645+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:01.467376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3833856,"num_objects":7,"num_object_clones":1,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":14,"num_read_kb":8,"num_write":18,"num_write_kb":3744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1,11,8,6,12],"acting":[2,9,1,11,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"34'8","reported_seq":30,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-05T08:32:47.136356+0000","last_change":"2023-11-05T08:32:44.109001+0000","last_active":"2023-11-05T08:32:47.136356+0000","last_peered":"2023-11-05T08:32:47.136356+0000","last_clean":"2023-11-05T08:32:47.136356+0000","last_became_active":"2023-11-05T08:32:44.106593+0000","last_became_peered":"2023-11-05T08:32:44.106593+0000","last_unstale":"2023-11-05T08:32:47.136356+0000","last_undegraded":"2023-11-05T08:32:47.136356+0000","last_fullsized":"2023-11-05T08:32:47.136356+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:47.237860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1556480,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":7,"num_write_kb":1520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,0,8,4],"acting":[7,3,14,10,0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"33'2","reported_seq":23,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-05T08:32:47.061392+0000","last_change":"2023-11-05T08:32:44.556288+0000","last_active":"2023-11-05T08:32:47.061392+0000","last_peered":"2023-11-05T08:32:47.061392+0000","last_clean":"2023-11-05T08:32:47.061392+0000","last_became_active":"2023-11-05T08:32:44.554077+0000","last_became_peered":"2023-11-05T08:32:44.554077+0000","last_unstale":"2023-11-05T08:32:47.061392+0000","last_undegraded":"2023-11-05T08:32:47.061392+0000","last_fullsized":"2023-11-05T08:32:47.061392+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:03.173834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":311296,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,0,10,6,12,5],"acting":[11,8,0,10,6,12,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"34'12","reported_seq":37,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-05T08:32:47.052132+0000","last_change":"2023-11-05T08:32:44.948306+0000","last_active":"2023-11-05T08:32:47.052132+0000","last_peered":"2023-11-05T08:32:47.052132+0000","last_clean":"2023-11-05T08:32:47.052132+0000","last_became_active":"2023-11-05T08:32:44.946070+0000","last_became_peered":"2023-11-05T08:32:44.946070+0000","last_unstale":"2023-11-05T08:32:47.052132+0000","last_undegraded":"2023-11-05T08:32:47.052132+0000","last_fullsized":"2023-11-05T08:32:47.052132+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:02.443240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2326528,"num_objects":4,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":12,"num_write_kb":2592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1,12,6,7,9],"acting":[4,0,1,12,6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"34'4","reported_seq":22,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-05T08:32:47.084381+0000","last_change":"2023-11-05T08:32:47.068277+0000","last_active":"2023-11-05T08:32:47.084381+0000","last_peered":"2023-11-05T08:32:47.084381+0000","last_clean":"2023-11-05T08:32:47.084381+0000","last_became_active":"2023-11-05T08:32:47.064950+0000","last_became_peered":"2023-11-05T08:32:47.064950+0000","last_unstale":"2023-11-05T08:32:47.084381+0000","last_undegraded":"2023-11-05T08:32:47.084381+0000","last_fullsized":"2023-11-05T08:32:47.084381+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:46.108924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966656,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":4,"num_write_kb":944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,9,14,6,11],"acting":[3,7,1,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"33'18","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-05T08:32:47.069478+0000","last_change":"2023-11-05T08:32:44.948205+0000","last_active":"2023-11-05T08:32:47.069478+0000","last_peered":"2023-11-05T08:32:47.069478+0000","last_clean":"2023-11-05T08:32:47.069478+0000","last_became_active":"2023-11-05T08:32:44.947415+0000","last_became_peered":"2023-11-05T08:32:44.947415+0000","last_unstale":"2023-11-05T08:32:47.069478+0000","last_undegraded":"2023-11-05T08:32:47.069478+0000","last_fullsized":"2023-11-05T08:32:47.069478+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:45.245507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4636672,"num_objects":7,"num_object_clones":0,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":19,"num_read_kb":1737,"num_write":18,"num_write_kb":4528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,0,3,15,8,10],"acting":[14,1,0,3,15,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.8","version":"34'13","reported_seq":37,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-05T08:32:47.125338+0000","last_change":"2023-11-05T08:32:44.952691+0000","last_active":"2023-11-05T08:32:47.125338+0000","last_peered":"2023-11-05T08:32:47.125338+0000","last_clean":"2023-11-05T08:32:47.125338+0000","last_became_active":"2023-11-05T08:32:44.947599+0000","last_became_peered":"2023-11-05T08:32:44.947599+0000","last_unstale":"2023-11-05T08:32:47.125338+0000","last_undegraded":"2023-11-05T08:32:47.125338+0000","last_fullsized":"2023-11-05T08:32:47.125338+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:57.141665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2523136,"num_objects":5,"num_object_clones":1,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":5,"num_write":31,"num_write_kb":2464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2,1,14,15,6],"acting":[5,0,2,1,14,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"25'157","reported_seq":218,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-05T08:32:47.043696+0000","last_change":"2023-11-05T08:32:28.761556+0000","last_active":"2023-11-05T08:32:47.043696+0000","last_peered":"2023-11-05T08:32:47.043696+0000","last_clean":"2023-11-05T08:32:47.043696+0000","last_became_active":"2023-11-05T08:32:28.758834+0000","last_became_peered":"2023-11-05T08:32:28.758834+0000","last_unstale":"2023-11-05T08:32:47.043696+0000","last_undegraded":"2023-11-05T08:32:47.043696+0000","last_fullsized":"2023-11-05T08:32:47.043696+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:27.693099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:27.693099+0000","last_clean_scrub_stamp":"2023-11-05T08:32:27.693099+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:40.584365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-05T08:32:46.019151+0000","last_change":"2023-11-05T08:32:31.940561+0000","last_active":"2023-11-05T08:32:46.019151+0000","last_peered":"2023-11-05T08:32:46.019151+0000","last_clean":"2023-11-05T08:32:46.019151+0000","last_became_active":"2023-11-05T08:32:31.939705+0000","last_became_peered":"2023-11-05T08:32:31.939705+0000","last_unstale":"2023-11-05T08:32:46.019151+0000","last_undegraded":"2023-11-05T08:32:46.019151+0000","last_fullsized":"2023-11-05T08:32:46.019151+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:54.088838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"33'5","reported_seq":30,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-11-05T08:32:48.048223+0000","last_change":"2023-11-05T08:32:44.952601+0000","last_active":"2023-11-05T08:32:48.048223+0000","last_peered":"2023-11-05T08:32:48.048223+0000","last_clean":"2023-11-05T08:32:48.048223+0000","last_became_active":"2023-11-05T08:32:44.947690+0000","last_became_peered":"2023-11-05T08:32:44.947690+0000","last_unstale":"2023-11-05T08:32:48.048223+0000","last_undegraded":"2023-11-05T08:32:48.048223+0000","last_fullsized":"2023-11-05T08:32:48.048223+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:46.802475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":950272,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":5,"num_write_kb":928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,3,6,15,1,11],"acting":[13,2,3,6,15,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":4,"reported_epoch":33,"state":"peering","last_fresh":"2023-11-05T08:32:46.036560+0000","last_change":"2023-11-05T08:32:46.036560+0000","last_active":"2023-11-05T08:32:30.915125+0000","last_peered":"2023-11-05T08:32:30.915125+0000","last_clean":"2023-11-05T08:32:30.915125+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-05T08:32:46.036560+0000","last_undegraded":"2023-11-05T08:32:46.036560+0000","last_fullsized":"2023-11-05T08:32:46.036560+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"33'11","reported_seq":41,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-05T08:32:47.036691+0000","last_change":"2023-11-05T08:32:44.109321+0000","last_active":"2023-11-05T08:32:47.036691+0000","last_peered":"2023-11-05T08:32:47.036691+0000","last_clean":"2023-11-05T08:32:47.036691+0000","last_became_active":"2023-11-05T08:32:44.107839+0000","last_became_peered":"2023-11-05T08:32:44.107839+0000","last_unstale":"2023-11-05T08:32:47.036691+0000","last_undegraded":"2023-11-05T08:32:47.036691+0000","last_fullsized":"2023-11-05T08:32:47.036691+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:51.549400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1703936,"num_objects":5,"num_object_clones":0,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":198,"num_write":11,"num_write_kb":1696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,5,10,2,13],"acting":[4,11,9,5,10,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-05T08:32:47.043789+0000","last_change":"2023-11-05T08:32:31.941357+0000","last_active":"2023-11-05T08:32:47.043789+0000","last_peered":"2023-11-05T08:32:47.043789+0000","last_clean":"2023-11-05T08:32:47.043789+0000","last_became_active":"2023-11-05T08:32:31.940564+0000","last_became_peered":"2023-11-05T08:32:31.940564+0000","last_unstale":"2023-11-05T08:32:47.043789+0000","last_undegraded":"2023-11-05T08:32:47.043789+0000","last_fullsized":"2023-11-05T08:32:47.043789+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:58.168161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.0","version":"33'9","reported_seq":40,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-05T08:32:47.043502+0000","last_change":"2023-11-05T08:32:44.954966+0000","last_active":"2023-11-05T08:32:47.043502+0000","last_peered":"2023-11-05T08:32:47.043502+0000","last_clean":"2023-11-05T08:32:47.043502+0000","last_became_active":"2023-11-05T08:32:44.953085+0000","last_became_peered":"2023-11-05T08:32:44.953085+0000","last_unstale":"2023-11-05T08:32:47.043502+0000","last_undegraded":"2023-11-05T08:32:47.043502+0000","last_fullsized":"2023-11-05T08:32:47.043502+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:02.668718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1605632,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":581,"num_write":9,"num_write_kb":1792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,15,1,5,6,4],"acting":[14,2,15,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-05T08:32:47.069496+0000","last_change":"2023-11-05T08:32:47.068689+0000","last_active":"2023-11-05T08:32:47.069496+0000","last_peered":"2023-11-05T08:32:47.069496+0000","last_clean":"2023-11-05T08:32:47.069496+0000","last_became_active":"2023-11-05T08:32:47.068343+0000","last_became_peered":"2023-11-05T08:32:47.068343+0000","last_unstale":"2023-11-05T08:32:47.069496+0000","last_undegraded":"2023-11-05T08:32:47.069496+0000","last_fullsized":"2023-11-05T08:32:47.069496+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:37.946832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3],"acting":[12,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.3","version":"35'10","reported_seq":34,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-11-05T08:32:48.062757+0000","last_change":"2023-11-05T08:32:44.039941+0000","last_active":"2023-11-05T08:32:48.062757+0000","last_peered":"2023-11-05T08:32:48.062757+0000","last_clean":"2023-11-05T08:32:48.062757+0000","last_became_active":"2023-11-05T08:32:44.037532+0000","last_became_peered":"2023-11-05T08:32:44.037532+0000","last_unstale":"2023-11-05T08:32:48.062757+0000","last_undegraded":"2023-11-05T08:32:48.062757+0000","last_fullsized":"2023-11-05T08:32:48.062757+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:00.990727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":8,"num_write_kb":1536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,4,15,2,5,3],"acting":[13,7,4,15,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.5","version":"33'9","reported_seq":36,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-05T08:32:47.040475+0000","last_change":"2023-11-05T08:32:44.554170+0000","last_active":"2023-11-05T08:32:47.040475+0000","last_peered":"2023-11-05T08:32:47.040475+0000","last_clean":"2023-11-05T08:32:47.040475+0000","last_became_active":"2023-11-05T08:32:44.551939+0000","last_became_peered":"2023-11-05T08:32:44.551939+0000","last_unstale":"2023-11-05T08:32:47.040475+0000","last_undegraded":"2023-11-05T08:32:47.040475+0000","last_fullsized":"2023-11-05T08:32:47.040475+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:54.739050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2228224,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":1589,"num_write":9,"num_write_kb":2176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,4,13,7,0,6],"acting":[15,8,4,13,7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-05T08:32:46.029476+0000","last_change":"2023-11-05T08:32:32.253735+0000","last_active":"2023-11-05T08:32:46.029476+0000","last_peered":"2023-11-05T08:32:46.029476+0000","last_clean":"2023-11-05T08:32:46.029476+0000","last_became_active":"2023-11-05T08:32:32.252836+0000","last_became_peered":"2023-11-05T08:32:32.252836+0000","last_unstale":"2023-11-05T08:32:46.029476+0000","last_undegraded":"2023-11-05T08:32:46.029476+0000","last_fullsized":"2023-11-05T08:32:46.029476+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:57.123923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"34'6","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-05T08:32:47.137224+0000","last_change":"2023-11-05T08:32:44.950386+0000","last_active":"2023-11-05T08:32:47.137224+0000","last_peered":"2023-11-05T08:32:47.137224+0000","last_clean":"2023-11-05T08:32:47.137224+0000","last_became_active":"2023-11-05T08:32:44.948602+0000","last_became_peered":"2023-11-05T08:32:44.948602+0000","last_unstale":"2023-11-05T08:32:47.137224+0000","last_undegraded":"2023-11-05T08:32:47.137224+0000","last_fullsized":"2023-11-05T08:32:47.137224+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:55.408135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1671168,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":5,"num_write_kb":1632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,4,1,0,14,13],"acting":[8,2,4,1,0,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-05T08:32:45.329451+0000","last_change":"2023-11-05T08:32:31.940904+0000","last_active":"2023-11-05T08:32:45.329451+0000","last_peered":"2023-11-05T08:32:45.329451+0000","last_clean":"2023-11-05T08:32:45.329451+0000","last_became_active":"2023-11-05T08:32:31.940312+0000","last_became_peered":"2023-11-05T08:32:31.940312+0000","last_unstale":"2023-11-05T08:32:45.329451+0000","last_undegraded":"2023-11-05T08:32:45.329451+0000","last_fullsized":"2023-11-05T08:32:45.329451+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:43.223063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"33'4","reported_seq":27,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-05T08:32:47.036642+0000","last_change":"2023-11-05T08:32:44.557497+0000","last_active":"2023-11-05T08:32:47.036642+0000","last_peered":"2023-11-05T08:32:47.036642+0000","last_clean":"2023-11-05T08:32:47.036642+0000","last_became_active":"2023-11-05T08:32:44.555262+0000","last_became_peered":"2023-11-05T08:32:44.555262+0000","last_unstale":"2023-11-05T08:32:47.036642+0000","last_undegraded":"2023-11-05T08:32:47.036642+0000","last_fullsized":"2023-11-05T08:32:47.036642+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:43.326614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":638976,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,0,8,5,11],"acting":[4,6,9,0,8,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-05T08:32:46.037975+0000","last_change":"2023-11-05T08:32:31.941613+0000","last_active":"2023-11-05T08:32:46.037975+0000","last_peered":"2023-11-05T08:32:46.037975+0000","last_clean":"2023-11-05T08:32:46.037975+0000","last_became_active":"2023-11-05T08:32:31.941207+0000","last_became_peered":"2023-11-05T08:32:31.941207+0000","last_unstale":"2023-11-05T08:32:46.037975+0000","last_undegraded":"2023-11-05T08:32:46.037975+0000","last_fullsized":"2023-11-05T08:32:46.037975+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:44.932076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.6","version":"33'2","reported_seq":21,"reported_epoch":33,"state":"stale+active+clean","last_fresh":"2023-11-05T08:32:46.320571+0000","last_change":"2023-11-05T08:32:44.950794+0000","last_active":"2023-11-05T08:32:46.320571+0000","last_peered":"2023-11-05T08:32:46.320571+0000","last_clean":"2023-11-05T08:32:46.320571+0000","last_became_active":"2023-11-05T08:32:44.949221+0000","last_became_peered":"2023-11-05T08:32:44.949221+0000","last_unstale":"2023-11-05T08:32:50.070216+0000","last_undegraded":"2023-11-05T08:32:46.320571+0000","last_fullsized":"2023-11-05T08:32:46.320571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:53.806668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":507904,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,15,12,7,9],"acting":[0,13,1,15,12,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-05T08:32:46.029087+0000","last_change":"2023-11-05T08:32:31.942607+0000","last_active":"2023-11-05T08:32:46.029087+0000","last_peered":"2023-11-05T08:32:46.029087+0000","last_clean":"2023-11-05T08:32:46.029087+0000","last_became_active":"2023-11-05T08:32:31.941858+0000","last_became_peered":"2023-11-05T08:32:31.941858+0000","last_unstale":"2023-11-05T08:32:46.029087+0000","last_undegraded":"2023-11-05T08:32:46.029087+0000","last_fullsized":"2023-11-05T08:32:46.029087+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:48.710632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":30261248,"num_objects":55,"num_object_clones":5,"num_object_copies":385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":134,"num_read_kb":4145,"num_write":156,"num_write_kb":30128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":54022144,"data_stored":57704448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143,"ondisk_log_size":143,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":13898,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":22,"seq":94489280519,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30720,"kb_used_data":3696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341120,"statfs":{"total":96636764160,"available":96605306880,"internally_reserved":0,"allocated":3784704,"data_stored":4264511,"data_compressed":30048,"data_compressed_allocated":86016,"data_compressed_original":618496,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.244}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5189999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0880000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83999999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.333}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5680000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.869}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.673}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6000000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.917}]}]},{"osd":15,"up_from":21,"seq":90194313223,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31192,"kb_used_data":4168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340648,"statfs":{"total":96636764160,"available":96604823552,"internally_reserved":0,"allocated":4268032,"data_stored":4222421,"data_compressed":23099,"data_compressed_allocated":53248,"data_compressed_original":106496,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,4,5,6,7,8,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.131}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96399999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4649999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3240000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1559999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0029999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5030000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.169}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3610000000000002}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5760000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.397}]}]},{"osd":14,"up_from":22,"seq":94489280518,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30572,"kb_used_data":3552,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341268,"statfs":{"total":96636764160,"available":96605458432,"internally_reserved":0,"allocated":3637248,"data_stored":3475498,"data_compressed":21230,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]}]},{"osd":0,"up_from":21,"seq":90194313223,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":4392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":4497408,"data_stored":4339168,"data_compressed":21230,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,9,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]}]},{"osd":3,"up_from":21,"seq":90194313223,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29228,"kb_used_data":2208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342612,"statfs":{"total":96636764160,"available":96606834688,"internally_reserved":0,"allocated":2260992,"data_stored":2099242,"data_compressed":21230,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.024}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96199999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2749999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.194}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91300000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4419999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.333}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.393}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.357}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.478}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.502}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.361}]}]},{"osd":6,"up_from":21,"seq":90194313222,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30392,"kb_used_data":3372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341448,"statfs":{"total":96636764160,"available":96605642752,"internally_reserved":0,"allocated":3452928,"data_stored":3294688,"data_compressed":21230,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[3,4,5,7,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1879999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.298}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2729999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4530000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.899}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6479999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6859999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.734}]}]},{"osd":1,"up_from":22,"seq":94489280518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31040,"kb_used_data":4020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340800,"statfs":{"total":96636764160,"available":96604979200,"internally_reserved":0,"allocated":4116480,"data_stored":3954730,"data_compressed":21230,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.018}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92400000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.871}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.268}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.181}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.613}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.306}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.216}]}]},{"osd":2,"up_from":22,"seq":94489280518,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31088,"kb_used_data":4068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340752,"statfs":{"total":96636764160,"available":96604930048,"internally_reserved":0,"allocated":4165632,"data_stored":4479562,"data_compressed":28179,"data_compressed_allocated":81920,"data_compressed_original":610304,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.236}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8839999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82199999999999995}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91600000000000004}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]}]},{"osd":4,"up_from":22,"seq":94489280518,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29868,"kb_used_data":2848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341972,"statfs":{"total":96636764160,"available":96606179328,"internally_reserved":0,"allocated":2916352,"data_stored":2758112,"data_compressed":21230,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.234}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.141}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.587}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.02}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.623}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.508}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.097}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.466}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7050000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8320000000000001}]}]},{"osd":5,"up_from":22,"seq":94489280518,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29388,"kb_used_data":2368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342452,"statfs":{"total":96636764160,"available":96606670848,"internally_reserved":0,"allocated":2424832,"data_stored":2263082,"data_compressed":21230,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,4,6,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0840000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80600000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1279999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2330000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98999999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86899999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3100000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.266}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.599}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5169999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.46}]}]},{"osd":7,"up_from":22,"seq":94489280519,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28868,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342972,"statfs":{"total":96636764160,"available":96607203328,"internally_reserved":0,"allocated":1892352,"data_stored":1730602,"data_compressed":21230,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,3,4,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.073}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79800000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1759999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.464}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2150000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.032}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5920000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.115}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5249999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4450000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3200000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.631}]}]},{"osd":8,"up_from":22,"seq":94489280518,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31036,"kb_used_data":4016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340804,"statfs":{"total":96636764160,"available":96604983296,"internally_reserved":0,"allocated":4112384,"data_stored":3950634,"data_compressed":21230,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,4,5,6,7,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3069999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0589999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97799999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89500000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4730000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.53}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.411}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]}]},{"osd":9,"up_from":21,"seq":90194313222,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30208,"kb_used_data":3184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341632,"statfs":{"total":96636764160,"available":96605831168,"internally_reserved":0,"allocated":3260416,"data_stored":3145759,"data_compressed":23099,"data_compressed_allocated":53248,"data_compressed_original":106496,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,3,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.141}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.407}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.458}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.569}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.631}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7809999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5449999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.306}]}]},{"osd":10,"up_from":22,"seq":94489280518,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30156,"kb_used_data":3132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341684,"statfs":{"total":96636764160,"available":96605884416,"internally_reserved":0,"allocated":3207168,"data_stored":3137567,"data_compressed":23099,"data_compressed_allocated":53248,"data_compressed_original":106496,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.212}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.073}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3109999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8199999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6779999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.601}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.893}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.2170000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.573}]}]},{"osd":11,"up_from":21,"seq":90194313223,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30204,"kb_used_data":3180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341636,"statfs":{"total":96636764160,"available":96605835264,"internally_reserved":0,"allocated":3256320,"data_stored":3141663,"data_compressed":23099,"data_compressed_allocated":53248,"data_compressed_original":106496,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,5,6,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.079}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94799999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89900000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2390000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3220000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.28}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.137}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4590000000000001}]}]},{"osd":12,"up_from":22,"seq":94489280518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28576,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343264,"statfs":{"total":96636764160,"available":96607502336,"internally_reserved":0,"allocated":1593344,"data_stored":1435104,"data_compressed":21230,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,6,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0309999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.427}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6499999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.609}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.218}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2529999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5069999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3819999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6949,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6949,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4165632,"data_stored":4165632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5173248,"data_stored":5505024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4272128,"data_stored":4603904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1925120,"data_stored":1925120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3325952,"data_stored":3588096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2088960,"data_stored":2203648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3121152,"data_stored":3121152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2707456,"data_stored":2801664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4526080,"data_stored":4837376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2912256,"data_stored":2961408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2859008,"data_stored":2953216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2908160,"data_stored":2957312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":2162688,"data_stored":2211840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":3903488,"data_stored":5406720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":4046848,"data_stored":4423680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":3923968,"data_stored":4038656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-05T08:32:51.403 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2023-11-05T08:32:51.403 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 26 2023-11-05T08:32:51.443 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:32:51.443+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:32:51.521 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:51.521 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:51.535 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2023-11-05T08:32:51.537 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-11-05T08:32:51.532+0000 7f966619d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35976) UID: 0 2023-11-05T08:32:51.639 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:32:51.635+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:32:51.654 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:51.655 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:51.669 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-11-05T08:32:51.740 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-11-05T08:32:51.736+0000 7f966619d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35976) UID: 0 2023-11-05T08:32:51.800 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:51.800 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:51.813 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-11-05T08:32:51.858 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:32:51.855+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:32:51.930 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:51.930 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:51.941 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:32:51.939+0000 7fdc5cd58640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:32:51.945 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-11-05T08:32:52.042 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:32:52.035+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:32:52.087 INFO:teuthology.orchestra.run.smithi002.stderr:set pool 3 pg_num to 26 2023-11-05T08:32:52.095 INFO:tasks.rados.rados.0.smithi090.stdout:86: done (3 left) 2023-11-05T08:32:52.095 INFO:tasks.rados.rados.0.smithi090.stdout:87: done (2 left) 2023-11-05T08:32:52.096 INFO:tasks.rados.rados.0.smithi090.stdout:88: done (1 left) 2023-11-05T08:32:52.096 INFO:tasks.rados.rados.0.smithi090.stdout:90: done (0 left) 2023-11-05T08:32:52.096 INFO:tasks.rados.rados.0.smithi090.stdout:91: read oid 43 snap 1 2023-11-05T08:32:52.096 INFO:tasks.rados.rados.0.smithi090.stdout:91: expect (ObjNum 42 snap 0 seq_num 42) 2023-11-05T08:32:52.096 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 40 current snap is 5 2023-11-05T08:32:52.096 INFO:tasks.rados.rados.0.smithi090.stdout:92: seq_num 64 ranges {1671168=704512,2375680=376832} 2023-11-05T08:32:52.104 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:52.105 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:52.120 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-11-05T08:32:52.134 INFO:tasks.rados.rados.0.smithi090.stdout:92: writing smithi09039252-40 from 1671168 to 2375680 tid 1 2023-11-05T08:32:52.141 INFO:tasks.rados.rados.0.smithi090.stdout:92: writing smithi09039252-40 from 2375680 to 2752512 tid 2 2023-11-05T08:32:52.141 INFO:tasks.rados.rados.0.smithi090.stdout:93: rmattr oid 39 current snap is 5 2023-11-05T08:32:52.141 INFO:tasks.rados.rados.0.smithi090.stdout:93: done (2 left) 2023-11-05T08:32:52.141 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 38 current snap is 5 2023-11-05T08:32:52.141 INFO:tasks.rados.rados.0.smithi090.stdout:94: seq_num 65 ranges {622592=720896,1343488=376832} 2023-11-05T08:32:52.158 INFO:tasks.rados.rados.0.smithi090.stdout:94: writing smithi09039252-38 from 622592 to 1343488 tid 1 2023-11-05T08:32:52.159 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:32:52.159+0000 7fdc5cd58640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:32:52.163 INFO:tasks.rados.rados.0.smithi090.stdout:94: writing smithi09039252-38 from 1343488 to 1720320 tid 2 2023-11-05T08:32:52.163 INFO:tasks.rados.rados.0.smithi090.stdout:95: read oid 37 snap -1 2023-11-05T08:32:52.163 INFO:tasks.rados.rados.0.smithi090.stdout:95: expect (ObjNum 36 snap 0 seq_num 36) 2023-11-05T08:32:52.163 INFO:tasks.rados.rados.0.smithi090.stdout:96: rollback oid 17 current snap is 5 2023-11-05T08:32:52.164 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 17 to 4 2023-11-05T08:32:52.164 INFO:tasks.rados.rados.0.smithi090.stdout:91: expect (ObjNum 42 snap 0 seq_num 42) 2023-11-05T08:32:52.215 INFO:tasks.rados.rados.0.smithi090.stdout:92: finishing write tid 1 to smithi09039252-40 2023-11-05T08:32:52.216 INFO:tasks.rados.rados.0.smithi090.stdout:92: finishing write tid 2 to smithi09039252-40 2023-11-05T08:32:52.216 INFO:tasks.rados.rados.0.smithi090.stdout:92: finishing write tid 3 to smithi09039252-40 2023-11-05T08:32:52.216 INFO:tasks.rados.rados.0.smithi090.stdout:92: finishing write tid 4 to smithi09039252-40 2023-11-05T08:32:52.217 INFO:tasks.rados.rados.0.smithi090.stdout:92: oid 40 updating version 0 to 12 2023-11-05T08:32:52.217 INFO:tasks.rados.rados.0.smithi090.stdout:92: oid 40 version 12 is already newer than 10 2023-11-05T08:32:52.218 INFO:tasks.rados.rados.0.smithi090.stdout:92: oid 40 version 12 is already newer than 11 2023-11-05T08:32:52.218 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 40 v 12 (ObjNum 64 snap 5 seq_num 64) dirty exists 2023-11-05T08:32:52.218 INFO:tasks.rados.rados.0.smithi090.stdout:92: left oid 40 (ObjNum 64 snap 5 seq_num 64) 2023-11-05T08:32:52.218 INFO:tasks.rados.rados.0.smithi090.stdout:94: finishing write tid 1 to smithi09039252-38 2023-11-05T08:32:52.219 INFO:tasks.rados.rados.0.smithi090.stdout:91: done (4 left) 2023-11-05T08:32:52.219 INFO:tasks.rados.rados.0.smithi090.stdout:92: done (3 left) 2023-11-05T08:32:52.219 INFO:tasks.rados.rados.0.smithi090.stdout:97: setattr oid 15 current snap is 5 2023-11-05T08:32:52.219 INFO:tasks.rados.rados.0.smithi090.stdout:96: finishing rollback tid 0 to smithi09039252-17 2023-11-05T08:32:52.219 INFO:tasks.rados.rados.0.smithi090.stdout:94: finishing write tid 2 to smithi09039252-38 2023-11-05T08:32:52.220 INFO:tasks.rados.rados.0.smithi090.stdout:96: finishing rollback tid 1 to smithi09039252-17 2023-11-05T08:32:52.220 INFO:tasks.rados.rados.0.smithi090.stdout:96: finishing rollback tid 2 to smithi09039252-17 2023-11-05T08:32:52.220 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 17 v 12 (ObjNum 16 snap 0 seq_num 16) dirty exists 2023-11-05T08:32:52.220 INFO:tasks.rados.rados.0.smithi090.stdout:94: finishing write tid 3 to smithi09039252-38 2023-11-05T08:32:52.220 INFO:tasks.rados.rados.0.smithi090.stdout:94: finishing write tid 4 to smithi09039252-38 2023-11-05T08:32:52.220 INFO:tasks.rados.rados.0.smithi090.stdout:94: oid 38 updating version 0 to 21 2023-11-05T08:32:52.221 INFO:tasks.rados.rados.0.smithi090.stdout:94: oid 38 version 21 is already newer than 20 2023-11-05T08:32:52.221 INFO:tasks.rados.rados.0.smithi090.stdout:94: oid 38 updating version 21 to 22 2023-11-05T08:32:52.221 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 38 v 22 (ObjNum 65 snap 5 seq_num 65) dirty exists 2023-11-05T08:32:52.221 INFO:tasks.rados.rados.0.smithi090.stdout:94: left oid 38 (ObjNum 65 snap 5 seq_num 65) 2023-11-05T08:32:52.221 INFO:tasks.rados.rados.0.smithi090.stdout:95: expect (ObjNum 36 snap 0 seq_num 36) 2023-11-05T08:32:52.229 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-11-05T08:32:52.243 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:32:52.239+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:32:52.248 INFO:tasks.rados.rados.0.smithi090.stdout:94: done (3 left) 2023-11-05T08:32:52.248 INFO:tasks.rados.rados.0.smithi090.stdout:95: done (2 left) 2023-11-05T08:32:52.248 INFO:tasks.rados.rados.0.smithi090.stdout:96: done (1 left) 2023-11-05T08:32:52.249 INFO:tasks.rados.rados.0.smithi090.stdout:98: snap_create 2023-11-05T08:32:52.255 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 15 v 6 (ObjNum 14 snap 0 seq_num 14) dirty exists 2023-11-05T08:32:52.287 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:52.288 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:52.301 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-11-05T08:32:52.344 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:32:52.339+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:32:52.445 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:32:52.439+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:32:52.462 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:52.462 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:52.476 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-11-05T08:32:52.566 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:32:52.563+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:32:52.645 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:52.645 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:52.646 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:32:52.645+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:32:52.659 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2023-11-05T08:32:52.748 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:32:52.743+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:32:52.770 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-05T08:32:52.824 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:52.824 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:52.838 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2023-11-05T08:32:52.849 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-11-05T08:32:52.844+0000 7f966619d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35976) UID: 0 2023-11-05T08:32:52.914 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-11-05T08:32:52.936 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1: {} 2023-11-05T08:32:52.936 INFO:teuthology.orchestra.run.smithi002.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-11-05T08:32:52.950 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:32:52.944+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:32:52.954 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2: {} 2023-11-05T08:32:52.954 INFO:teuthology.orchestra.run.smithi002.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-11-05T08:32:52.973 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3: {} 2023-11-05T08:32:52.974 INFO:teuthology.orchestra.run.smithi002.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-11-05T08:32:52.990 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4: {} 2023-11-05T08:32:52.991 INFO:teuthology.orchestra.run.smithi002.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-11-05T08:32:53.002 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:53.002 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:53.011 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5: {} 2023-11-05T08:32:53.012 INFO:teuthology.orchestra.run.smithi002.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-11-05T08:32:53.016 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2023-11-05T08:32:53.033 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6: {} 2023-11-05T08:32:53.033 INFO:teuthology.orchestra.run.smithi002.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-11-05T08:32:53.051 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:32:53.051+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:32:53.054 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7: {} 2023-11-05T08:32:53.054 INFO:teuthology.orchestra.run.smithi002.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-11-05T08:32:53.073 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8: {} 2023-11-05T08:32:53.074 INFO:teuthology.orchestra.run.smithi002.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-11-05T08:32:53.096 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9: {} 2023-11-05T08:32:53.096 INFO:teuthology.orchestra.run.smithi002.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-11-05T08:32:53.097 INFO:teuthology.orchestra.run.smithi002.stderr:noscrub is set 2023-11-05T08:32:53.108 INFO:tasks.rados.rados.0.smithi090.stdout:97: done (1 left) 2023-11-05T08:32:53.109 INFO:tasks.rados.rados.0.smithi090.stdout:98: done (0 left) 2023-11-05T08:32:53.109 INFO:tasks.rados.rados.0.smithi090.stdout:99: read oid 11 snap 2 2023-11-05T08:32:53.109 INFO:tasks.rados.rados.0.smithi090.stdout:99: expect (ObjNum 54 snap 0 seq_num 54) 2023-11-05T08:32:53.109 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 8 current snap is 6 2023-11-05T08:32:53.109 INFO:tasks.rados.rados.0.smithi090.stdout:100: seq_num 66 ranges {131072=737280,868352=376832} 2023-11-05T08:32:53.129 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10: {} 2023-11-05T08:32:53.130 INFO:teuthology.orchestra.run.smithi002.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-11-05T08:32:53.132 INFO:tasks.rados.rados.0.smithi090.stdout:100: writing smithi09039252-8 from 131072 to 868352 tid 1 2023-11-05T08:32:53.142 INFO:tasks.rados.rados.0.smithi090.stdout:100: writing smithi09039252-8 from 868352 to 1245184 tid 2 2023-11-05T08:32:53.142 INFO:tasks.rados.rados.0.smithi090.stdout:101: copy_from oid 40 from oid 27 current snap is 6 2023-11-05T08:32:53.142 INFO:tasks.rados.rados.0.smithi090.stdout:102: rmattr oid 31 current snap is 6 2023-11-05T08:32:53.143 INFO:tasks.rados.rados.0.smithi090.stdout:102: done (3 left) 2023-11-05T08:32:53.143 INFO:tasks.rados.rados.0.smithi090.stdout:103: read oid 45 snap -1 2023-11-05T08:32:53.143 INFO:tasks.rados.rados.0.smithi090.stdout:103: expect (ObjNum 44 snap 0 seq_num 44) 2023-11-05T08:32:53.143 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 43 current snap is 6 2023-11-05T08:32:53.144 INFO:tasks.rados.rados.0.smithi090.stdout:104: seq_num 67 ranges {720896=753664,1474560=376832} 2023-11-05T08:32:53.152 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:32:53.147+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:32:53.155 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11: {} 2023-11-05T08:32:53.155 INFO:teuthology.orchestra.run.smithi002.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-11-05T08:32:53.157 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:32:53.157 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 20, 2023-11-05T08:32:53.158 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 600, 2023-11-05T08:32:53.158 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [ 2023-11-05T08:32:53.158 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.158 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.ds3 36/30 ECSubWrite(tid=20, reqid=client.4691.0:233, at_version=36'10, trim_to=0'0, roll_forward_to=34'8))", 2023-11-05T08:32:53.159 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.066903+0000", 2023-11-05T08:32:53.159 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 4.089966617, 2023-11-05T08:32:53.159 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.004733414, 2023-11-05T08:32:53.159 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:32:53.159 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:32:53.160 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:32:53.160 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.160 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:32:53.160 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.066903+0000", 2023-11-05T08:32:53.160 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:53.161 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.161 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.161 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:32:53.161 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.066903+0000", 2023-11-05T08:32:53.161 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:53.161 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.162 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.162 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:32:53.162 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.066903+0000", 2023-11-05T08:32:53.162 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.3999999999999997e-07 2023-11-05T08:32:53.162 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.163 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.163 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:32:53.163 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.066944+0000", 2023-11-05T08:32:53.163 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.0657000000000001e-05 2023-11-05T08:32:53.163 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.164 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.164 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:32:53.164 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.066968+0000", 2023-11-05T08:32:53.164 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3981999999999998e-05 2023-11-05T08:32:53.164 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.165 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.165 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:32:53.165 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.066985+0000", 2023-11-05T08:32:53.165 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6915999999999999e-05 2023-11-05T08:32:53.165 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.166 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.166 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:32:53.166 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.067184+0000", 2023-11-05T08:32:53.166 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000199507 2023-11-05T08:32:53.166 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.166 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.167 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:32:53.167 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.067213+0000", 2023-11-05T08:32:53.167 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8719000000000001e-05 2023-11-05T08:32:53.167 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.167 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.168 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:32:53.168 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.071537+0000", 2023-11-05T08:32:53.168 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0043238959999999998 2023-11-05T08:32:53.168 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.168 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.169 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:32:53.169 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.071636+0000", 2023-11-05T08:32:53.169 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.9396999999999997e-05 2023-11-05T08:32:53.169 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:53.169 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:53.169 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:53.170 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.170 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.170 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.ds3 36/30 ECSubWrite(tid=21, reqid=client.4691.0:234, at_version=36'11, trim_to=0'0, roll_forward_to=34'8))", 2023-11-05T08:32:53.170 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.069983+0000", 2023-11-05T08:32:53.170 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 4.0868861240000003, 2023-11-05T08:32:53.170 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0031732639999999999, 2023-11-05T08:32:53.171 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:32:53.171 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:32:53.171 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:32:53.171 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.171 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:32:53.172 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.069983+0000", 2023-11-05T08:32:53.172 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:53.172 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.172 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.172 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:32:53.172 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.069983+0000", 2023-11-05T08:32:53.173 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:53.173 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.173 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.173 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:32:53.173 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.069985+0000", 2023-11-05T08:32:53.173 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.95e-06 2023-11-05T08:32:53.174 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.174 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.174 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:32:53.174 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.070037+0000", 2023-11-05T08:32:53.174 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.1683e-05 2023-11-05T08:32:53.175 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.175 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.175 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:32:53.175 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.070070+0000", 2023-11-05T08:32:53.175 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.3244000000000002e-05 2023-11-05T08:32:53.175 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.177 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.177 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:32:53.177 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.070095+0000", 2023-11-05T08:32:53.178 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5096000000000002e-05 2023-11-05T08:32:53.178 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.178 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.178 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:32:53.178 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.070275+0000", 2023-11-05T08:32:53.179 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00018010500000000001 2023-11-05T08:32:53.179 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.179 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.180 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:32:53.180 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.070299+0000", 2023-11-05T08:32:53.180 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4051999999999999e-05 2023-11-05T08:32:53.180 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.180 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.181 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:32:53.181 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.073065+0000", 2023-11-05T08:32:53.181 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0027658930000000002 2023-11-05T08:32:53.182 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.182 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.182 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:32:53.183 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.073157+0000", 2023-11-05T08:32:53.183 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.1241000000000003e-05 2023-11-05T08:32:53.183 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:53.184 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:53.184 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:53.184 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.185 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.185 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.ds3 36/30 ECSubWrite(tid=22, reqid=client.4691.0:235, at_version=36'12, trim_to=0'0, roll_forward_to=34'8))", 2023-11-05T08:32:53.185 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.077098+0000", 2023-11-05T08:32:53.185 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 4.0797717259999997, 2023-11-05T08:32:53.186 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.002195247, 2023-11-05T08:32:53.186 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:32:53.186 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:32:53.187 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:32:53.187 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.187 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:32:53.187 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.077098+0000", 2023-11-05T08:32:53.187 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:53.188 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.188 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.188 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:32:53.188 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.077098+0000", 2023-11-05T08:32:53.188 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:53.189 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.189 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.189 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:32:53.189 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.077100+0000", 2023-11-05T08:32:53.189 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.829e-06 2023-11-05T08:32:53.189 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.190 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.190 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:32:53.190 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.077146+0000", 2023-11-05T08:32:53.190 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.6044000000000002e-05 2023-11-05T08:32:53.191 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.191 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.191 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:32:53.191 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.077193+0000", 2023-11-05T08:32:53.191 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.7170999999999999e-05 2023-11-05T08:32:53.192 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.192 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.192 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:32:53.192 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.077219+0000", 2023-11-05T08:32:53.192 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6327999999999999e-05 2023-11-05T08:32:53.192 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.193 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.193 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:32:53.193 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.077446+0000", 2023-11-05T08:32:53.193 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00022655200000000001 2023-11-05T08:32:53.193 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.194 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.194 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:32:53.194 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.077471+0000", 2023-11-05T08:32:53.194 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5483e-05 2023-11-05T08:32:53.194 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.195 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.195 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:32:53.195 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079200+0000", 2023-11-05T08:32:53.195 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0017290960000000001 2023-11-05T08:32:53.195 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.196 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.196 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:32:53.196 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079293+0000", 2023-11-05T08:32:53.196 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.2744000000000005e-05 2023-11-05T08:32:53.196 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:53.196 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:53.197 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:53.197 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.197 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.197 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.ds3 36/30 ECSubWrite(tid=23, reqid=osd.7.0:23, at_version=36'14, trim_to=0'0, roll_forward_to=34'8))", 2023-11-05T08:32:53.197 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.077282+0000", 2023-11-05T08:32:53.198 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 4.0795878449999998, 2023-11-05T08:32:53.198 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.005326842, 2023-11-05T08:32:53.198 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:32:53.198 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:32:53.198 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:32:53.198 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.199 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:32:53.199 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.077282+0000", 2023-11-05T08:32:53.199 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:53.199 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.199 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.199 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:32:53.200 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.077282+0000", 2023-11-05T08:32:53.200 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:53.200 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.200 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.200 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:32:53.200 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.077283+0000", 2023-11-05T08:32:53.201 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.1699999999999997e-07 2023-11-05T08:32:53.201 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.201 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.201 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:32:53.201 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.077374+0000", 2023-11-05T08:32:53.202 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.1181000000000007e-05 2023-11-05T08:32:53.202 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.202 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.202 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:32:53.202 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.077399+0000", 2023-11-05T08:32:53.202 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5452e-05 2023-11-05T08:32:53.203 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.203 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.203 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:32:53.203 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.077430+0000", 2023-11-05T08:32:53.203 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0768000000000001e-05 2023-11-05T08:32:53.204 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.204 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.204 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:32:53.204 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.077988+0000", 2023-11-05T08:32:53.204 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00055794899999999997 2023-11-05T08:32:53.204 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.205 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.205 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:32:53.205 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078010+0000", 2023-11-05T08:32:53.205 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2325000000000001e-05 2023-11-05T08:32:53.205 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.206 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.206 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:32:53.206 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.082479+0000", 2023-11-05T08:32:53.206 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0044689609999999996 2023-11-05T08:32:53.206 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.207 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.207 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:32:53.207 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.082608+0000", 2023-11-05T08:32:53.207 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00012928899999999999 2023-11-05T08:32:53.208 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:53.208 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:53.208 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:53.208 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.209 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.209 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.ds3 36/30 ECSubWrite(tid=24, reqid=osd.7.0:23, at_version=0'0, trim_to=0'0, roll_forward_to=36'14))", 2023-11-05T08:32:53.209 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.089743+0000", 2023-11-05T08:32:53.209 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 4.0671260010000001, 2023-11-05T08:32:53.209 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0062502979999999996, 2023-11-05T08:32:53.209 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:32:53.210 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:32:53.210 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:32:53.210 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.210 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:32:53.210 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089743+0000", 2023-11-05T08:32:53.211 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:53.211 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.211 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.211 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:32:53.211 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089743+0000", 2023-11-05T08:32:53.211 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:53.212 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.212 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.212 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:32:53.212 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089745+0000", 2023-11-05T08:32:53.212 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.034e-06 2023-11-05T08:32:53.213 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.213 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.213 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:32:53.213 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089768+0000", 2023-11-05T08:32:53.213 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3195000000000001e-05 2023-11-05T08:32:53.214 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.214 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.214 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:32:53.214 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089791+0000", 2023-11-05T08:32:53.214 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2804999999999999e-05 2023-11-05T08:32:53.215 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.215 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.215 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:32:53.215 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089843+0000", 2023-11-05T08:32:53.215 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.27e-05 2023-11-05T08:32:53.215 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.216 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.216 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:32:53.216 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.090167+0000", 2023-11-05T08:32:53.216 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00032410700000000002 2023-11-05T08:32:53.216 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.217 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.217 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:32:53.217 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.090192+0000", 2023-11-05T08:32:53.217 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4819000000000001e-05 2023-11-05T08:32:53.218 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.218 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.218 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:32:53.218 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.095900+0000", 2023-11-05T08:32:53.218 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0057078540000000001 2023-11-05T08:32:53.219 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.219 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.219 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:32:53.219 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.095994+0000", 2023-11-05T08:32:53.219 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.3783999999999995e-05 2023-11-05T08:32:53.220 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:53.220 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:53.220 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:53.220 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.220 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.221 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.fs5 36/30 ECSubWrite(tid=30, reqid=osd.11.0:30, at_version=36'17, trim_to=0'0, roll_forward_to=35'15))", 2023-11-05T08:32:53.221 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.091636+0000", 2023-11-05T08:32:53.221 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 4.0652333570000003, 2023-11-05T08:32:53.222 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0043942149999999999, 2023-11-05T08:32:53.222 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:32:53.222 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:32:53.222 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:32:53.222 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.223 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:32:53.223 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.091636+0000", 2023-11-05T08:32:53.223 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:53.224 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.224 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.225 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:32:53.225 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.091636+0000", 2023-11-05T08:32:53.225 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:53.226 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.226 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.226 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:32:53.227 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.091637+0000", 2023-11-05T08:32:53.227 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.37e-07 2023-11-05T08:32:53.227 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.227 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.228 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:32:53.228 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.091690+0000", 2023-11-05T08:32:53.228 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.3643000000000001e-05 2023-11-05T08:32:53.229 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.229 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.229 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:32:53.229 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.091713+0000", 2023-11-05T08:32:53.229 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2901999999999999e-05 2023-11-05T08:32:53.230 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.230 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.230 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:32:53.230 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.091734+0000", 2023-11-05T08:32:53.230 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0639e-05 2023-11-05T08:32:53.231 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.231 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.231 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:32:53.231 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.091916+0000", 2023-11-05T08:32:53.231 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00018221200000000001 2023-11-05T08:32:53.231 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.232 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.232 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:32:53.232 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.091940+0000", 2023-11-05T08:32:53.232 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4397999999999999e-05 2023-11-05T08:32:53.232 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.232 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.233 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:32:53.233 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.095937+0000", 2023-11-05T08:32:53.233 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0039963000000000004 2023-11-05T08:32:53.233 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.233 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.234 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:32:53.234 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.096030+0000", 2023-11-05T08:32:53.234 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.3584000000000004e-05 2023-11-05T08:32:53.234 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:53.234 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:53.234 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:53.235 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.235 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.235 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.fs5 36/30 ECSubWrite(tid=31, reqid=osd.11.0:30, at_version=0'0, trim_to=0'0, roll_forward_to=36'17))", 2023-11-05T08:32:53.235 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.097957+0000", 2023-11-05T08:32:53.235 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 4.0589128739999998, 2023-11-05T08:32:53.235 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001748028, 2023-11-05T08:32:53.236 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:32:53.236 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:32:53.236 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:32:53.236 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.236 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:32:53.237 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.097957+0000", 2023-11-05T08:32:53.237 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:53.237 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.237 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.237 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:32:53.238 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.097957+0000", 2023-11-05T08:32:53.238 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:53.238 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.238 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.238 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:32:53.239 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.097963+0000", 2023-11-05T08:32:53.239 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.083e-06 2023-11-05T08:32:53.239 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.239 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.239 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:32:53.240 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.097995+0000", 2023-11-05T08:32:53.240 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2784000000000003e-05 2023-11-05T08:32:53.240 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.241 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.241 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:32:53.241 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.098022+0000", 2023-11-05T08:32:53.241 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6152e-05 2023-11-05T08:32:53.242 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.242 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.242 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:32:53.242 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.098044+0000", 2023-11-05T08:32:53.243 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2291999999999999e-05 2023-11-05T08:32:53.243 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.243 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.244 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:32:53.244 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.098197+0000", 2023-11-05T08:32:53.245 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00015278700000000001 2023-11-05T08:32:53.245 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.245 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.245 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:32:53.246 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.098224+0000", 2023-11-05T08:32:53.246 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6999e-05 2023-11-05T08:32:53.246 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.246 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.246 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:32:53.247 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.099623+0000", 2023-11-05T08:32:53.247 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0013996359999999999 2023-11-05T08:32:53.247 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.247 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.247 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:32:53.248 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.099705+0000", 2023-11-05T08:32:53.248 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.1295e-05 2023-11-05T08:32:53.248 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:53.249 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:53.249 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:53.249 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:53.249 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:53.249 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:32:53.250 INFO:teuthology.orchestra.run.smithi002.stdout:osd.12: {} 2023-11-05T08:32:53.251 INFO:teuthology.orchestra.run.smithi002.stderr:osd.12: osd_enable_op_tracker = 'true' 2023-11-05T08:32:53.252 INFO:teuthology.orchestra.run.smithi002.stdout:osd.13: {} 2023-11-05T08:32:53.252 INFO:teuthology.orchestra.run.smithi002.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-11-05T08:32:53.252 INFO:tasks.rados.rados.0.smithi090.stdout:104: writing smithi09039252-43 from 720896 to 1474560 tid 1 2023-11-05T08:32:53.253 INFO:tasks.rados.rados.0.smithi090.stdout:104: writing smithi09039252-43 from 1474560 to 1851392 tid 2 2023-11-05T08:32:53.253 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 16 current snap is 6 2023-11-05T08:32:53.253 INFO:tasks.rados.rados.0.smithi090.stdout:105: seq_num 68 ranges {262144=770048,1032192=376832} 2023-11-05T08:32:53.253 INFO:tasks.rados.rados.0.smithi090.stdout:105: writing smithi09039252-16 from 262144 to 1032192 tid 1 2023-11-05T08:32:53.253 INFO:tasks.rados.rados.0.smithi090.stdout:105: writing smithi09039252-16 from 1032192 to 1409024 tid 2 2023-11-05T08:32:53.254 INFO:tasks.rados.rados.0.smithi090.stdout:106: snap_remove snap 1 2023-11-05T08:32:53.254 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2023-11-05T08:32:53.255 INFO:teuthology.orchestra.run.smithi002.stdout:osd.14: {} 2023-11-05T08:32:53.255 INFO:teuthology.orchestra.run.smithi002.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-11-05T08:32:53.255 INFO:teuthology.orchestra.run.smithi002.stdout:osd.15: {} 2023-11-05T08:32:53.256 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-05T08:32:53.257 INFO:teuthology.orchestra.run.smithi002.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-11-05T08:32:53.257 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-05T08:32:53.295 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:32:53.291+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:32:53.357 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:32:53.353+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:32:53.421 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:32:53.421 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2023-11-05T08:32:53.421 INFO:teuthology.orchestra.run.smithi069.stdout: "num_ops": 0 2023-11-05T08:32:53.421 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:32:53.435 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2023-11-05T08:32:53.459 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:32:53.455+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:32:53.552 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:32:53.552 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2023-11-05T08:32:53.553 INFO:teuthology.orchestra.run.smithi069.stdout: "complaint_time": 30, 2023-11-05T08:32:53.553 INFO:teuthology.orchestra.run.smithi069.stdout: "num_blocked_ops": 0 2023-11-05T08:32:53.553 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:32:53.559 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:32:53.557+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:32:53.566 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2023-11-05T08:32:53.660 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:32:53.656+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:32:53.685 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:32:53.685 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 20, 2023-11-05T08:32:53.685 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 600, 2023-11-05T08:32:53.686 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [ 2023-11-05T08:32:53.686 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.686 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.ds2 36/30 ECSubWrite(tid=20, reqid=client.4691.0:233, at_version=36'10, trim_to=0'0, roll_forward_to=34'8))", 2023-11-05T08:32:53.686 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.066883+0000", 2023-11-05T08:32:53.687 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 4.6177443179999997, 2023-11-05T08:32:53.687 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0054212649999999998, 2023-11-05T08:32:53.687 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:32:53.687 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:32:53.687 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:32:53.688 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.688 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:32:53.688 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.066883+0000", 2023-11-05T08:32:53.688 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:53.688 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.689 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.689 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:32:53.689 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.066883+0000", 2023-11-05T08:32:53.689 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:53.689 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.690 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.690 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:32:53.690 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.066884+0000", 2023-11-05T08:32:53.690 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.6300000000000005e-07 2023-11-05T08:32:53.690 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.691 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.691 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:32:53.691 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.066944+0000", 2023-11-05T08:32:53.691 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.9570000000000001e-05 2023-11-05T08:32:53.691 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.692 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.692 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:32:53.692 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.066968+0000", 2023-11-05T08:32:53.692 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4885e-05 2023-11-05T08:32:53.692 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.692 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.693 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:32:53.693 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.066990+0000", 2023-11-05T08:32:53.693 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1729000000000002e-05 2023-11-05T08:32:53.693 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.693 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.694 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:32:53.694 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.067176+0000", 2023-11-05T08:32:53.694 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00018593799999999999 2023-11-05T08:32:53.694 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.694 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.695 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:32:53.695 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.067207+0000", 2023-11-05T08:32:53.695 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.1325000000000002e-05 2023-11-05T08:32:53.695 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.695 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.695 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:32:53.696 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.072220+0000", 2023-11-05T08:32:53.696 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0050125600000000001 2023-11-05T08:32:53.696 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.696 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.696 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:32:53.696 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.072305+0000", 2023-11-05T08:32:53.697 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.4695000000000001e-05 2023-11-05T08:32:53.697 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:53.697 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:53.697 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:53.698 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.698 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.698 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.ds2 36/30 ECSubWrite(tid=21, reqid=client.4691.0:234, at_version=36'11, trim_to=0'0, roll_forward_to=34'8))", 2023-11-05T08:32:53.698 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.070068+0000", 2023-11-05T08:32:53.698 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 4.614559442, 2023-11-05T08:32:53.699 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0044083250000000003, 2023-11-05T08:32:53.699 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:32:53.699 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:32:53.699 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:32:53.699 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.700 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:32:53.700 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.070068+0000", 2023-11-05T08:32:53.700 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:53.700 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.700 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.700 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:32:53.701 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.070068+0000", 2023-11-05T08:32:53.701 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:53.701 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.701 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.701 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:32:53.701 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.070070+0000", 2023-11-05T08:32:53.702 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8300000000000001e-06 2023-11-05T08:32:53.702 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.702 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.702 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:32:53.702 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.070119+0000", 2023-11-05T08:32:53.703 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.8730000000000003e-05 2023-11-05T08:32:53.703 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.703 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.703 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:32:53.703 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.070155+0000", 2023-11-05T08:32:53.704 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.6575000000000001e-05 2023-11-05T08:32:53.704 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.704 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.704 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:32:53.704 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.070179+0000", 2023-11-05T08:32:53.704 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3218999999999999e-05 2023-11-05T08:32:53.705 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.705 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.705 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:32:53.705 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.070335+0000", 2023-11-05T08:32:53.705 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000156587 2023-11-05T08:32:53.706 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.706 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.706 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:32:53.706 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.070360+0000", 2023-11-05T08:32:53.707 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4711000000000001e-05 2023-11-05T08:32:53.707 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.707 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.707 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:32:53.707 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.074384+0000", 2023-11-05T08:32:53.708 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0040235690000000003 2023-11-05T08:32:53.708 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.708 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.708 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:32:53.708 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.074477+0000", 2023-11-05T08:32:53.709 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.3103999999999995e-05 2023-11-05T08:32:53.709 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:53.709 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:53.709 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:53.709 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.709 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.710 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.ds2 36/30 ECSubWrite(tid=22, reqid=client.4691.0:235, at_version=36'12, trim_to=0'0, roll_forward_to=34'8))", 2023-11-05T08:32:53.710 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.078278+0000", 2023-11-05T08:32:53.710 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 4.6063500709999996, 2023-11-05T08:32:53.710 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0043972519999999999, 2023-11-05T08:32:53.710 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:32:53.711 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:32:53.711 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:32:53.711 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.711 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:32:53.711 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078278+0000", 2023-11-05T08:32:53.712 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:53.712 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.712 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.712 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:32:53.713 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078278+0000", 2023-11-05T08:32:53.713 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:53.713 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.714 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.714 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:32:53.714 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078280+0000", 2023-11-05T08:32:53.715 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.933e-06 2023-11-05T08:32:53.715 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.715 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.716 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:32:53.716 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078314+0000", 2023-11-05T08:32:53.716 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.4734000000000003e-05 2023-11-05T08:32:53.717 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.717 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.717 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:32:53.718 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078344+0000", 2023-11-05T08:32:53.718 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.9302000000000002e-05 2023-11-05T08:32:53.718 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.719 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.719 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:32:53.719 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078366+0000", 2023-11-05T08:32:53.719 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2751e-05 2023-11-05T08:32:53.720 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.720 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.720 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:32:53.720 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078520+0000", 2023-11-05T08:32:53.721 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00015408500000000001 2023-11-05T08:32:53.721 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.721 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.721 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:32:53.721 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078543+0000", 2023-11-05T08:32:53.721 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2430999999999999e-05 2023-11-05T08:32:53.722 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.722 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.722 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:32:53.722 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.082583+0000", 2023-11-05T08:32:53.722 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0040398079999999998 2023-11-05T08:32:53.723 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.723 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.723 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:32:53.723 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.082675+0000", 2023-11-05T08:32:53.723 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.2207999999999995e-05 2023-11-05T08:32:53.723 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:53.724 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:53.724 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:53.724 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.724 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.724 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.ds2 36/30 ECSubWrite(tid=23, reqid=osd.7.0:23, at_version=36'14, trim_to=0'0, roll_forward_to=34'8))", 2023-11-05T08:32:53.724 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.078426+0000", 2023-11-05T08:32:53.725 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 4.6062017470000001, 2023-11-05T08:32:53.725 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0071789940000000002, 2023-11-05T08:32:53.725 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:32:53.725 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:32:53.725 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:32:53.725 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.726 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:32:53.726 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078426+0000", 2023-11-05T08:32:53.726 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:53.726 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.726 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.727 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:32:53.727 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078426+0000", 2023-11-05T08:32:53.727 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:53.727 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.727 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.727 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:32:53.728 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078427+0000", 2023-11-05T08:32:53.728 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.093e-06 2023-11-05T08:32:53.728 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.728 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.728 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:32:53.728 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078484+0000", 2023-11-05T08:32:53.729 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.7240000000000001e-05 2023-11-05T08:32:53.729 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.729 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.729 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:32:53.729 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078508+0000", 2023-11-05T08:32:53.730 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4032e-05 2023-11-05T08:32:53.730 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.730 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.730 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:32:53.730 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078527+0000", 2023-11-05T08:32:53.731 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8754999999999999e-05 2023-11-05T08:32:53.731 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.731 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.731 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:32:53.731 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079037+0000", 2023-11-05T08:32:53.732 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000509722 2023-11-05T08:32:53.732 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.732 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.732 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:32:53.732 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079059+0000", 2023-11-05T08:32:53.732 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2467999999999999e-05 2023-11-05T08:32:53.733 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.733 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.733 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:32:53.733 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.085512+0000", 2023-11-05T08:32:53.733 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0064529059999999996 2023-11-05T08:32:53.733 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.734 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.734 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:32:53.734 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.085605+0000", 2023-11-05T08:32:53.734 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.2777999999999998e-05 2023-11-05T08:32:53.734 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:53.734 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:53.735 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:53.735 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.735 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.735 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.4s5 36/30 ECSubWrite(tid=10, reqid=osd.8.0:10, at_version=36'8, trim_to=0'0, roll_forward_to=34'6))", 2023-11-05T08:32:53.735 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.078716+0000", 2023-11-05T08:32:53.735 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 4.6059119620000004, 2023-11-05T08:32:53.736 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.006931782, 2023-11-05T08:32:53.736 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:32:53.736 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:32:53.736 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:32:53.736 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.737 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:32:53.737 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078716+0000", 2023-11-05T08:32:53.737 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:53.737 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.737 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.737 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:32:53.738 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078716+0000", 2023-11-05T08:32:53.738 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:53.738 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.738 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.738 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:32:53.739 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078716+0000", 2023-11-05T08:32:53.739 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.0399999999999995e-07 2023-11-05T08:32:53.739 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.739 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.739 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:32:53.740 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078773+0000", 2023-11-05T08:32:53.740 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.6149000000000001e-05 2023-11-05T08:32:53.740 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.740 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.740 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:32:53.740 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078804+0000", 2023-11-05T08:32:53.741 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.1677999999999997e-05 2023-11-05T08:32:53.741 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.741 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.741 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:32:53.741 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078827+0000", 2023-11-05T08:32:53.742 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2469000000000001e-05 2023-11-05T08:32:53.742 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.742 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.743 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:32:53.743 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079007+0000", 2023-11-05T08:32:53.743 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000180242 2023-11-05T08:32:53.743 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.744 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.744 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:32:53.744 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079032+0000", 2023-11-05T08:32:53.744 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4915000000000001e-05 2023-11-05T08:32:53.744 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.745 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.745 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:32:53.745 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.085566+0000", 2023-11-05T08:32:53.746 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0065336140000000001 2023-11-05T08:32:53.746 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.746 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.747 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:32:53.747 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.085648+0000", 2023-11-05T08:32:53.747 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.2010999999999998e-05 2023-11-05T08:32:53.748 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:53.748 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:53.748 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:53.748 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.749 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.749 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.8s4 36/30 ECSubWrite(tid=23, reqid=osd.5.0:23, at_version=36'15, trim_to=0'0, roll_forward_to=34'13))", 2023-11-05T08:32:53.749 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.079985+0000", 2023-11-05T08:32:53.750 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 4.6046429470000003, 2023-11-05T08:32:53.750 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.007626023, 2023-11-05T08:32:53.750 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:32:53.750 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:32:53.750 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:32:53.751 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.751 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:32:53.751 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079985+0000", 2023-11-05T08:32:53.751 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:53.751 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.752 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.752 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:32:53.752 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079985+0000", 2023-11-05T08:32:53.752 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:53.752 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.752 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.753 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:32:53.753 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079985+0000", 2023-11-05T08:32:53.753 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.6300000000000005e-07 2023-11-05T08:32:53.753 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.753 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.754 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:32:53.754 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.080040+0000", 2023-11-05T08:32:53.754 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.4796999999999997e-05 2023-11-05T08:32:53.754 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.754 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.755 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:32:53.755 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.080073+0000", 2023-11-05T08:32:53.755 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.3034999999999998e-05 2023-11-05T08:32:53.756 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.756 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.756 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:32:53.756 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.080095+0000", 2023-11-05T08:32:53.756 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1841999999999999e-05 2023-11-05T08:32:53.756 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.757 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.757 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:32:53.757 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.080255+0000", 2023-11-05T08:32:53.757 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00016012700000000001 2023-11-05T08:32:53.757 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.758 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.758 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:32:53.758 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.080280+0000", 2023-11-05T08:32:53.759 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4953999999999999e-05 2023-11-05T08:32:53.759 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.759 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.759 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:32:53.759 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.087517+0000", 2023-11-05T08:32:53.759 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0072370799999999999 2023-11-05T08:32:53.760 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.760 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.760 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:32:53.760 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.087611+0000", 2023-11-05T08:32:53.761 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.3524999999999996e-05 2023-11-05T08:32:53.761 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:53.761 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:53.762 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:53.762 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.763 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.763 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.4s5 36/30 ECSubWrite(tid=11, reqid=osd.8.0:10, at_version=0'0, trim_to=0'0, roll_forward_to=36'8))", 2023-11-05T08:32:53.763 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.086922+0000", 2023-11-05T08:32:53.764 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 4.597706069, 2023-11-05T08:32:53.764 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0022353429999999999, 2023-11-05T08:32:53.764 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:32:53.764 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:32:53.764 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:32:53.764 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.765 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:32:53.765 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.086922+0000", 2023-11-05T08:32:53.765 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:53.766 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.766 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.766 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:32:53.766 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.086922+0000", 2023-11-05T08:32:53.767 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:53.767 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.767 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.767 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:32:53.767 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.086924+0000", 2023-11-05T08:32:53.767 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.874e-06 2023-11-05T08:32:53.768 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.768 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.768 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:32:53.768 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.086953+0000", 2023-11-05T08:32:53.768 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.9068e-05 2023-11-05T08:32:53.769 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.769 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.769 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:32:53.769 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.086986+0000", 2023-11-05T08:32:53.769 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.3864e-05 2023-11-05T08:32:53.769 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.770 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.770 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:32:53.770 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.087009+0000", 2023-11-05T08:32:53.770 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2506000000000001e-05 2023-11-05T08:32:53.770 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.771 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.771 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:32:53.771 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.087168+0000", 2023-11-05T08:32:53.771 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00015917700000000001 2023-11-05T08:32:53.771 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.771 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.772 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:32:53.772 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.087198+0000", 2023-11-05T08:32:53.772 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.9674000000000001e-05 2023-11-05T08:32:53.772 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.772 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.773 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:32:53.773 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089023+0000", 2023-11-05T08:32:53.773 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0018252819999999999 2023-11-05T08:32:53.773 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.773 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.773 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:32:53.774 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089157+0000", 2023-11-05T08:32:53.774 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000133898 2023-11-05T08:32:53.774 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:53.774 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:53.774 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:53.775 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.775 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.775 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.ds2 36/30 ECSubWrite(tid=24, reqid=osd.7.0:23, at_version=0'0, trim_to=0'0, roll_forward_to=36'14))", 2023-11-05T08:32:53.775 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.089710+0000", 2023-11-05T08:32:53.775 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 4.5949176170000001, 2023-11-05T08:32:53.775 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0067357069999999996, 2023-11-05T08:32:53.776 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:32:53.776 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:32:53.776 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:32:53.776 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.776 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:32:53.777 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089710+0000", 2023-11-05T08:32:53.777 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:53.777 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.777 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.777 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:32:53.778 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089710+0000", 2023-11-05T08:32:53.778 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:53.778 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.778 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.778 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:32:53.779 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089711+0000", 2023-11-05T08:32:53.779 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1429999999999999e-06 2023-11-05T08:32:53.779 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.779 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.779 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:32:53.779 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089733+0000", 2023-11-05T08:32:53.780 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1559000000000001e-05 2023-11-05T08:32:53.780 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.780 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.780 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:32:53.780 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089757+0000", 2023-11-05T08:32:53.781 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4515000000000002e-05 2023-11-05T08:32:53.781 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.781 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.781 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:32:53.781 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089780+0000", 2023-11-05T08:32:53.781 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2789000000000002e-05 2023-11-05T08:32:53.782 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.782 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.782 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:32:53.782 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.090049+0000", 2023-11-05T08:32:53.782 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000269363 2023-11-05T08:32:53.782 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.783 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.783 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:32:53.783 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.090074+0000", 2023-11-05T08:32:53.783 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4071999999999999e-05 2023-11-05T08:32:53.783 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.783 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.784 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:32:53.784 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.096355+0000", 2023-11-05T08:32:53.784 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0062814489999999997 2023-11-05T08:32:53.784 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.784 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.784 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:32:53.785 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.096446+0000", 2023-11-05T08:32:53.785 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.0816999999999995e-05 2023-11-05T08:32:53.785 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:53.785 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:53.785 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:53.786 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.786 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.786 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.8s4 36/30 ECSubWrite(tid=24, reqid=osd.5.0:23, at_version=0'0, trim_to=0'0, roll_forward_to=36'15))", 2023-11-05T08:32:53.786 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.089889+0000", 2023-11-05T08:32:53.786 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 4.594739208, 2023-11-05T08:32:53.786 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0065355869999999998, 2023-11-05T08:32:53.787 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:32:53.787 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:32:53.787 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:32:53.787 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.787 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:32:53.788 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089889+0000", 2023-11-05T08:32:53.788 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:53.788 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.788 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.788 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:32:53.788 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089889+0000", 2023-11-05T08:32:53.789 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:53.789 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.789 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.789 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:32:53.789 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089890+0000", 2023-11-05T08:32:53.789 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1599999999999999e-06 2023-11-05T08:32:53.790 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.790 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.790 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:32:53.790 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089908+0000", 2023-11-05T08:32:53.790 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8448999999999999e-05 2023-11-05T08:32:53.791 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.791 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.791 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:32:53.791 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089929+0000", 2023-11-05T08:32:53.791 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0534999999999999e-05 2023-11-05T08:32:53.792 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.792 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.792 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:32:53.792 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089943+0000", 2023-11-05T08:32:53.792 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4469e-05 2023-11-05T08:32:53.793 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.793 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.793 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:32:53.793 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.090098+0000", 2023-11-05T08:32:53.793 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00015513699999999999 2023-11-05T08:32:53.794 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.794 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.794 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:32:53.795 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.090114+0000", 2023-11-05T08:32:53.795 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5962999999999999e-05 2023-11-05T08:32:53.795 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.796 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.796 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:32:53.796 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.096334+0000", 2023-11-05T08:32:53.797 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0062193320000000002 2023-11-05T08:32:53.797 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.797 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.798 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:32:53.798 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.096424+0000", 2023-11-05T08:32:53.798 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.0542000000000001e-05 2023-11-05T08:32:53.798 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:53.799 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:53.799 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:53.799 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.800 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.800 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.fs2 36/30 ECSubWrite(tid=30, reqid=osd.11.0:30, at_version=36'17, trim_to=0'0, roll_forward_to=35'15))", 2023-11-05T08:32:53.800 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.092682+0000", 2023-11-05T08:32:53.800 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 4.5919455940000002, 2023-11-05T08:32:53.800 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0038556520000000002, 2023-11-05T08:32:53.801 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:32:53.801 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:32:53.801 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:32:53.801 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.801 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:32:53.802 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.092682+0000", 2023-11-05T08:32:53.802 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:53.802 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.802 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.802 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:32:53.803 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.092682+0000", 2023-11-05T08:32:53.803 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:53.803 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.803 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.803 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:32:53.803 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.092683+0000", 2023-11-05T08:32:53.804 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.6000000000000003e-07 2023-11-05T08:32:53.804 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.804 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.804 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:32:53.804 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.092739+0000", 2023-11-05T08:32:53.805 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.6020000000000002e-05 2023-11-05T08:32:53.805 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.805 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.805 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:32:53.805 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.092771+0000", 2023-11-05T08:32:53.806 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2234999999999999e-05 2023-11-05T08:32:53.806 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.806 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.806 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:32:53.806 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.092793+0000", 2023-11-05T08:32:53.807 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1922000000000001e-05 2023-11-05T08:32:53.807 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.807 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.807 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:32:53.807 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.093090+0000", 2023-11-05T08:32:53.808 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00029674500000000001 2023-11-05T08:32:53.808 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.808 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.808 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:32:53.808 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.093114+0000", 2023-11-05T08:32:53.808 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4158000000000001e-05 2023-11-05T08:32:53.809 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.809 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.809 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:32:53.809 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.096444+0000", 2023-11-05T08:32:53.809 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0033304250000000001 2023-11-05T08:32:53.810 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.810 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.810 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:32:53.810 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.096538+0000", 2023-11-05T08:32:53.810 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.3387000000000006e-05 2023-11-05T08:32:53.810 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:53.811 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:53.811 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:53.811 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.811 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.811 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.fs2 36/30 ECSubWrite(tid=31, reqid=osd.11.0:30, at_version=0'0, trim_to=0'0, roll_forward_to=36'17))", 2023-11-05T08:32:53.812 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.098040+0000", 2023-11-05T08:32:53.812 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 4.5865873150000001, 2023-11-05T08:32:53.812 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0018674519999999999, 2023-11-05T08:32:53.812 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:32:53.812 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:32:53.812 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:32:53.813 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.813 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:32:53.813 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.098040+0000", 2023-11-05T08:32:53.813 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:53.813 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.814 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.814 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:32:53.814 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.098040+0000", 2023-11-05T08:32:53.814 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:32:53.814 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.815 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.815 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:32:53.815 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.098042+0000", 2023-11-05T08:32:53.815 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5200000000000001e-06 2023-11-05T08:32:53.815 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.816 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.816 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:32:53.816 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.098064+0000", 2023-11-05T08:32:53.816 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2478999999999999e-05 2023-11-05T08:32:53.816 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.816 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.817 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:32:53.817 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.098088+0000", 2023-11-05T08:32:53.817 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4062000000000001e-05 2023-11-05T08:32:53.817 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.817 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.818 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:32:53.818 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.098110+0000", 2023-11-05T08:32:53.818 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1818e-05 2023-11-05T08:32:53.818 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.818 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.819 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:32:53.819 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.098250+0000", 2023-11-05T08:32:53.819 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00013997799999999999 2023-11-05T08:32:53.819 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.819 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.820 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:32:53.820 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.098264+0000", 2023-11-05T08:32:53.820 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3793e-05 2023-11-05T08:32:53.820 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.820 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.820 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:32:53.821 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.099813+0000", 2023-11-05T08:32:53.821 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0015487910000000001 2023-11-05T08:32:53.821 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:32:53.821 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:32:53.821 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:32:53.822 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.099908+0000", 2023-11-05T08:32:53.822 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.5011000000000002e-05 2023-11-05T08:32:53.822 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:53.822 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:53.822 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:53.823 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:32:53.823 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:32:53.823 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:32:53.824 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-11-05T08:32:53.826 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:32:53.823+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:32:53.926 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-11-05T08:32:53.920+0000 7f966619d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35976) UID: 0 2023-11-05T08:32:53.957 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:32:53.957 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-11-05T08:32:53.957 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-11-05T08:32:53.957 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:32:53.971 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-11-05T08:32:54.059 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:32:54.056+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:32:54.118 INFO:tasks.rados.rados.0.smithi090.stdout:100: finishing write tid 1 to smithi09039252-8 2023-11-05T08:32:54.119 INFO:tasks.rados.rados.0.smithi090.stdout:100: finishing write tid 2 to smithi09039252-8 2023-11-05T08:32:54.119 INFO:tasks.rados.rados.0.smithi090.stdout:100: finishing write tid 3 to smithi09039252-8 2023-11-05T08:32:54.119 INFO:tasks.rados.rados.0.smithi090.stdout:100: finishing write tid 4 to smithi09039252-8 2023-11-05T08:32:54.119 INFO:tasks.rados.rados.0.smithi090.stdout:100: oid 8 updating version 0 to 23 2023-11-05T08:32:54.120 INFO:tasks.rados.rados.0.smithi090.stdout:100: oid 8 updating version 23 to 24 2023-11-05T08:32:54.120 INFO:tasks.rados.rados.0.smithi090.stdout:100: oid 8 updating version 24 to 25 2023-11-05T08:32:54.120 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 8 v 25 (ObjNum 66 snap 6 seq_num 66) dirty exists 2023-11-05T08:32:54.120 INFO:tasks.rados.rados.0.smithi090.stdout:100: left oid 8 (ObjNum 66 snap 6 seq_num 66) 2023-11-05T08:32:54.120 INFO:tasks.rados.rados.0.smithi090.stdout:104: finishing write tid 1 to smithi09039252-43 2023-11-05T08:32:54.121 INFO:tasks.rados.rados.0.smithi090.stdout:104: finishing write tid 2 to smithi09039252-43 2023-11-05T08:32:54.121 INFO:tasks.rados.rados.0.smithi090.stdout:104: finishing write tid 3 to smithi09039252-43 2023-11-05T08:32:54.121 INFO:tasks.rados.rados.0.smithi090.stdout:104: finishing write tid 4 to smithi09039252-43 2023-11-05T08:32:54.121 INFO:tasks.rados.rados.0.smithi090.stdout:104: oid 43 updating version 0 to 9 2023-11-05T08:32:54.122 INFO:tasks.rados.rados.0.smithi090.stdout:104: oid 43 version 9 is already newer than 8 2023-11-05T08:32:54.122 INFO:tasks.rados.rados.0.smithi090.stdout:104: oid 43 updating version 9 to 10 2023-11-05T08:32:54.122 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 43 v 10 (ObjNum 67 snap 6 seq_num 67) dirty exists 2023-11-05T08:32:54.122 INFO:tasks.rados.rados.0.smithi090.stdout:104: left oid 43 (ObjNum 67 snap 6 seq_num 67) 2023-11-05T08:32:54.123 INFO:tasks.rados.rados.0.smithi090.stdout:100: done (6 left) 2023-11-05T08:32:54.123 INFO:tasks.rados.rados.0.smithi090.stdout:104: done (5 left) 2023-11-05T08:32:54.123 INFO:tasks.rados.rados.0.smithi090.stdout:106: done (4 left) 2023-11-05T08:32:54.123 INFO:tasks.rados.rados.0.smithi090.stdout:107: delete oid 7 current snap is 6 2023-11-05T08:32:54.128 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:32:54.125+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:32:54.152 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:32:54.152 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-11-05T08:32:54.152 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-11-05T08:32:54.153 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-11-05T08:32:54.153 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:32:54.171 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-11-05T08:32:54.182 INFO:tasks.rados.rados.0.smithi090.stdout:107: done (4 left) 2023-11-05T08:32:54.183 INFO:tasks.rados.rados.0.smithi090.stdout:108: read oid 13 snap -1 2023-11-05T08:32:54.183 INFO:tasks.rados.rados.0.smithi090.stdout:108: expect (ObjNum 50 snap 0 seq_num 50) 2023-11-05T08:32:54.183 INFO:tasks.rados.rados.0.smithi090.stdout:109: snap_remove snap 3 2023-11-05T08:32:54.229 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:32:54.225+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:32:54.303 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:32:54.303 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-11-05T08:32:54.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-11-05T08:32:54.304 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-11-05T08:32:54.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.304 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.3s6 35/30 ECSubWrite(tid=179, reqid=client.4691.0:218, at_version=35'8, trim_to=0'0, roll_forward_to=33'6))", 2023-11-05T08:32:54.304 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:48.055170+0000", 2023-11-05T08:32:54.305 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 6.2453830049999999, 2023-11-05T08:32:54.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0044007339999999999, 2023-11-05T08:32:54.305 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:54.305 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:32:54.305 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:54.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:54.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:48.055170+0000", 2023-11-05T08:32:54.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:54.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:48.055170+0000", 2023-11-05T08:32:54.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:54.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:48.055170+0000", 2023-11-05T08:32:54.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9700000000000002e-07 2023-11-05T08:32:54.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:54.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:48.055235+0000", 2023-11-05T08:32:54.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5060000000000004e-05 2023-11-05T08:32:54.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:54.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:48.055266+0000", 2023-11-05T08:32:54.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0967999999999999e-05 2023-11-05T08:32:54.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:54.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:48.055292+0000", 2023-11-05T08:32:54.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6026000000000001e-05 2023-11-05T08:32:54.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:54.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:48.055462+0000", 2023-11-05T08:32:54.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000169202 2023-11-05T08:32:54.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:32:54.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:48.055481+0000", 2023-11-05T08:32:54.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9255000000000001e-05 2023-11-05T08:32:54.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:32:54.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:48.059476+0000", 2023-11-05T08:32:54.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003994984 2023-11-05T08:32:54.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:54.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:48.059571+0000", 2023-11-05T08:32:54.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4841999999999998e-05 2023-11-05T08:32:54.315 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.315 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:54.315 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.315 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.3s6 35/30 ECSubWrite(tid=180, reqid=client.4691.0:219, at_version=35'9, trim_to=0'0, roll_forward_to=33'6))", 2023-11-05T08:32:54.316 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:48.057759+0000", 2023-11-05T08:32:54.316 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 6.2427937819999997, 2023-11-05T08:32:54.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0044064580000000002, 2023-11-05T08:32:54.316 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:54.316 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:32:54.317 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:54.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:54.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:48.057759+0000", 2023-11-05T08:32:54.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:54.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:48.057759+0000", 2023-11-05T08:32:54.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:54.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:48.057761+0000", 2023-11-05T08:32:54.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7260000000000001e-06 2023-11-05T08:32:54.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:54.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:48.057810+0000", 2023-11-05T08:32:54.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9481000000000003e-05 2023-11-05T08:32:54.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:54.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:48.057844+0000", 2023-11-05T08:32:54.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3185000000000002e-05 2023-11-05T08:32:54.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:54.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:48.057939+0000", 2023-11-05T08:32:54.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5624999999999993e-05 2023-11-05T08:32:54.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:54.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:48.058172+0000", 2023-11-05T08:32:54.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000232628 2023-11-05T08:32:54.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:32:54.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:48.058196+0000", 2023-11-05T08:32:54.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4528999999999998e-05 2023-11-05T08:32:54.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:32:54.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:48.062107+0000", 2023-11-05T08:32:54.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039107819999999998 2023-11-05T08:32:54.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:54.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:48.062166+0000", 2023-11-05T08:32:54.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8502000000000003e-05 2023-11-05T08:32:54.329 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.329 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:54.329 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.330 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.3s6 35/30 ECSubWrite(tid=181, reqid=client.4691.0:220, at_version=35'10, trim_to=0'0, roll_forward_to=33'6))", 2023-11-05T08:32:54.330 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:48.059874+0000", 2023-11-05T08:32:54.330 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 6.240679407, 2023-11-05T08:32:54.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040051280000000002, 2023-11-05T08:32:54.331 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:54.331 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:32:54.331 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:54.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:54.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:48.059874+0000", 2023-11-05T08:32:54.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:54.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:48.059874+0000", 2023-11-05T08:32:54.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:54.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:48.059875+0000", 2023-11-05T08:32:54.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5269999999999999e-06 2023-11-05T08:32:54.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:54.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:48.059911+0000", 2023-11-05T08:32:54.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5515000000000001e-05 2023-11-05T08:32:54.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:54.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:48.059943+0000", 2023-11-05T08:32:54.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2648000000000003e-05 2023-11-05T08:32:54.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:54.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:48.059965+0000", 2023-11-05T08:32:54.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1549e-05 2023-11-05T08:32:54.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:54.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:48.060231+0000", 2023-11-05T08:32:54.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026619000000000002 2023-11-05T08:32:54.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:32:54.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:48.060256+0000", 2023-11-05T08:32:54.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4799000000000001e-05 2023-11-05T08:32:54.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:32:54.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:48.063807+0000", 2023-11-05T08:32:54.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0035512199999999999 2023-11-05T08:32:54.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:54.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:48.063879+0000", 2023-11-05T08:32:54.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1680000000000005e-05 2023-11-05T08:32:54.340 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.341 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:54.341 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.342 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.ds1 36/30 ECSubWrite(tid=20, reqid=client.4691.0:233, at_version=36'10, trim_to=0'0, roll_forward_to=34'8))", 2023-11-05T08:32:54.342 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.065953+0000", 2023-11-05T08:32:54.343 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.2346003720000001, 2023-11-05T08:32:54.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.007294994, 2023-11-05T08:32:54.344 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:54.344 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:32:54.344 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:54.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:54.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.065953+0000", 2023-11-05T08:32:54.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:54.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.065953+0000", 2023-11-05T08:32:54.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:54.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.065953+0000", 2023-11-05T08:32:54.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1699999999999997e-07 2023-11-05T08:32:54.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:54.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.066009+0000", 2023-11-05T08:32:54.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6100000000000002e-05 2023-11-05T08:32:54.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:54.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.066042+0000", 2023-11-05T08:32:54.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2709000000000001e-05 2023-11-05T08:32:54.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:54.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.066073+0000", 2023-11-05T08:32:54.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0531000000000003e-05 2023-11-05T08:32:54.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:54.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.066327+0000", 2023-11-05T08:32:54.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025432500000000002 2023-11-05T08:32:54.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.352 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:32:54.352 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.066388+0000", 2023-11-05T08:32:54.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.109e-05 2023-11-05T08:32:54.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.353 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:32:54.353 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.073147+0000", 2023-11-05T08:32:54.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0067585370000000002 2023-11-05T08:32:54.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:54.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.073248+0000", 2023-11-05T08:32:54.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100985 2023-11-05T08:32:54.355 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.355 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:54.355 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.356 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.3s6 36/30 ECSubWrite(tid=183, reqid=osd.13.0:183, at_version=36'12, trim_to=0'0, roll_forward_to=35'10))", 2023-11-05T08:32:54.356 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.070346+0000", 2023-11-05T08:32:54.356 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.2302066900000002, 2023-11-05T08:32:54.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052910370000000002, 2023-11-05T08:32:54.357 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:54.357 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:32:54.357 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:54.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:54.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.070346+0000", 2023-11-05T08:32:54.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:54.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.070346+0000", 2023-11-05T08:32:54.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:54.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.070347+0000", 2023-11-05T08:32:54.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4000000000000001e-07 2023-11-05T08:32:54.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:54.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.070475+0000", 2023-11-05T08:32:54.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000128243 2023-11-05T08:32:54.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:54.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.070508+0000", 2023-11-05T08:32:54.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2738999999999999e-05 2023-11-05T08:32:54.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:54.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.070533+0000", 2023-11-05T08:32:54.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4854999999999998e-05 2023-11-05T08:32:54.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:54.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.070856+0000", 2023-11-05T08:32:54.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032318400000000001 2023-11-05T08:32:54.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:32:54.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.070880+0000", 2023-11-05T08:32:54.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4332e-05 2023-11-05T08:32:54.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:32:54.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.075537+0000", 2023-11-05T08:32:54.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0046567020000000004 2023-11-05T08:32:54.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:54.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.075637+0000", 2023-11-05T08:32:54.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100342 2023-11-05T08:32:54.367 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.368 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:54.368 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.368 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.ds1 36/30 ECSubWrite(tid=21, reqid=client.4691.0:234, at_version=36'11, trim_to=0'0, roll_forward_to=34'8))", 2023-11-05T08:32:54.368 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.070693+0000", 2023-11-05T08:32:54.369 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.2298604869999998, 2023-11-05T08:32:54.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050258359999999997, 2023-11-05T08:32:54.369 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:54.369 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:32:54.369 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:54.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:54.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.070693+0000", 2023-11-05T08:32:54.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:54.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.070693+0000", 2023-11-05T08:32:54.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:54.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.070695+0000", 2023-11-05T08:32:54.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.046e-06 2023-11-05T08:32:54.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:54.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.070744+0000", 2023-11-05T08:32:54.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9224000000000002e-05 2023-11-05T08:32:54.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:54.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.070777+0000", 2023-11-05T08:32:54.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3382e-05 2023-11-05T08:32:54.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:54.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.070800+0000", 2023-11-05T08:32:54.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2498999999999999e-05 2023-11-05T08:32:54.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:54.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.071798+0000", 2023-11-05T08:32:54.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00099803700000000001 2023-11-05T08:32:54.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:32:54.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.071816+0000", 2023-11-05T08:32:54.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8278999999999999e-05 2023-11-05T08:32:54.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:32:54.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.075638+0000", 2023-11-05T08:32:54.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038223200000000001 2023-11-05T08:32:54.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:54.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.075718+0000", 2023-11-05T08:32:54.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0049000000000006e-05 2023-11-05T08:32:54.382 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.382 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:54.383 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.383 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.ds1 36/30 ECSubWrite(tid=22, reqid=client.4691.0:235, at_version=36'12, trim_to=0'0, roll_forward_to=34'8))", 2023-11-05T08:32:54.383 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.077689+0000", 2023-11-05T08:32:54.384 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.2228637119999997, 2023-11-05T08:32:54.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0060916909999999998, 2023-11-05T08:32:54.384 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:54.384 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:32:54.384 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:54.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:54.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.077689+0000", 2023-11-05T08:32:54.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:54.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.077689+0000", 2023-11-05T08:32:54.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:54.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.077691+0000", 2023-11-05T08:32:54.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8700000000000001e-06 2023-11-05T08:32:54.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:54.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.077736+0000", 2023-11-05T08:32:54.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5238000000000003e-05 2023-11-05T08:32:54.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.389 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:54.389 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.077771+0000", 2023-11-05T08:32:54.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5005000000000001e-05 2023-11-05T08:32:54.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:54.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.077805+0000", 2023-11-05T08:32:54.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3095000000000001e-05 2023-11-05T08:32:54.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:54.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.077952+0000", 2023-11-05T08:32:54.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000147696 2023-11-05T08:32:54.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:32:54.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.077977+0000", 2023-11-05T08:32:54.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4269000000000001e-05 2023-11-05T08:32:54.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:32:54.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.083688+0000", 2023-11-05T08:32:54.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0057118159999999998 2023-11-05T08:32:54.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:54.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.083781+0000", 2023-11-05T08:32:54.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2701999999999994e-05 2023-11-05T08:32:54.394 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.394 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:54.395 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.395 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.ds1 36/30 ECSubWrite(tid=23, reqid=osd.7.0:23, at_version=36'14, trim_to=0'0, roll_forward_to=34'8))", 2023-11-05T08:32:54.395 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.077873+0000", 2023-11-05T08:32:54.395 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.2226799440000002, 2023-11-05T08:32:54.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0059694570000000001, 2023-11-05T08:32:54.396 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:54.396 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:32:54.396 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:54.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:54.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.077873+0000", 2023-11-05T08:32:54.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:54.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.077873+0000", 2023-11-05T08:32:54.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:54.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.077874+0000", 2023-11-05T08:32:54.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5999999999999991e-07 2023-11-05T08:32:54.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:54.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.077957+0000", 2023-11-05T08:32:54.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3338999999999996e-05 2023-11-05T08:32:54.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:54.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.077983+0000", 2023-11-05T08:32:54.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5548999999999999e-05 2023-11-05T08:32:54.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:54.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.078001+0000", 2023-11-05T08:32:54.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8482e-05 2023-11-05T08:32:54.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:54.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.078500+0000", 2023-11-05T08:32:54.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049819199999999995 2023-11-05T08:32:54.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:32:54.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.078522+0000", 2023-11-05T08:32:54.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2282000000000001e-05 2023-11-05T08:32:54.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.407 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:32:54.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.083789+0000", 2023-11-05T08:32:54.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052672789999999997 2023-11-05T08:32:54.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:54.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.083843+0000", 2023-11-05T08:32:54.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3374000000000001e-05 2023-11-05T08:32:54.409 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.409 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:54.409 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.410 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.3s6 36/30 ECSubWrite(tid=184, reqid=osd.13.0:183, at_version=0'0, trim_to=0'0, roll_forward_to=36'12))", 2023-11-05T08:32:54.410 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.080069+0000", 2023-11-05T08:32:54.410 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.2204838100000002, 2023-11-05T08:32:54.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055189690000000003, 2023-11-05T08:32:54.410 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:54.411 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:32:54.411 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:54.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:54.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.080069+0000", 2023-11-05T08:32:54.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:54.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.080069+0000", 2023-11-05T08:32:54.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:54.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.080071+0000", 2023-11-05T08:32:54.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8700000000000001e-06 2023-11-05T08:32:54.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:54.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.080100+0000", 2023-11-05T08:32:54.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8932e-05 2023-11-05T08:32:54.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:54.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.080130+0000", 2023-11-05T08:32:54.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0215000000000001e-05 2023-11-05T08:32:54.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:54.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.080154+0000", 2023-11-05T08:32:54.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3376000000000001e-05 2023-11-05T08:32:54.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:54.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.080372+0000", 2023-11-05T08:32:54.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000218209 2023-11-05T08:32:54.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:32:54.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.080396+0000", 2023-11-05T08:32:54.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4445000000000001e-05 2023-11-05T08:32:54.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:32:54.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.085499+0000", 2023-11-05T08:32:54.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0051029700000000001 2023-11-05T08:32:54.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:54.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.085588+0000", 2023-11-05T08:32:54.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8951999999999998e-05 2023-11-05T08:32:54.420 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.420 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:54.421 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.421 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.ds1 36/30 ECSubWrite(tid=24, reqid=osd.7.0:23, at_version=0'0, trim_to=0'0, roll_forward_to=36'14))", 2023-11-05T08:32:54.421 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.086314+0000", 2023-11-05T08:32:54.421 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.2142391330000004, 2023-11-05T08:32:54.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036483319999999998, 2023-11-05T08:32:54.422 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:54.422 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:32:54.422 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:54.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:54.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.086314+0000", 2023-11-05T08:32:54.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:54.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.086314+0000", 2023-11-05T08:32:54.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:54.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.086316+0000", 2023-11-05T08:32:54.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0099999999999998e-06 2023-11-05T08:32:54.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:54.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.086344+0000", 2023-11-05T08:32:54.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8208000000000002e-05 2023-11-05T08:32:54.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:54.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.086388+0000", 2023-11-05T08:32:54.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4113999999999998e-05 2023-11-05T08:32:54.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:54.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.086413+0000", 2023-11-05T08:32:54.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4433000000000001e-05 2023-11-05T08:32:54.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:54.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.086810+0000", 2023-11-05T08:32:54.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039754599999999998 2023-11-05T08:32:54.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:32:54.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.086834+0000", 2023-11-05T08:32:54.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3629e-05 2023-11-05T08:32:54.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:32:54.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.089863+0000", 2023-11-05T08:32:54.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030289700000000002 2023-11-05T08:32:54.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:54.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.089962+0000", 2023-11-05T08:32:54.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9421999999999998e-05 2023-11-05T08:32:54.432 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.432 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:54.432 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.433 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.2s2 37/30 ECSubWrite(tid=185, reqid=client.4691.0:240, at_version=37'7, trim_to=0'0, roll_forward_to=33'5))", 2023-11-05T08:32:54.433 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:50.089130+0000", 2023-11-05T08:32:54.433 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.2114225530000002, 2023-11-05T08:32:54.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033691630000000001, 2023-11-05T08:32:54.433 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:54.434 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:32:54.434 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:54.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:54.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.089130+0000", 2023-11-05T08:32:54.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:54.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.089130+0000", 2023-11-05T08:32:54.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:54.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.089131+0000", 2023-11-05T08:32:54.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4700000000000001e-07 2023-11-05T08:32:54.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:54.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.089192+0000", 2023-11-05T08:32:54.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0446999999999999e-05 2023-11-05T08:32:54.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:54.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.089225+0000", 2023-11-05T08:32:54.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3751000000000003e-05 2023-11-05T08:32:54.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:54.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.089250+0000", 2023-11-05T08:32:54.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4519e-05 2023-11-05T08:32:54.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:54.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.089482+0000", 2023-11-05T08:32:54.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023183899999999999 2023-11-05T08:32:54.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:32:54.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.089513+0000", 2023-11-05T08:32:54.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0775000000000003e-05 2023-11-05T08:32:54.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:32:54.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.092411+0000", 2023-11-05T08:32:54.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028980159999999998 2023-11-05T08:32:54.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:54.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.092500+0000", 2023-11-05T08:32:54.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9068999999999997e-05 2023-11-05T08:32:54.443 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.444 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:54.444 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.444 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.as0 40/33 ECSubWrite(tid=30, reqid=client.4691.0:283, at_version=40'8, trim_to=0'0, roll_forward_to=39'6))", 2023-11-05T08:32:54.444 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:53.168485+0000", 2023-11-05T08:32:54.445 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.1320675499999999, 2023-11-05T08:32:54.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0057294080000000001, 2023-11-05T08:32:54.445 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:54.445 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:32:54.445 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:54.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:54.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.168485+0000", 2023-11-05T08:32:54.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:54.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.168485+0000", 2023-11-05T08:32:54.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:54.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.168487+0000", 2023-11-05T08:32:54.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2899999999999999e-06 2023-11-05T08:32:54.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:54.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.168989+0000", 2023-11-05T08:32:54.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050223399999999995 2023-11-05T08:32:54.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:54.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.169043+0000", 2023-11-05T08:32:54.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4197999999999998e-05 2023-11-05T08:32:54.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:54.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.169103+0000", 2023-11-05T08:32:54.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9917000000000002e-05 2023-11-05T08:32:54.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:54.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.169406+0000", 2023-11-05T08:32:54.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030249299999999998 2023-11-05T08:32:54.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:32:54.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.169438+0000", 2023-11-05T08:32:54.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2276000000000001e-05 2023-11-05T08:32:54.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:32:54.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.174110+0000", 2023-11-05T08:32:54.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0046725739999999997 2023-11-05T08:32:54.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:54.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.174215+0000", 2023-11-05T08:32:54.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010442600000000001 2023-11-05T08:32:54.455 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.455 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:54.455 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.456 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.as0 40/33 ECSubWrite(tid=31, reqid=client.4691.0:284, at_version=40'9, trim_to=0'0, roll_forward_to=39'6))", 2023-11-05T08:32:54.456 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:53.174093+0000", 2023-11-05T08:32:54.456 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.1264599040000001, 2023-11-05T08:32:54.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003856085, 2023-11-05T08:32:54.457 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:54.457 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:32:54.457 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:54.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:54.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.174093+0000", 2023-11-05T08:32:54.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:54.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.174093+0000", 2023-11-05T08:32:54.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:54.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.174095+0000", 2023-11-05T08:32:54.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9640000000000002e-06 2023-11-05T08:32:54.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:54.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.174321+0000", 2023-11-05T08:32:54.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000226343 2023-11-05T08:32:54.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:54.462 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.174351+0000", 2023-11-05T08:32:54.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9589000000000001e-05 2023-11-05T08:32:54.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.462 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:54.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.174375+0000", 2023-11-05T08:32:54.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3609e-05 2023-11-05T08:32:54.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:54.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.174548+0000", 2023-11-05T08:32:54.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000172929 2023-11-05T08:32:54.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:32:54.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.174578+0000", 2023-11-05T08:32:54.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0241000000000001e-05 2023-11-05T08:32:54.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:32:54.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.177852+0000", 2023-11-05T08:32:54.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032739309999999999 2023-11-05T08:32:54.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:54.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.177949+0000", 2023-11-05T08:32:54.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7479000000000006e-05 2023-11-05T08:32:54.467 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.467 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:54.467 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.468 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.as0 40/33 ECSubWrite(tid=33, reqid=client.4691.0:285, at_version=0'0, trim_to=0'0, roll_forward_to=40'10))", 2023-11-05T08:32:54.468 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:53.181600+0000", 2023-11-05T08:32:54.468 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.1189526860000001, 2023-11-05T08:32:54.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002502808, 2023-11-05T08:32:54.469 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:54.469 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:32:54.469 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:54.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:54.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.181600+0000", 2023-11-05T08:32:54.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:54.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.181600+0000", 2023-11-05T08:32:54.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:54.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.181603+0000", 2023-11-05T08:32:54.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2730000000000001e-06 2023-11-05T08:32:54.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:54.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.181644+0000", 2023-11-05T08:32:54.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0992000000000003e-05 2023-11-05T08:32:54.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:54.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.181707+0000", 2023-11-05T08:32:54.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3627000000000006e-05 2023-11-05T08:32:54.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:54.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.181732+0000", 2023-11-05T08:32:54.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5196000000000001e-05 2023-11-05T08:32:54.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:54.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.181913+0000", 2023-11-05T08:32:54.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000180602 2023-11-05T08:32:54.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:32:54.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.181942+0000", 2023-11-05T08:32:54.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8671e-05 2023-11-05T08:32:54.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:32:54.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.184020+0000", 2023-11-05T08:32:54.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020786839999999999 2023-11-05T08:32:54.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:54.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.184103+0000", 2023-11-05T08:32:54.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2762999999999993e-05 2023-11-05T08:32:54.478 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.479 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:54.479 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.479 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.as0 41/33 ECSubWrite(tid=35, reqid=osd.7.0:35, at_version=41'12, trim_to=0'0, roll_forward_to=40'10))", 2023-11-05T08:32:54.479 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:54.127965+0000", 2023-11-05T08:32:54.480 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.17258779099999999, 2023-11-05T08:32:54.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01003041, 2023-11-05T08:32:54.480 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:54.480 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:32:54.480 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:54.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:54.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.127965+0000", 2023-11-05T08:32:54.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:54.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.127965+0000", 2023-11-05T08:32:54.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:54.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.127966+0000", 2023-11-05T08:32:54.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5000000000000003e-07 2023-11-05T08:32:54.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:54.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.128020+0000", 2023-11-05T08:32:54.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4101000000000003e-05 2023-11-05T08:32:54.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:54.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.128052+0000", 2023-11-05T08:32:54.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2568999999999999e-05 2023-11-05T08:32:54.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:54.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.128076+0000", 2023-11-05T08:32:54.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3059000000000001e-05 2023-11-05T08:32:54.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:54.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.134590+0000", 2023-11-05T08:32:54.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0065148649999999999 2023-11-05T08:32:54.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:32:54.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.134627+0000", 2023-11-05T08:32:54.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6995000000000001e-05 2023-11-05T08:32:54.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:32:54.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.137963+0000", 2023-11-05T08:32:54.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033360439999999998 2023-11-05T08:32:54.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:54.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.137996+0000", 2023-11-05T08:32:54.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2227000000000002e-05 2023-11-05T08:32:54.490 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.490 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:54.490 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.491 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.as0 41/33 ECSubWrite(tid=36, reqid=osd.7.0:36, at_version=41'14, trim_to=0'0, roll_forward_to=40'10))", 2023-11-05T08:32:54.491 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:54.130247+0000", 2023-11-05T08:32:54.491 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.170306124, 2023-11-05T08:32:54.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011577618, 2023-11-05T08:32:54.492 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:54.492 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:32:54.492 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:54.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:54.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.130247+0000", 2023-11-05T08:32:54.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:54.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.130247+0000", 2023-11-05T08:32:54.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:54.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.130249+0000", 2023-11-05T08:32:54.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.764e-06 2023-11-05T08:32:54.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:54.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.130345+0000", 2023-11-05T08:32:54.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6042e-05 2023-11-05T08:32:54.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:54.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.130378+0000", 2023-11-05T08:32:54.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3464999999999999e-05 2023-11-05T08:32:54.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:54.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.130401+0000", 2023-11-05T08:32:54.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2996000000000001e-05 2023-11-05T08:32:54.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:54.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.136976+0000", 2023-11-05T08:32:54.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0065751159999999998 2023-11-05T08:32:54.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:32:54.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.137003+0000", 2023-11-05T08:32:54.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6251999999999999e-05 2023-11-05T08:32:54.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:32:54.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.141742+0000", 2023-11-05T08:32:54.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0047398400000000004 2023-11-05T08:32:54.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:54.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.141825+0000", 2023-11-05T08:32:54.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2143000000000003e-05 2023-11-05T08:32:54.502 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.502 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:54.502 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.503 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.2s2 41/40 ECSubWrite(tid=187, reqid=client.4691.0:292, at_version=41'9, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:32:54.503 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:54.161574+0000", 2023-11-05T08:32:54.503 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.13897915199999999, 2023-11-05T08:32:54.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036144240000000002, 2023-11-05T08:32:54.503 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:54.503 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:32:54.504 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:54.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:54.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.161574+0000", 2023-11-05T08:32:54.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:54.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.161574+0000", 2023-11-05T08:32:54.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:54.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.161575+0000", 2023-11-05T08:32:54.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.012e-06 2023-11-05T08:32:54.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:54.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.161608+0000", 2023-11-05T08:32:54.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3115999999999999e-05 2023-11-05T08:32:54.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:54.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.161626+0000", 2023-11-05T08:32:54.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7881999999999999e-05 2023-11-05T08:32:54.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:54.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.161653+0000", 2023-11-05T08:32:54.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7375000000000001e-05 2023-11-05T08:32:54.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:54.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.161839+0000", 2023-11-05T08:32:54.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000186143 2023-11-05T08:32:54.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:32:54.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.161867+0000", 2023-11-05T08:32:54.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7974999999999999e-05 2023-11-05T08:32:54.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:32:54.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.165105+0000", 2023-11-05T08:32:54.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032380590000000002 2023-11-05T08:32:54.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:54.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.165188+0000", 2023-11-05T08:32:54.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2861999999999994e-05 2023-11-05T08:32:54.513 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.513 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:54.514 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.514 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.as0 41/33 ECSubWrite(tid=37, reqid=osd.7.0:36, at_version=0'0, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:32:54.514 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:54.167904+0000", 2023-11-05T08:32:54.515 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.132649132, 2023-11-05T08:32:54.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002291216, 2023-11-05T08:32:54.515 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:54.515 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:32:54.515 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:54.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:54.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.167904+0000", 2023-11-05T08:32:54.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:54.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.167904+0000", 2023-11-05T08:32:54.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:54.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.167906+0000", 2023-11-05T08:32:54.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0169999999999999e-06 2023-11-05T08:32:54.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:54.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.167931+0000", 2023-11-05T08:32:54.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5544000000000001e-05 2023-11-05T08:32:54.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:54.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.167979+0000", 2023-11-05T08:32:54.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7126000000000002e-05 2023-11-05T08:32:54.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:54.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.167999+0000", 2023-11-05T08:32:54.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0290999999999999e-05 2023-11-05T08:32:54.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:54.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.168076+0000", 2023-11-05T08:32:54.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6754000000000004e-05 2023-11-05T08:32:54.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:32:54.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.168086+0000", 2023-11-05T08:32:54.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0528e-05 2023-11-05T08:32:54.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:32:54.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.170167+0000", 2023-11-05T08:32:54.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020805540000000001 2023-11-05T08:32:54.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:54.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.170195+0000", 2023-11-05T08:32:54.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8402e-05 2023-11-05T08:32:54.524 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.525 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:54.525 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.525 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.2s2 41/40 ECSubWrite(tid=188, reqid=osd.13.0:188, at_version=41'11, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:32:54.525 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:54.169325+0000", 2023-11-05T08:32:54.526 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.131227551, 2023-11-05T08:32:54.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002153903, 2023-11-05T08:32:54.526 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:54.526 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:32:54.526 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:54.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:54.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.169325+0000", 2023-11-05T08:32:54.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:54.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.169325+0000", 2023-11-05T08:32:54.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:54.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.169326+0000", 2023-11-05T08:32:54.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1399999999999996e-07 2023-11-05T08:32:54.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:54.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.169366+0000", 2023-11-05T08:32:54.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9505999999999997e-05 2023-11-05T08:32:54.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:54.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.169385+0000", 2023-11-05T08:32:54.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9462000000000001e-05 2023-11-05T08:32:54.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:54.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.169402+0000", 2023-11-05T08:32:54.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7124000000000001e-05 2023-11-05T08:32:54.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:54.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.169752+0000", 2023-11-05T08:32:54.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000349632 2023-11-05T08:32:54.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:32:54.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.169772+0000", 2023-11-05T08:32:54.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9820999999999999e-05 2023-11-05T08:32:54.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:32:54.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.171444+0000", 2023-11-05T08:32:54.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001672368 2023-11-05T08:32:54.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:54.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.171479+0000", 2023-11-05T08:32:54.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5176000000000003e-05 2023-11-05T08:32:54.536 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.536 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:54.536 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.537 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.13s6 41/40 ECSubWrite(tid=190, reqid=osd.13.0:190, at_version=41'14, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:32:54.537 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:54.189260+0000", 2023-11-05T08:32:54.537 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.111293015, 2023-11-05T08:32:54.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043001610000000003, 2023-11-05T08:32:54.537 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:54.538 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:32:54.538 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:54.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:54.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.189260+0000", 2023-11-05T08:32:54.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:54.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.189260+0000", 2023-11-05T08:32:54.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:54.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.189261+0000", 2023-11-05T08:32:54.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8000000000000005e-07 2023-11-05T08:32:54.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:54.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.189289+0000", 2023-11-05T08:32:54.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8285000000000001e-05 2023-11-05T08:32:54.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:54.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.189304+0000", 2023-11-05T08:32:54.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4533e-05 2023-11-05T08:32:54.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:54.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.189320+0000", 2023-11-05T08:32:54.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6064000000000001e-05 2023-11-05T08:32:54.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:54.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.189460+0000", 2023-11-05T08:32:54.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000139963 2023-11-05T08:32:54.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:32:54.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.189472+0000", 2023-11-05T08:32:54.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2288e-05 2023-11-05T08:32:54.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:32:54.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.193464+0000", 2023-11-05T08:32:54.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039923550000000004 2023-11-05T08:32:54.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:54.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.193560+0000", 2023-11-05T08:32:54.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5892999999999998e-05 2023-11-05T08:32:54.547 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.547 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:54.548 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.548 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:54.548 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:54.548 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:32:54.550 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-11-05T08:32:54.551 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:32:54.547+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:32:54.650 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:32:54.644+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:32:54.681 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:32:54.681 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-11-05T08:32:54.681 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-11-05T08:32:54.681 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:32:54.695 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-11-05T08:32:54.751 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:32:54.749+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:32:54.852 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:32:54.847+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:32:54.855 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:32:54.855 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-11-05T08:32:54.856 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-11-05T08:32:54.856 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-11-05T08:32:54.856 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:32:54.869 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-11-05T08:32:54.953 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-11-05T08:32:54.948+0000 7f966619d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35976) UID: 0 2023-11-05T08:32:54.988 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:32:54.989 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-11-05T08:32:54.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-11-05T08:32:54.989 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-11-05T08:32:54.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.990 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:233 3.ds0 3:bf7e600d:::smithi09039252-23:head [append 0~0] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2023-11-05T08:32:54.990 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.059123+0000", 2023-11-05T08:32:54.990 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.9268160879999998, 2023-11-05T08:32:54.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019416857999999999, 2023-11-05T08:32:54.990 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:54.991 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:54.991 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:32:54.991 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:32:54.991 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:32:54.992 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 233 2023-11-05T08:32:54.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.992 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:54.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:54.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059123+0000", 2023-11-05T08:32:54.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:54.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059123+0000", 2023-11-05T08:32:54.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:54.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:54.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059126+0000", 2023-11-05T08:32:54.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8370000000000001e-06 2023-11-05T08:32:54.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:54.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059140+0000", 2023-11-05T08:32:54.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3669000000000001e-05 2023-11-05T08:32:54.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:54.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059176+0000", 2023-11-05T08:32:54.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6548000000000003e-05 2023-11-05T08:32:54.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:54.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059198+0000", 2023-11-05T08:32:54.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1489e-05 2023-11-05T08:32:54.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:54.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059401+0000", 2023-11-05T08:32:54.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020313300000000001 2023-11-05T08:32:54.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:54.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:54.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-11-05T08:32:54.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059422+0000", 2023-11-05T08:32:55.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1212e-05 2023-11-05T08:32:55.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:55.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.063981+0000", 2023-11-05T08:32:55.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004558604 2023-11-05T08:32:55.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:32:55.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.064128+0000", 2023-11-05T08:32:55.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014734899999999999 2023-11-05T08:32:55.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:32:55.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.066042+0000", 2023-11-05T08:32:55.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019138079999999999 2023-11-05T08:32:55.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:32:55.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.077186+0000", 2023-11-05T08:32:55.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011144536 2023-11-05T08:32:55.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:32:55.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.078357+0000", 2023-11-05T08:32:55.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011709089999999999 2023-11-05T08:32:55.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:55.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.078540+0000", 2023-11-05T08:32:55.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000182764 2023-11-05T08:32:55.005 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.006 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:55.006 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.006 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:234 3.ds0 3:bf7e600d:::smithi09039252-23:head [rollback 3] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2023-11-05T08:32:55.007 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.059247+0000", 2023-11-05T08:32:55.007 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.9266924909999998, 2023-11-05T08:32:55.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019739059, 2023-11-05T08:32:55.007 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:55.007 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:55.008 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:32:55.008 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:32:55.008 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:32:55.008 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 234 2023-11-05T08:32:55.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.009 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:55.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:55.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059247+0000", 2023-11-05T08:32:55.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:55.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:55.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059247+0000", 2023-11-05T08:32:55.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:55.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:55.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059248+0000", 2023-11-05T08:32:55.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.077e-06 2023-11-05T08:32:55.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:55.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059256+0000", 2023-11-05T08:32:55.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1359999999999997e-06 2023-11-05T08:32:55.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:55.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059283+0000", 2023-11-05T08:32:55.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6855e-05 2023-11-05T08:32:55.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:55.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059298+0000", 2023-11-05T08:32:55.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5356e-05 2023-11-05T08:32:55.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:55.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059582+0000", 2023-11-05T08:32:55.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028341299999999999 2023-11-05T08:32:55.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:32:55.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059592+0000", 2023-11-05T08:32:55.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0103e-05 2023-11-05T08:32:55.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:55.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.067711+0000", 2023-11-05T08:32:55.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0081197660000000005 2023-11-05T08:32:55.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:32:55.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.067965+0000", 2023-11-05T08:32:55.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025397000000000001 2023-11-05T08:32:55.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:32:55.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.069296+0000", 2023-11-05T08:32:55.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013308910000000001 2023-11-05T08:32:55.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:32:55.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.077189+0000", 2023-11-05T08:32:55.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0078931079999999994 2023-11-05T08:32:55.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:32:55.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.078829+0000", 2023-11-05T08:32:55.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016394459999999999 2023-11-05T08:32:55.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:55.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.078986+0000", 2023-11-05T08:32:55.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015693799999999999 2023-11-05T08:32:55.023 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.023 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:55.023 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.024 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:235 3.ds0 3:bf7e600d:::smithi09039252-23:head [append 0~0] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2023-11-05T08:32:55.024 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.059370+0000", 2023-11-05T08:32:55.024 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.9265692770000005, 2023-11-05T08:32:55.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025190444999999999, 2023-11-05T08:32:55.025 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:55.025 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:55.025 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:32:55.025 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:32:55.026 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:32:55.026 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 235 2023-11-05T08:32:55.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.026 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:55.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:55.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059370+0000", 2023-11-05T08:32:55.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:55.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:55.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059370+0000", 2023-11-05T08:32:55.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:55.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:55.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059371+0000", 2023-11-05T08:32:55.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2899999999999999e-06 2023-11-05T08:32:55.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:55.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059378+0000", 2023-11-05T08:32:55.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0729999999999999e-06 2023-11-05T08:32:55.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:55.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059408+0000", 2023-11-05T08:32:55.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9904999999999999e-05 2023-11-05T08:32:55.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:55.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059418+0000", 2023-11-05T08:32:55.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7659999999999993e-06 2023-11-05T08:32:55.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:55.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059648+0000", 2023-11-05T08:32:55.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022963200000000001 2023-11-05T08:32:55.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:32:55.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059657+0000", 2023-11-05T08:32:55.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9700000000000005e-06 2023-11-05T08:32:55.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:55.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.070350+0000", 2023-11-05T08:32:55.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010693342 2023-11-05T08:32:55.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:32:55.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.070601+0000", 2023-11-05T08:32:55.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025082000000000001 2023-11-05T08:32:55.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:32:55.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.071898+0000", 2023-11-05T08:32:55.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012976190000000001 2023-11-05T08:32:55.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:32:55.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.077192+0000", 2023-11-05T08:32:55.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052931840000000003 2023-11-05T08:32:55.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:32:55.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.084398+0000", 2023-11-05T08:32:55.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0072062619999999997 2023-11-05T08:32:55.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:55.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.084560+0000", 2023-11-05T08:32:55.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016258200000000001 2023-11-05T08:32:55.039 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.039 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:55.040 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.040 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.3s1 36/30 ECSubWrite(tid=183, reqid=osd.13.0:183, at_version=36'12, trim_to=0'0, roll_forward_to=35'10))", 2023-11-05T08:32:55.040 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.070500+0000", 2023-11-05T08:32:55.041 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.9154396340000002, 2023-11-05T08:32:55.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0046789010000000001, 2023-11-05T08:32:55.041 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:55.041 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:32:55.041 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:55.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:55.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.070500+0000", 2023-11-05T08:32:55.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:55.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:55.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.070500+0000", 2023-11-05T08:32:55.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:55.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:55.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.070500+0000", 2023-11-05T08:32:55.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7e-07 2023-11-05T08:32:55.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:55.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.070555+0000", 2023-11-05T08:32:55.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4954000000000003e-05 2023-11-05T08:32:55.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:55.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.070585+0000", 2023-11-05T08:32:55.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0264999999999999e-05 2023-11-05T08:32:55.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:55.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.070610+0000", 2023-11-05T08:32:55.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4569000000000001e-05 2023-11-05T08:32:55.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:55.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.070856+0000", 2023-11-05T08:32:55.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024564200000000002 2023-11-05T08:32:55.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:32:55.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.070880+0000", 2023-11-05T08:32:55.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4915000000000001e-05 2023-11-05T08:32:55.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:32:55.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.075078+0000", 2023-11-05T08:32:55.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0041974009999999999 2023-11-05T08:32:55.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:55.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.075179+0000", 2023-11-05T08:32:55.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100685 2023-11-05T08:32:55.051 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.051 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:55.051 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.052 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.5s4 36/30 ECSubWrite(tid=25, reqid=osd.15.0:25, at_version=36'15, trim_to=0'0, roll_forward_to=35'13))", 2023-11-05T08:32:55.052 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.076226+0000", 2023-11-05T08:32:55.052 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.9097135410000003, 2023-11-05T08:32:55.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0086175350000000008, 2023-11-05T08:32:55.052 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:55.053 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:32:55.053 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:55.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:55.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.076226+0000", 2023-11-05T08:32:55.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:55.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:55.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.076226+0000", 2023-11-05T08:32:55.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:55.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:55.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.076226+0000", 2023-11-05T08:32:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9599999999999999e-07 2023-11-05T08:32:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.076281+0000", 2023-11-05T08:32:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4577000000000003e-05 2023-11-05T08:32:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.076307+0000", 2023-11-05T08:32:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6196000000000001e-05 2023-11-05T08:32:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.076326+0000", 2023-11-05T08:32:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8872000000000002e-05 2023-11-05T08:32:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.077601+0000", 2023-11-05T08:32:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012746669999999999 2023-11-05T08:32:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:32:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.077622+0000", 2023-11-05T08:32:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1396e-05 2023-11-05T08:32:55.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:32:55.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.084758+0000", 2023-11-05T08:32:55.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0071358919999999996 2023-11-05T08:32:55.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:55.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.084843+0000", 2023-11-05T08:32:55.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5239000000000002e-05 2023-11-05T08:32:55.062 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.063 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:55.063 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.063 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.3s1 36/30 ECSubWrite(tid=184, reqid=osd.13.0:183, at_version=0'0, trim_to=0'0, roll_forward_to=36'12))", 2023-11-05T08:32:55.064 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.080158+0000", 2023-11-05T08:32:55.064 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.9057814730000002, 2023-11-05T08:32:55.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0042752110000000001, 2023-11-05T08:32:55.064 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:55.064 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:32:55.065 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:55.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:55.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.080158+0000", 2023-11-05T08:32:55.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:55.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:55.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.080158+0000", 2023-11-05T08:32:55.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:55.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:55.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.080160+0000", 2023-11-05T08:32:55.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.827e-06 2023-11-05T08:32:55.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:55.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.080195+0000", 2023-11-05T08:32:55.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4941000000000003e-05 2023-11-05T08:32:55.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:55.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.080227+0000", 2023-11-05T08:32:55.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2598999999999997e-05 2023-11-05T08:32:55.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:55.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.080249+0000", 2023-11-05T08:32:55.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1974999999999999e-05 2023-11-05T08:32:55.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:55.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.080436+0000", 2023-11-05T08:32:55.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018702399999999999 2023-11-05T08:32:55.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:32:55.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.080459+0000", 2023-11-05T08:32:55.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2959000000000002e-05 2023-11-05T08:32:55.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:32:55.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.084341+0000", 2023-11-05T08:32:55.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038819100000000001 2023-11-05T08:32:55.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:55.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.084433+0000", 2023-11-05T08:32:55.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1976000000000002e-05 2023-11-05T08:32:55.074 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.074 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:55.074 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.075 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 36/30 ECSubWriteReply(tid=23, last_complete=36'14, committed=1, applied=1))", 2023-11-05T08:32:55.075 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.081662+0000", 2023-11-05T08:32:55.075 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.9042776110000004, 2023-11-05T08:32:55.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021258940000000001, 2023-11-05T08:32:55.076 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:55.076 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:32:55.076 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:55.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:55.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.081662+0000", 2023-11-05T08:32:55.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:55.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:55.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.081662+0000", 2023-11-05T08:32:55.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:55.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:55.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.081664+0000", 2023-11-05T08:32:55.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1730000000000002e-06 2023-11-05T08:32:55.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:55.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.081675+0000", 2023-11-05T08:32:55.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0847e-05 2023-11-05T08:32:55.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:55.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.081698+0000", 2023-11-05T08:32:55.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3268000000000002e-05 2023-11-05T08:32:55.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:55.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.081717+0000", 2023-11-05T08:32:55.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9046e-05 2023-11-05T08:32:55.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:55.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.083730+0000", 2023-11-05T08:32:55.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020127330000000001 2023-11-05T08:32:55.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:55.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.083788+0000", 2023-11-05T08:32:55.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7827e-05 2023-11-05T08:32:55.084 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.084 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:55.084 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.085 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 36/30 ECSubWriteReply(tid=23, last_complete=36'14, committed=1, applied=1))", 2023-11-05T08:32:55.085 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.085216+0000", 2023-11-05T08:32:55.085 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.9007231640000004, 2023-11-05T08:32:55.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002133107, 2023-11-05T08:32:55.085 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:55.085 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:32:55.086 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:55.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:55.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.085216+0000", 2023-11-05T08:32:55.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:55.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:55.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.085216+0000", 2023-11-05T08:32:55.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:55.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:55.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.085218+0000", 2023-11-05T08:32:55.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6130000000000001e-06 2023-11-05T08:32:55.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:55.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.085228+0000", 2023-11-05T08:32:55.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0796e-05 2023-11-05T08:32:55.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:55.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.085263+0000", 2023-11-05T08:32:55.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4088000000000003e-05 2023-11-05T08:32:55.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:55.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.085282+0000", 2023-11-05T08:32:55.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9409e-05 2023-11-05T08:32:55.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:55.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.085475+0000", 2023-11-05T08:32:55.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000192924 2023-11-05T08:32:55.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:55.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.087349+0000", 2023-11-05T08:32:55.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0018742769999999999 2023-11-05T08:32:55.093 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.093 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:55.094 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.094 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.5s4 36/30 ECSubWrite(tid=26, reqid=osd.15.0:25, at_version=0'0, trim_to=0'0, roll_forward_to=36'15))", 2023-11-05T08:32:55.094 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.088451+0000", 2023-11-05T08:32:55.095 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.8974884479999998, 2023-11-05T08:32:55.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0069175390000000003, 2023-11-05T08:32:55.095 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:55.095 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:32:55.095 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:55.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:55.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.088451+0000", 2023-11-05T08:32:55.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:55.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:55.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.088451+0000", 2023-11-05T08:32:55.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:55.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:55.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.088452+0000", 2023-11-05T08:32:55.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5060000000000001e-06 2023-11-05T08:32:55.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:55.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.088477+0000", 2023-11-05T08:32:55.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4814999999999999e-05 2023-11-05T08:32:55.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:55.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.088536+0000", 2023-11-05T08:32:55.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8788e-05 2023-11-05T08:32:55.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:55.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.088559+0000", 2023-11-05T08:32:55.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3281999999999998e-05 2023-11-05T08:32:55.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:55.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.088930+0000", 2023-11-05T08:32:55.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000370431 2023-11-05T08:32:55.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:32:55.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.088953+0000", 2023-11-05T08:32:55.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3013000000000001e-05 2023-11-05T08:32:55.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:32:55.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.095297+0000", 2023-11-05T08:32:55.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0063447240000000004 2023-11-05T08:32:55.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:55.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.095368+0000", 2023-11-05T08:32:55.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0980000000000001e-05 2023-11-05T08:32:55.105 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.105 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:55.105 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.106 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:283 3.as1 3:5396e455:::smithi09039252-43:head [append 0~753664] snapc 8=[8,7,6,5,3] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:32:55.106 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:53.164549+0000", 2023-11-05T08:32:55.106 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.8213897669999999, 2023-11-05T08:32:55.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012425772, 2023-11-05T08:32:55.106 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:55.107 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:55.107 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:32:55.107 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:32:55.107 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:32:55.107 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 283 2023-11-05T08:32:55.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.108 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:55.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:55.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.164549+0000", 2023-11-05T08:32:55.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:55.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:55.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.164549+0000", 2023-11-05T08:32:55.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:55.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:55.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.164552+0000", 2023-11-05T08:32:55.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9069999999999999e-06 2023-11-05T08:32:55.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:55.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.165387+0000", 2023-11-05T08:32:55.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00083439899999999999 2023-11-05T08:32:55.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:55.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.165421+0000", 2023-11-05T08:32:55.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4344999999999997e-05 2023-11-05T08:32:55.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:55.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.165439+0000", 2023-11-05T08:32:55.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7793000000000001e-05 2023-11-05T08:32:55.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:55.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.165639+0000", 2023-11-05T08:32:55.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019997400000000001 2023-11-05T08:32:55.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:32:55.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.165813+0000", 2023-11-05T08:32:55.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017405599999999999 2023-11-05T08:32:55.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:32:55.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.167863+0000", 2023-11-05T08:32:55.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002050315 2023-11-05T08:32:55.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:32:55.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.176139+0000", 2023-11-05T08:32:55.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0082754379999999995 2023-11-05T08:32:55.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:32:55.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.176765+0000", 2023-11-05T08:32:55.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00062584200000000004 2023-11-05T08:32:55.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:55.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.176975+0000", 2023-11-05T08:32:55.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021070299999999999 2023-11-05T08:32:55.119 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.119 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:55.119 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.120 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:284 3.as1 3:5396e455:::smithi09039252-43:head [append 0~376832] snapc 8=[8,7,6,5,3] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:32:55.120 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:53.169363+0000", 2023-11-05T08:32:55.120 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.816576202, 2023-11-05T08:32:55.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010455229999999999, 2023-11-05T08:32:55.121 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:55.121 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:55.121 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:32:55.121 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:32:55.121 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:32:55.122 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 284 2023-11-05T08:32:55.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.122 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:55.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:55.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.169363+0000", 2023-11-05T08:32:55.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:55.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:55.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.169363+0000", 2023-11-05T08:32:55.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:55.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:55.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.169366+0000", 2023-11-05T08:32:55.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1839999999999999e-06 2023-11-05T08:32:55.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:55.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.169824+0000", 2023-11-05T08:32:55.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045731599999999998 2023-11-05T08:32:55.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:55.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.169866+0000", 2023-11-05T08:32:55.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2042000000000002e-05 2023-11-05T08:32:55.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:55.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.169891+0000", 2023-11-05T08:32:55.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5122000000000001e-05 2023-11-05T08:32:55.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:55.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.172059+0000", 2023-11-05T08:32:55.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021685440000000001 2023-11-05T08:32:55.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:32:55.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.172256+0000", 2023-11-05T08:32:55.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000197173 2023-11-05T08:32:55.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:32:55.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.173722+0000", 2023-11-05T08:32:55.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001465491 2023-11-05T08:32:55.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:32:55.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.178878+0000", 2023-11-05T08:32:55.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0051563329999999999 2023-11-05T08:32:55.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:32:55.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.179641+0000", 2023-11-05T08:32:55.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000762566 2023-11-05T08:32:55.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:55.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.179818+0000", 2023-11-05T08:32:55.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017745900000000001 2023-11-05T08:32:55.133 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.133 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:55.134 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.134 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:285 3.as1 3:5396e455:::smithi09039252-43:head [setxattr _header (56) in=7b] snapc 8=[8,7,6,5,3] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:32:55.134 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:53.169940+0000", 2023-11-05T08:32:55.134 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.8159995900000001, 2023-11-05T08:32:55.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011368382, 2023-11-05T08:32:55.135 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:55.135 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:32:55.135 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:32:55.135 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:32:55.136 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:32:55.136 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 285 2023-11-05T08:32:55.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.136 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:55.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:55.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.169940+0000", 2023-11-05T08:32:55.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:55.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:55.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.169940+0000", 2023-11-05T08:32:55.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:55.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:55.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.169941+0000", 2023-11-05T08:32:55.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7889999999999999e-06 2023-11-05T08:32:55.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:55.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.169951+0000", 2023-11-05T08:32:55.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6530000000000006e-06 2023-11-05T08:32:55.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:55.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.169973+0000", 2023-11-05T08:32:55.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2028999999999999e-05 2023-11-05T08:32:55.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:55.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.169990+0000", 2023-11-05T08:32:55.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7082999999999999e-05 2023-11-05T08:32:55.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:55.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.177259+0000", 2023-11-05T08:32:55.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.007269045 2023-11-05T08:32:55.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:32:55.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.177423+0000", 2023-11-05T08:32:55.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016353599999999999 2023-11-05T08:32:55.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:32:55.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.178314+0000", 2023-11-05T08:32:55.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089105399999999996 2023-11-05T08:32:55.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:32:55.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.180575+0000", 2023-11-05T08:32:55.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022613249999999998 2023-11-05T08:32:55.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:32:55.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.181127+0000", 2023-11-05T08:32:55.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00055179600000000001 2023-11-05T08:32:55.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:55.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.181308+0000", 2023-11-05T08:32:55.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000181072 2023-11-05T08:32:55.147 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.148 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:55.148 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.148 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:286 3.as1 3:5396e455:::smithi09039252-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:32:55.149 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:53.170019+0000", 2023-11-05T08:32:55.149 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.8159198509999999, 2023-11-05T08:32:55.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014799489000000001, 2023-11-05T08:32:55.149 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:55.149 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:32:55.149 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:32:55.150 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:32:55.150 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:32:55.150 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 286 2023-11-05T08:32:55.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.150 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:55.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:55.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.170019+0000", 2023-11-05T08:32:55.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:55.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:55.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.170019+0000", 2023-11-05T08:32:55.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:55.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:55.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.170021+0000", 2023-11-05T08:32:55.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.877e-06 2023-11-05T08:32:55.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:55.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.170029+0000", 2023-11-05T08:32:55.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6960000000000005e-06 2023-11-05T08:32:55.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:55.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.170056+0000", 2023-11-05T08:32:55.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6928999999999999e-05 2023-11-05T08:32:55.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:55.156 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.170072+0000", 2023-11-05T08:32:55.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6515999999999999e-05 2023-11-05T08:32:55.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.156 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:55.156 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.180292+0000", 2023-11-05T08:32:55.157 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010219293000000001 2023-11-05T08:32:55.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.157 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-11-05T08:32:55.157 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.180558+0000", 2023-11-05T08:32:55.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026664499999999998 2023-11-05T08:32:55.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.158 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:55.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.182159+0000", 2023-11-05T08:32:55.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016003650000000001 2023-11-05T08:32:55.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:32:55.159 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.182378+0000", 2023-11-05T08:32:55.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021884699999999999 2023-11-05T08:32:55.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:55.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.184819+0000", 2023-11-05T08:32:55.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024413210000000002 2023-11-05T08:32:55.161 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.161 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:55.161 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.162 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWriteReply(3.as1 40/33 ECSubWriteReply(tid=30, last_complete=40'8, committed=1, applied=1))", 2023-11-05T08:32:55.162 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:53.171451+0000", 2023-11-05T08:32:55.162 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.8144878009999998, 2023-11-05T08:32:55.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0042832850000000004, 2023-11-05T08:32:55.162 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:55.163 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:32:55.163 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:55.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:55.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.171451+0000", 2023-11-05T08:32:55.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:55.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.164 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:55.164 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.171451+0000", 2023-11-05T08:32:55.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:55.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:55.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.171454+0000", 2023-11-05T08:32:55.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0779999999999998e-06 2023-11-05T08:32:55.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:55.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.171467+0000", 2023-11-05T08:32:55.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3144e-05 2023-11-05T08:32:55.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:55.167 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.171491+0000", 2023-11-05T08:32:55.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4413000000000002e-05 2023-11-05T08:32:55.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:55.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.171505+0000", 2023-11-05T08:32:55.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3658999999999999e-05 2023-11-05T08:32:55.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:55.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.175704+0000", 2023-11-05T08:32:55.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0041990739999999997 2023-11-05T08:32:55.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:55.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.175735+0000", 2023-11-05T08:32:55.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0917000000000002e-05 2023-11-05T08:32:55.170 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.171 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:55.171 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.171 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWriteReply(3.as1 40/33 ECSubWriteReply(tid=30, last_complete=40'8, committed=1, applied=1))", 2023-11-05T08:32:55.171 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:53.172066+0000", 2023-11-05T08:32:55.172 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.8138731720000001, 2023-11-05T08:32:55.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003834128, 2023-11-05T08:32:55.172 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:55.172 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:32:55.172 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:55.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:55.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.172066+0000", 2023-11-05T08:32:55.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:55.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:55.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.172066+0000", 2023-11-05T08:32:55.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:55.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:55.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.172068+0000", 2023-11-05T08:32:55.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0169999999999999e-06 2023-11-05T08:32:55.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:55.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.172081+0000", 2023-11-05T08:32:55.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2576e-05 2023-11-05T08:32:55.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:55.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.172114+0000", 2023-11-05T08:32:55.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3121999999999999e-05 2023-11-05T08:32:55.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:55.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.172136+0000", 2023-11-05T08:32:55.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1792000000000001e-05 2023-11-05T08:32:55.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:55.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.175829+0000", 2023-11-05T08:32:55.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036938520000000001 2023-11-05T08:32:55.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:55.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.175900+0000", 2023-11-05T08:32:55.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0769e-05 2023-11-05T08:32:55.180 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.180 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:55.180 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.181 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWriteReply(3.as1 40/33 ECSubWriteReply(tid=30, last_complete=40'8, committed=1, applied=1))", 2023-11-05T08:32:55.181 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:53.172653+0000", 2023-11-05T08:32:55.181 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.8132863079999999, 2023-11-05T08:32:55.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003338054, 2023-11-05T08:32:55.182 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:55.182 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:32:55.182 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:55.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:55.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.172653+0000", 2023-11-05T08:32:55.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:55.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:55.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.172653+0000", 2023-11-05T08:32:55.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:55.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:55.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.172655+0000", 2023-11-05T08:32:55.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9829999999999999e-06 2023-11-05T08:32:55.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:55.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.172670+0000", 2023-11-05T08:32:55.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4620000000000001e-05 2023-11-05T08:32:55.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:55.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.172704+0000", 2023-11-05T08:32:55.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4830999999999999e-05 2023-11-05T08:32:55.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:55.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.172725+0000", 2023-11-05T08:32:55.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0815999999999999e-05 2023-11-05T08:32:55.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:55.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.175950+0000", 2023-11-05T08:32:55.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032252019999999999 2023-11-05T08:32:55.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:55.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.175991+0000", 2023-11-05T08:32:55.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0602000000000002e-05 2023-11-05T08:32:55.190 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.190 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:55.190 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.191 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWriteReply(3.as1 40/33 ECSubWriteReply(tid=30, last_complete=40'8, committed=1, applied=1))", 2023-11-05T08:32:55.191 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:53.172966+0000", 2023-11-05T08:32:55.191 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.8129734690000001, 2023-11-05T08:32:55.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034610589999999998, 2023-11-05T08:32:55.191 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:55.191 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:32:55.192 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:55.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:55.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.172966+0000", 2023-11-05T08:32:55.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:55.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:55.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.172966+0000", 2023-11-05T08:32:55.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:55.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:55.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.172967+0000", 2023-11-05T08:32:55.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.72e-06 2023-11-05T08:32:55.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:55.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.172978+0000", 2023-11-05T08:32:55.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0665999999999999e-05 2023-11-05T08:32:55.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:55.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.173008+0000", 2023-11-05T08:32:55.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9379000000000001e-05 2023-11-05T08:32:55.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:55.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.173026+0000", 2023-11-05T08:32:55.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8739000000000001e-05 2023-11-05T08:32:55.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:55.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.176351+0000", 2023-11-05T08:32:55.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003324455 2023-11-05T08:32:55.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:55.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.176427+0000", 2023-11-05T08:32:55.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6100000000000007e-05 2023-11-05T08:32:55.202 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.202 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:55.202 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.203 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWriteReply(3.as1 40/33 ECSubWriteReply(tid=30, last_complete=40'8, committed=1, applied=1))", 2023-11-05T08:32:55.203 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:53.173775+0000", 2023-11-05T08:32:55.203 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.812164688, 2023-11-05T08:32:55.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032428650000000002, 2023-11-05T08:32:55.203 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:55.204 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:32:55.204 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:55.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:55.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.173775+0000", 2023-11-05T08:32:55.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:55.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:55.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.173775+0000", 2023-11-05T08:32:55.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:55.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:55.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.173776+0000", 2023-11-05T08:32:55.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8190000000000001e-06 2023-11-05T08:32:55.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:55.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.173788+0000", 2023-11-05T08:32:55.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.182e-05 2023-11-05T08:32:55.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:55.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.173819+0000", 2023-11-05T08:32:55.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0672e-05 2023-11-05T08:32:55.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:55.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.173842+0000", 2023-11-05T08:32:55.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3485e-05 2023-11-05T08:32:55.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:55.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.176533+0000", 2023-11-05T08:32:55.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026905169999999999 2023-11-05T08:32:55.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:55.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.177017+0000", 2023-11-05T08:32:55.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00048455200000000002 2023-11-05T08:32:55.211 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.212 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:55.212 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.212 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWriteReply(3.as1 40/33 ECSubWriteReply(tid=31, last_complete=40'9, committed=1, applied=1))", 2023-11-05T08:32:55.212 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:53.177185+0000", 2023-11-05T08:32:55.213 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.8087541040000001, 2023-11-05T08:32:55.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020509249999999999, 2023-11-05T08:32:55.213 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:55.213 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:32:55.213 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:55.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:55.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.177185+0000", 2023-11-05T08:32:55.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:55.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:55.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.177185+0000", 2023-11-05T08:32:55.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:55.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:55.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.177187+0000", 2023-11-05T08:32:55.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.753e-06 2023-11-05T08:32:55.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:55.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.177199+0000", 2023-11-05T08:32:55.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1875999999999999e-05 2023-11-05T08:32:55.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:55.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.177234+0000", 2023-11-05T08:32:55.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5522000000000003e-05 2023-11-05T08:32:55.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:55.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.177250+0000", 2023-11-05T08:32:55.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5262999999999999e-05 2023-11-05T08:32:55.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:55.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.179192+0000", 2023-11-05T08:32:55.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019425429999999999 2023-11-05T08:32:55.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:55.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.179236+0000", 2023-11-05T08:32:55.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3968000000000003e-05 2023-11-05T08:32:55.221 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.221 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:55.221 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.222 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.13s1 41/40 ECSubWrite(tid=190, reqid=osd.13.0:190, at_version=41'14, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:32:55.222 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:54.189311+0000", 2023-11-05T08:32:55.222 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.79662787999999995, 2023-11-05T08:32:55.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039707900000000001, 2023-11-05T08:32:55.223 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:32:55.223 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:32:55.223 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:32:55.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:32:55.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.189311+0000", 2023-11-05T08:32:55.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:55.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:32:55.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.189311+0000", 2023-11-05T08:32:55.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:32:55.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:32:55.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.189312+0000", 2023-11-05T08:32:55.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8599999999999996e-07 2023-11-05T08:32:55.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:32:55.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.189340+0000", 2023-11-05T08:32:55.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7606e-05 2023-11-05T08:32:55.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:32:55.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.189361+0000", 2023-11-05T08:32:55.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.084e-05 2023-11-05T08:32:55.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:32:55.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.189372+0000", 2023-11-05T08:32:55.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1609e-05 2023-11-05T08:32:55.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:32:55.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.189445+0000", 2023-11-05T08:32:55.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2955999999999997e-05 2023-11-05T08:32:55.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:32:55.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.189455+0000", 2023-11-05T08:32:55.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0077e-05 2023-11-05T08:32:55.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:32:55.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.193227+0000", 2023-11-05T08:32:55.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037716189999999999 2023-11-05T08:32:55.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:32:55.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:32:55.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:32:55.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.193282+0000", 2023-11-05T08:32:55.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5096999999999998e-05 2023-11-05T08:32:55.232 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.233 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:55.233 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.233 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:32:55.233 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:32:55.233 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:32:55.234 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-05T08:32:55.234 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-11-05T08:32:55.236 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2023-11-05T08:32:55.237 INFO:tasks.rados.rados.0.smithi090.stdout:109: done (5 left) 2023-11-05T08:32:55.237 INFO:tasks.rados.rados.0.smithi090.stdout:110: snap_create 2023-11-05T08:32:55.237 INFO:tasks.rados.rados.0.smithi090.stdout:101: finishing copy_from racing read to smithi09039252-40 2023-11-05T08:32:55.237 INFO:tasks.rados.rados.0.smithi090.stdout:103: expect (ObjNum 44 snap 0 seq_num 44) 2023-11-05T08:32:55.237 INFO:tasks.rados.rados.0.smithi090.stdout:105: finishing write tid 1 to smithi09039252-16 2023-11-05T08:32:55.238 INFO:tasks.rados.rados.0.smithi090.stdout:99: expect (ObjNum 54 snap 0 seq_num 54) 2023-11-05T08:32:55.238 INFO:tasks.rados.rados.0.smithi090.stdout:105: finishing write tid 2 to smithi09039252-16 2023-11-05T08:32:55.238 INFO:tasks.rados.rados.0.smithi090.stdout:105: finishing write tid 3 to smithi09039252-16 2023-11-05T08:32:55.238 INFO:tasks.rados.rados.0.smithi090.stdout:101: finishing copy_from to smithi09039252-40 2023-11-05T08:32:55.238 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 40 v 10 (ObjNum 26 snap 0 seq_num 26) dirty exists 2023-11-05T08:32:55.239 INFO:tasks.rados.rados.0.smithi090.stdout:108: expect (ObjNum 50 snap 0 seq_num 50) 2023-11-05T08:32:55.239 INFO:tasks.rados.rados.0.smithi090.stdout:105: finishing write tid 4 to smithi09039252-16 2023-11-05T08:32:55.239 INFO:tasks.rados.rados.0.smithi090.stdout:105: oid 16 updating version 0 to 14 2023-11-05T08:32:55.239 INFO:tasks.rados.rados.0.smithi090.stdout:105: oid 16 updating version 14 to 15 2023-11-05T08:32:55.239 INFO:tasks.rados.rados.0.smithi090.stdout:105: oid 16 version 15 is already newer than 13 2023-11-05T08:32:55.240 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 16 v 15 (ObjNum 68 snap 6 seq_num 68) dirty exists 2023-11-05T08:32:55.240 INFO:tasks.rados.rados.0.smithi090.stdout:105: left oid 16 (ObjNum 68 snap 6 seq_num 68) 2023-11-05T08:32:55.258 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-05T08:32:55.336 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:32:55.333+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:32:55.403 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:32:55.404 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-11-05T08:32:55.404 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-11-05T08:32:55.404 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:32:55.407 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-11-05T08:32:55.418 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2023-11-05T08:32:55.429 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1: {} 2023-11-05T08:32:55.429 INFO:teuthology.orchestra.run.smithi002.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-11-05T08:32:55.449 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2: {} 2023-11-05T08:32:55.450 INFO:teuthology.orchestra.run.smithi002.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-11-05T08:32:55.463 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-11-05T08:32:55.460+0000 7f966619d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35976) UID: 0 2023-11-05T08:32:55.484 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3: {} 2023-11-05T08:32:55.484 INFO:teuthology.orchestra.run.smithi002.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-11-05T08:32:55.517 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4: {} 2023-11-05T08:32:55.518 INFO:teuthology.orchestra.run.smithi002.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-11-05T08:32:55.537 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:32:55.532+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:32:55.538 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5: {} 2023-11-05T08:32:55.538 INFO:teuthology.orchestra.run.smithi002.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-11-05T08:32:55.540 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:32:55.540 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-11-05T08:32:55.540 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-11-05T08:32:55.540 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-11-05T08:32:55.541 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:32:55.555 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2023-11-05T08:32:55.558 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6: {} 2023-11-05T08:32:55.558 INFO:teuthology.orchestra.run.smithi002.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-11-05T08:32:55.577 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7: {} 2023-11-05T08:32:55.577 INFO:teuthology.orchestra.run.smithi002.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-11-05T08:32:55.594 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8: {} 2023-11-05T08:32:55.595 INFO:teuthology.orchestra.run.smithi002.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-11-05T08:32:55.615 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9: {} 2023-11-05T08:32:55.616 INFO:teuthology.orchestra.run.smithi002.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-11-05T08:32:55.636 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10: {} 2023-11-05T08:32:55.637 INFO:teuthology.orchestra.run.smithi002.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-11-05T08:32:55.643 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:32:55.640+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:32:55.656 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11: {} 2023-11-05T08:32:55.656 INFO:teuthology.orchestra.run.smithi002.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-11-05T08:32:55.676 INFO:teuthology.orchestra.run.smithi002.stdout:osd.12: {} 2023-11-05T08:32:55.676 INFO:teuthology.orchestra.run.smithi002.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-11-05T08:32:55.698 INFO:teuthology.orchestra.run.smithi002.stdout:osd.13: {} 2023-11-05T08:32:55.699 INFO:teuthology.orchestra.run.smithi002.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-11-05T08:32:55.720 INFO:teuthology.orchestra.run.smithi002.stdout:osd.14: {} 2023-11-05T08:32:55.721 INFO:teuthology.orchestra.run.smithi002.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-11-05T08:32:55.730 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:55.730 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:55.741 INFO:teuthology.orchestra.run.smithi002.stdout:osd.15: {} 2023-11-05T08:32:55.741 INFO:teuthology.orchestra.run.smithi002.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-11-05T08:32:55.742 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:32:55.741+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:32:55.744 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2023-11-05T08:32:55.746 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-05T08:32:55.747 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-05T08:32:55.839 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-05T08:32:55.911 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:55.912 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:55.926 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2023-11-05T08:32:55.941 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:32:55.939+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:32:56.042 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:56.042 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:56.043 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:32:56.041+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:32:56.057 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2023-11-05T08:32:56.133 INFO:teuthology.orchestra.run.smithi002.stderr:nodeep-scrub is set 2023-11-05T08:32:56.139 INFO:tasks.rados.rados.0.smithi090.stdout:99: done (5 left) 2023-11-05T08:32:56.140 INFO:tasks.rados.rados.0.smithi090.stdout:101: done (4 left) 2023-11-05T08:32:56.140 INFO:tasks.rados.rados.0.smithi090.stdout:103: done (3 left) 2023-11-05T08:32:56.140 INFO:tasks.rados.rados.0.smithi090.stdout:105: done (2 left) 2023-11-05T08:32:56.140 INFO:tasks.rados.rados.0.smithi090.stdout:108: done (1 left) 2023-11-05T08:32:56.140 INFO:tasks.rados.rados.0.smithi090.stdout:110: done (0 left) 2023-11-05T08:32:56.141 INFO:tasks.rados.rados.0.smithi090.stdout:111: read oid 13 snap -1 2023-11-05T08:32:56.141 INFO:tasks.rados.rados.0.smithi090.stdout:111: expect (ObjNum 50 snap 0 seq_num 50) 2023-11-05T08:32:56.141 INFO:tasks.rados.rados.0.smithi090.stdout:112: rollback oid 25 current snap is 7 2023-11-05T08:32:56.142 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 25 to 4 2023-11-05T08:32:56.142 INFO:tasks.rados.rados.0.smithi090.stdout:113: setattr oid 28 current snap is 7 2023-11-05T08:32:56.142 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-11-05T08:32:56.143 INFO:tasks.rados.rados.0.smithi090.stdout:114: snap_remove snap 5 2023-11-05T08:32:56.144 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:32:56.139+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:32:56.189 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:56.190 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:56.205 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-11-05T08:32:56.245 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:32:56.241+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:32:56.322 INFO:teuthology.orchestra.run.smithi002.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-05T08:32:56.325 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-05T08:32:56.326 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-11-05T08:32:56.371 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:32:56.371+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:32:56.439 INFO:teuthology.orchestra.run.smithi002.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-05T08:32:56.442 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-05T08:32:56.443 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-11-05T08:32:56.445 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:32:56.441+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:32:56.546 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:32:56.543+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:32:56.597 INFO:teuthology.orchestra.run.smithi002.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-05T08:32:56.600 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-05T08:32:56.600 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-11-05T08:32:56.647 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-11-05T08:32:56.644+0000 7f966619d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35976) UID: 0 2023-11-05T08:32:56.719 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:56.720 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:56.734 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-11-05T08:32:56.779 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:32:56.779+0000 7fdc5cd58640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:32:56.848 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:32:56.847+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:32:56.853 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:56.854 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:56.866 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-11-05T08:32:56.949 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:32:56.947+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:32:57.034 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:57.034 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:57.048 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2023-11-05T08:32:57.049 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-05T08:32:57.112 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2023-11-05T08:32:57.112 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-11-05T08:32:57.112 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2023-11-05T08:32:57.113 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-11-05T08:32:57.113 INFO:teuthology.orchestra.run:waiting for 300 2023-11-05T08:32:57.146 INFO:tasks.rados.rados.0.smithi090.stdout:114: done (3 left) 2023-11-05T08:32:57.146 INFO:tasks.rados.rados.0.smithi090.stdout:115: rollback oid 29 current snap is 7 2023-11-05T08:32:57.146 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 29 to 2 2023-11-05T08:32:57.147 INFO:tasks.rados.rados.0.smithi090.stdout:116: snap_remove snap 6 2023-11-05T08:32:57.157 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:32:57.148+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:32:57.194 INFO:tasks.ceph.osd.3:Stopped 2023-11-05T08:32:57.194 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.3 2023-11-05T08:32:57.195 INFO:tasks.thrashosds.thrasher:remote for osd 3 is ubuntu@smithi090.front.sepia.ceph.com 2023-11-05T08:32:57.195 DEBUG:teuthology.orchestra.run.smithi090:> type ceph-objectstore-tool 2023-11-05T08:32:57.196 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:57.196 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:57.215 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2023-11-05T08:32:57.238 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2023-11-05T08:32:57.239 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op list-pgs 2023-11-05T08:32:57.263 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:32:57.263+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:32:57.345 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:57.346 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:57.353 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:32:57.349+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:32:57.360 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2023-11-05T08:32:57.453 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:32:57.451+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:32:57.530 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:57.530 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:57.544 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2023-11-05T08:32:57.595 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:32:57.595+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:32:57.655 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:32:57.655+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:32:57.671 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:57.671 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:57.685 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2023-11-05T08:32:57.748 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-05T08:32:57.756 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:32:57.752+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:32:57.856 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:57.856 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:57.857 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:32:57.856+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:32:57.871 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2023-11-05T08:32:57.906 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-11-05T08:32:57.928 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1: {} 2023-11-05T08:32:57.929 INFO:teuthology.orchestra.run.smithi002.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-11-05T08:32:57.949 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2: {} 2023-11-05T08:32:57.949 INFO:teuthology.orchestra.run.smithi002.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-11-05T08:32:57.958 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:32:57.955+0000 7fdc5cd58640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:32:57.989 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:32:57.989 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:32:58.004 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-11-05T08:32:58.058 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:32:58.059+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:32:58.138 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:32:58.138 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2023-11-05T08:32:58.139 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 0 2023-11-05T08:32:58.139 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:32:58.154 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-11-05T08:32:58.165 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-11-05T08:32:58.177 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-11-05T08:32:58.190 INFO:tasks.rados.rados.0.smithi090.stdout:116: done (4 left) 2023-11-05T08:32:58.191 INFO:tasks.rados.rados.0.smithi090.stdout:117: read oid 1 snap 4 2023-11-05T08:32:58.191 INFO:tasks.rados.rados.0.smithi090.stdout:117: expect (ObjNum 0 snap 0 seq_num 0) 2023-11-05T08:32:58.192 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 30 current snap is 7 2023-11-05T08:32:58.192 INFO:tasks.rados.rados.0.smithi090.stdout:118: seq_num 70 ranges {491520=802816,1294336=376832} 2023-11-05T08:32:58.197 INFO:tasks.rados.rados.0.smithi090.stdout:118: writing smithi09039252-30 from 491520 to 1294336 tid 1 2023-11-05T08:32:58.198 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:32:58.197+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:32:58.201 INFO:tasks.rados.rados.0.smithi090.stdout:118: writing smithi09039252-30 from 1294336 to 1671168 tid 2 2023-11-05T08:32:58.201 INFO:tasks.rados.rados.0.smithi090.stdout:112: finishing rollback tid 0 to smithi09039252-25 2023-11-05T08:32:58.201 INFO:tasks.rados.rados.0.smithi090.stdout:112: finishing rollback tid 1 to smithi09039252-25 2023-11-05T08:32:58.201 INFO:tasks.rados.rados.0.smithi090.stdout:112: finishing rollback tid 2 to smithi09039252-25 2023-11-05T08:32:58.201 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 25 v 33 (ObjNum 52 snap 0 seq_num 52) dirty exists 2023-11-05T08:32:58.202 INFO:tasks.rados.rados.0.smithi090.stdout:111: expect (ObjNum 50 snap 0 seq_num 50) 2023-11-05T08:32:58.210 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4: {} 2023-11-05T08:32:58.210 INFO:teuthology.orchestra.run.smithi002.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-11-05T08:32:58.230 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5: {} 2023-11-05T08:32:58.230 INFO:teuthology.orchestra.run.smithi002.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-11-05T08:32:58.238 INFO:tasks.rados.rados.0.smithi090.stdout:115: finishing rollback tid 0 to smithi09039252-29 2023-11-05T08:32:58.238 INFO:tasks.rados.rados.0.smithi090.stdout:115: finishing rollback tid 1 to smithi09039252-29 2023-11-05T08:32:58.238 INFO:tasks.rados.rados.0.smithi090.stdout:115: finishing rollback tid 2 to smithi09039252-29 2023-11-05T08:32:58.238 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 29 v 16 (ObjNum 28 snap 0 seq_num 28) dirty exists 2023-11-05T08:32:58.238 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 28 v 17 (ObjNum 27 snap 0 seq_num 27) dirty exists 2023-11-05T08:32:58.239 INFO:tasks.rados.rados.0.smithi090.stdout:118: finishing write tid 1 to smithi09039252-30 2023-11-05T08:32:58.239 INFO:tasks.rados.rados.0.smithi090.stdout:118: finishing write tid 2 to smithi09039252-30 2023-11-05T08:32:58.239 INFO:tasks.rados.rados.0.smithi090.stdout:118: finishing write tid 3 to smithi09039252-30 2023-11-05T08:32:58.239 INFO:tasks.rados.rados.0.smithi090.stdout:118: finishing write tid 4 to smithi09039252-30 2023-11-05T08:32:58.239 INFO:tasks.rados.rados.0.smithi090.stdout:118: oid 30 updating version 0 to 24 2023-11-05T08:32:58.240 INFO:tasks.rados.rados.0.smithi090.stdout:118: oid 30 updating version 24 to 25 2023-11-05T08:32:58.240 INFO:tasks.rados.rados.0.smithi090.stdout:118: oid 30 version 25 is already newer than 23 2023-11-05T08:32:58.240 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 30 v 25 (ObjNum 70 snap 7 seq_num 70) dirty exists 2023-11-05T08:32:58.240 INFO:tasks.rados.rados.0.smithi090.stdout:118: left oid 30 (ObjNum 70 snap 7 seq_num 70) 2023-11-05T08:32:58.240 INFO:tasks.rados.rados.0.smithi090.stdout:111: done (5 left) 2023-11-05T08:32:58.241 INFO:tasks.rados.rados.0.smithi090.stdout:112: done (4 left) 2023-11-05T08:32:58.241 INFO:tasks.rados.rados.0.smithi090.stdout:113: done (3 left) 2023-11-05T08:32:58.241 INFO:tasks.rados.rados.0.smithi090.stdout:115: done (2 left) 2023-11-05T08:32:58.241 INFO:tasks.rados.rados.0.smithi090.stdout:118: done (1 left) 2023-11-05T08:32:58.241 INFO:tasks.rados.rados.0.smithi090.stdout:119: delete oid 17 current snap is 7 2023-11-05T08:32:58.250 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6: {} 2023-11-05T08:32:58.250 INFO:teuthology.orchestra.run.smithi002.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-11-05T08:32:58.260 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:32:58.259+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:32:58.272 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7: {} 2023-11-05T08:32:58.273 INFO:teuthology.orchestra.run.smithi002.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-11-05T08:32:58.287 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:32:58.287 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2023-11-05T08:32:58.288 INFO:teuthology.orchestra.run.smithi053.stdout: "complaint_time": 30, 2023-11-05T08:32:58.288 INFO:teuthology.orchestra.run.smithi053.stdout: "num_blocked_ops": 0 2023-11-05T08:32:58.288 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:32:58.294 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8: {} 2023-11-05T08:32:58.294 INFO:teuthology.orchestra.run.smithi002.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-11-05T08:32:58.301 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-11-05T08:32:58.315 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9: {} 2023-11-05T08:32:58.316 INFO:teuthology.orchestra.run.smithi002.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-11-05T08:32:58.336 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10: {} 2023-11-05T08:32:58.336 INFO:teuthology.orchestra.run.smithi002.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-11-05T08:32:58.355 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11: {} 2023-11-05T08:32:58.355 INFO:teuthology.orchestra.run.smithi002.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-11-05T08:32:58.361 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:32:58.359+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:32:58.374 INFO:teuthology.orchestra.run.smithi002.stdout:osd.12: {} 2023-11-05T08:32:58.374 INFO:teuthology.orchestra.run.smithi002.stderr:osd.12: osd_enable_op_tracker = 'true' 2023-11-05T08:32:58.392 INFO:teuthology.orchestra.run.smithi002.stdout:osd.13: {} 2023-11-05T08:32:58.392 INFO:teuthology.orchestra.run.smithi002.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-11-05T08:32:58.413 INFO:teuthology.orchestra.run.smithi002.stdout:osd.14: {} 2023-11-05T08:32:58.413 INFO:teuthology.orchestra.run.smithi002.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-11-05T08:32:58.432 INFO:teuthology.orchestra.run.smithi002.stdout:osd.15: {} 2023-11-05T08:32:58.433 INFO:teuthology.orchestra.run.smithi002.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-11-05T08:32:58.438 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-05T08:32:58.439 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-05T08:32:58.462 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:32:58.456+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:32:58.467 INFO:teuthology.orchestra.run.smithi090.stdout:3.ds1 2023-11-05T08:32:58.467 INFO:teuthology.orchestra.run.smithi090.stdout:3.as0 2023-11-05T08:32:58.468 INFO:teuthology.orchestra.run.smithi090.stdout:3.9s3 2023-11-05T08:32:58.468 INFO:teuthology.orchestra.run.smithi090.stdout:3.3s6 2023-11-05T08:32:58.468 INFO:teuthology.orchestra.run.smithi090.stdout:3.19s3 2023-11-05T08:32:58.468 INFO:teuthology.orchestra.run.smithi090.stdout:3.13s6 2023-11-05T08:32:58.468 INFO:teuthology.orchestra.run.smithi090.stdout:3.14s0 2023-11-05T08:32:58.469 INFO:teuthology.orchestra.run.smithi090.stdout:3.12s2 2023-11-05T08:32:58.469 INFO:teuthology.orchestra.run.smithi090.stdout:3.2s2 2023-11-05T08:32:58.469 INFO:teuthology.orchestra.run.smithi090.stdout:2.2 2023-11-05T08:32:58.469 INFO:teuthology.orchestra.run.smithi090.stdout:2.0 2023-11-05T08:32:58.471 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:32:58.471 INFO:teuthology.orchestra.run.smithi053.stdout: "size": 20, 2023-11-05T08:32:58.471 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 600, 2023-11-05T08:32:58.472 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2023-11-05T08:32:58.472 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.472 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.6s2 35/30 ECSubWrite(tid=8, reqid=client.4691.0:225, at_version=35'7, trim_to=0'0, roll_forward_to=34'5))", 2023-11-05T08:32:58.472 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:48.092578+0000", 2023-11-05T08:32:58.472 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 10.375384793, 2023-11-05T08:32:58.473 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0056698920000000002, 2023-11-05T08:32:58.473 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:58.473 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:58.473 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:58.473 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.474 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:58.474 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.092578+0000", 2023-11-05T08:32:58.474 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.474 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.474 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.475 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:58.475 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.092578+0000", 2023-11-05T08:32:58.475 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.475 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.475 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.476 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:58.476 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.092579+0000", 2023-11-05T08:32:58.476 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.5000000000000002e-07 2023-11-05T08:32:58.476 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.476 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.476 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:58.477 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.092639+0000", 2023-11-05T08:32:58.477 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0288000000000003e-05 2023-11-05T08:32:58.477 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.477 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.477 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:58.477 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.092675+0000", 2023-11-05T08:32:58.478 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.6087999999999997e-05 2023-11-05T08:32:58.478 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.478 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.478 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:58.478 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.092700+0000", 2023-11-05T08:32:58.478 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5205999999999999e-05 2023-11-05T08:32:58.479 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.479 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.479 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:58.479 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.092915+0000", 2023-11-05T08:32:58.479 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00021489600000000001 2023-11-05T08:32:58.480 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.480 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.480 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:58.480 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.092932+0000", 2023-11-05T08:32:58.480 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6362999999999999e-05 2023-11-05T08:32:58.481 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.481 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.481 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:58.481 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.098160+0000", 2023-11-05T08:32:58.481 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0052284539999999996 2023-11-05T08:32:58.481 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.482 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.482 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:58.482 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.098248+0000", 2023-11-05T08:32:58.482 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.7847000000000001e-05 2023-11-05T08:32:58.482 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.483 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:58.483 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.483 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.483 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.483 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.6s2 35/30 ECSubWrite(tid=9, reqid=client.4691.0:226, at_version=35'8, trim_to=0'0, roll_forward_to=34'5))", 2023-11-05T08:32:58.484 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:48.093891+0000", 2023-11-05T08:32:58.484 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 10.374071614, 2023-11-05T08:32:58.484 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0051649729999999998, 2023-11-05T08:32:58.484 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:58.484 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:58.485 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:58.485 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.485 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:58.485 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.093891+0000", 2023-11-05T08:32:58.485 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.485 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.486 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.486 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:58.486 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.093891+0000", 2023-11-05T08:32:58.486 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.486 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.487 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.487 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:58.489 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.093893+0000", 2023-11-05T08:32:58.490 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5859999999999999e-06 2023-11-05T08:32:58.490 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.490 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.490 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:58.490 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.093943+0000", 2023-11-05T08:32:58.491 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.0028999999999998e-05 2023-11-05T08:32:58.491 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.491 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.491 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:58.491 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.093976+0000", 2023-11-05T08:32:58.492 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2869e-05 2023-11-05T08:32:58.492 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.492 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.492 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:58.492 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.094001+0000", 2023-11-05T08:32:58.493 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5678999999999999e-05 2023-11-05T08:32:58.493 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.493 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.493 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:58.493 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.095043+0000", 2023-11-05T08:32:58.494 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001041978 2023-11-05T08:32:58.494 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.494 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.494 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:58.494 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.095067+0000", 2023-11-05T08:32:58.495 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4076000000000001e-05 2023-11-05T08:32:58.495 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.495 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.495 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:58.495 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.098980+0000", 2023-11-05T08:32:58.496 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0039126609999999996 2023-11-05T08:32:58.496 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.496 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.496 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:58.496 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.099056+0000", 2023-11-05T08:32:58.497 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.6094999999999996e-05 2023-11-05T08:32:58.497 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.497 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:58.497 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.497 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.498 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.498 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.6s2 35/30 ECSubWrite(tid=10, reqid=client.4691.0:227, at_version=35'9, trim_to=0'0, roll_forward_to=34'5))", 2023-11-05T08:32:58.498 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:48.094938+0000", 2023-11-05T08:32:58.498 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 10.373024327, 2023-11-05T08:32:58.498 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0052729760000000004, 2023-11-05T08:32:58.499 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:58.499 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:58.499 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:58.499 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.499 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:58.499 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.094938+0000", 2023-11-05T08:32:58.500 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.500 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.500 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.500 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:58.501 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.094938+0000", 2023-11-05T08:32:58.501 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.501 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.501 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.501 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:58.502 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.094940+0000", 2023-11-05T08:32:58.502 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.68e-06 2023-11-05T08:32:58.502 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.502 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.502 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:58.503 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.094984+0000", 2023-11-05T08:32:58.503 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3458999999999998e-05 2023-11-05T08:32:58.503 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.503 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.503 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:58.504 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.095047+0000", 2023-11-05T08:32:58.504 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3324999999999994e-05 2023-11-05T08:32:58.504 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.504 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.504 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:58.505 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.095072+0000", 2023-11-05T08:32:58.505 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4919e-05 2023-11-05T08:32:58.505 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.505 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.505 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:58.505 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.097031+0000", 2023-11-05T08:32:58.506 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0019587739999999999 2023-11-05T08:32:58.506 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.506 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.506 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:58.506 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.097057+0000", 2023-11-05T08:32:58.506 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6142000000000001e-05 2023-11-05T08:32:58.507 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.507 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.507 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:58.507 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.100141+0000", 2023-11-05T08:32:58.507 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0030844599999999998 2023-11-05T08:32:58.508 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.508 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.508 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:58.508 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.100211+0000", 2023-11-05T08:32:58.508 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.0216999999999995e-05 2023-11-05T08:32:58.509 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.509 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:58.509 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.509 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.509 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.509 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.8s3 36/30 ECSubWrite(tid=23, reqid=osd.5.0:23, at_version=36'15, trim_to=0'0, roll_forward_to=34'13))", 2023-11-05T08:32:58.510 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:49.078476+0000", 2023-11-05T08:32:58.510 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 9.3894872809999992, 2023-11-05T08:32:58.510 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0057340960000000002, 2023-11-05T08:32:58.510 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:58.511 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:58.511 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:58.511 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.511 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:58.511 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.078476+0000", 2023-11-05T08:32:58.512 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.512 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.512 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.512 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:58.512 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.078476+0000", 2023-11-05T08:32:58.513 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.513 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.513 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.513 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:58.513 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.078476+0000", 2023-11-05T08:32:58.514 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1299999999999998e-07 2023-11-05T08:32:58.514 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.514 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.514 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:58.514 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.078542+0000", 2023-11-05T08:32:58.515 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5121000000000002e-05 2023-11-05T08:32:58.515 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.515 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.515 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:58.515 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.078572+0000", 2023-11-05T08:32:58.516 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0812000000000002e-05 2023-11-05T08:32:58.516 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.516 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.516 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:58.516 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.078599+0000", 2023-11-05T08:32:58.517 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6653e-05 2023-11-05T08:32:58.517 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.517 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.517 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:58.517 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.078773+0000", 2023-11-05T08:32:58.518 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00017369099999999999 2023-11-05T08:32:58.518 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.518 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.518 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:58.518 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.078798+0000", 2023-11-05T08:32:58.519 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5153000000000001e-05 2023-11-05T08:32:58.519 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.519 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.519 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:58.519 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.084110+0000", 2023-11-05T08:32:58.520 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0053123190000000002 2023-11-05T08:32:58.520 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.520 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.520 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:58.520 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.084210+0000", 2023-11-05T08:32:58.521 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.9433999999999997e-05 2023-11-05T08:32:58.521 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.521 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:58.521 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.521 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.522 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.522 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as2 40/33 ECSubWrite(tid=30, reqid=client.4691.0:283, at_version=40'8, trim_to=0'0, roll_forward_to=39'6))", 2023-11-05T08:32:58.522 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:53.167536+0000", 2023-11-05T08:32:58.522 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 5.300426356, 2023-11-05T08:32:58.522 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0046219049999999999, 2023-11-05T08:32:58.523 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:58.523 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:58.523 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:58.523 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.524 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:58.524 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.167536+0000", 2023-11-05T08:32:58.524 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.524 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.524 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.525 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:58.525 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.167536+0000", 2023-11-05T08:32:58.525 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.525 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.525 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.526 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:58.526 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.167537+0000", 2023-11-05T08:32:58.526 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1399999999999998e-07 2023-11-05T08:32:58.526 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.526 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.527 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:58.527 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.168135+0000", 2023-11-05T08:32:58.527 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00059850699999999997 2023-11-05T08:32:58.527 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.527 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.528 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:58.528 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.168166+0000", 2023-11-05T08:32:58.528 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0518999999999997e-05 2023-11-05T08:32:58.528 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.528 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.529 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:58.529 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.168181+0000", 2023-11-05T08:32:58.529 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4800000000000001e-05 2023-11-05T08:32:58.529 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.529 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.530 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:58.530 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.168479+0000", 2023-11-05T08:32:58.530 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00029859100000000001 2023-11-05T08:32:58.530 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.530 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.531 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:58.531 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.168514+0000", 2023-11-05T08:32:58.531 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4912e-05 2023-11-05T08:32:58.531 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.531 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.532 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:58.532 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.172074+0000", 2023-11-05T08:32:58.532 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0035596239999999999 2023-11-05T08:32:58.532 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.533 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.533 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:58.533 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.172158+0000", 2023-11-05T08:32:58.533 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.4637999999999998e-05 2023-11-05T08:32:58.533 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.533 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:58.534 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.534 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.534 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.534 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es2 41/30 ECSubWrite(tid=40, reqid=osd.2.0:40, at_version=41'27, trim_to=0'0, roll_forward_to=40'25))", 2023-11-05T08:32:58.534 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.127197+0000", 2023-11-05T08:32:58.535 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4.3407655759999999, 2023-11-05T08:32:58.535 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.019700031, 2023-11-05T08:32:58.535 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:58.535 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:58.536 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:58.536 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.536 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:58.536 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127197+0000", 2023-11-05T08:32:58.536 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.537 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.537 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.537 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:58.537 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127197+0000", 2023-11-05T08:32:58.537 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.538 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.538 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.538 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:58.538 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127198+0000", 2023-11-05T08:32:58.538 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.0999999999999998e-07 2023-11-05T08:32:58.539 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.539 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.539 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:58.539 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127255+0000", 2023-11-05T08:32:58.539 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6762000000000003e-05 2023-11-05T08:32:58.540 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.540 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.540 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:58.540 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127291+0000", 2023-11-05T08:32:58.540 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5948999999999998e-05 2023-11-05T08:32:58.541 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.541 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.541 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:58.541 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127314+0000", 2023-11-05T08:32:58.541 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3686e-05 2023-11-05T08:32:58.542 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.542 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.542 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:58.542 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127534+0000", 2023-11-05T08:32:58.542 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00021922700000000001 2023-11-05T08:32:58.543 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.543 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.543 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:58.543 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127562+0000", 2023-11-05T08:32:58.543 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8099e-05 2023-11-05T08:32:58.544 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.544 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.544 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:58.544 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.146742+0000", 2023-11-05T08:32:58.544 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.019180135000000001 2023-11-05T08:32:58.544 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.545 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.545 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:58.545 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.146897+0000", 2023-11-05T08:32:58.545 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000155463 2023-11-05T08:32:58.546 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.546 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:58.546 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.546 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.546 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.547 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as2 41/33 ECSubWrite(tid=35, reqid=osd.7.0:35, at_version=41'12, trim_to=0'0, roll_forward_to=40'10))", 2023-11-05T08:32:58.547 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.130596+0000", 2023-11-05T08:32:58.547 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4.3373663530000002, 2023-11-05T08:32:58.547 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022182642999999998, 2023-11-05T08:32:58.547 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:58.548 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:58.548 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:58.548 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.548 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:58.548 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130596+0000", 2023-11-05T08:32:58.549 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.549 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.549 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.549 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:58.549 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130596+0000", 2023-11-05T08:32:58.550 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.550 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.550 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.550 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:58.550 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130597+0000", 2023-11-05T08:32:58.551 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.7600000000000006e-07 2023-11-05T08:32:58.551 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.551 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.551 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:58.551 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130660+0000", 2023-11-05T08:32:58.551 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3015000000000006e-05 2023-11-05T08:32:58.552 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.552 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.552 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:58.552 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130725+0000", 2023-11-05T08:32:58.553 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4988999999999997e-05 2023-11-05T08:32:58.553 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.553 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.553 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:58.553 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130750+0000", 2023-11-05T08:32:58.554 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4802000000000001e-05 2023-11-05T08:32:58.554 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.554 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.554 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:58.554 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.131037+0000", 2023-11-05T08:32:58.554 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00028702900000000001 2023-11-05T08:32:58.555 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.555 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.555 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:58.555 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.131064+0000", 2023-11-05T08:32:58.556 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7022e-05 2023-11-05T08:32:58.556 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.556 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.556 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:58.556 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.152706+0000", 2023-11-05T08:32:58.556 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.021641734999999999 2023-11-05T08:32:58.557 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.557 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.557 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:58.557 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.152779+0000", 2023-11-05T08:32:58.557 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.3075000000000001e-05 2023-11-05T08:32:58.558 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.558 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:58.558 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.558 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.558 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.559 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as2 41/33 ECSubWrite(tid=36, reqid=osd.7.0:36, at_version=41'14, trim_to=0'0, roll_forward_to=40'10))", 2023-11-05T08:32:58.559 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.130806+0000", 2023-11-05T08:32:58.559 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4.3371564730000003, 2023-11-05T08:32:58.559 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.035467152000000002, 2023-11-05T08:32:58.559 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:58.559 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:58.560 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:58.560 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.560 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:58.560 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130806+0000", 2023-11-05T08:32:58.560 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.561 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.561 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.561 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:58.561 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130806+0000", 2023-11-05T08:32:58.561 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.562 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.562 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.562 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:58.562 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130807+0000", 2023-11-05T08:32:58.562 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1000000000000001e-06 2023-11-05T08:32:58.563 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.563 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.563 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:58.563 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130852+0000", 2023-11-05T08:32:58.563 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4428999999999998e-05 2023-11-05T08:32:58.564 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.564 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.564 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:58.564 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130876+0000", 2023-11-05T08:32:58.564 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3975999999999999e-05 2023-11-05T08:32:58.565 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.565 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.565 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:58.565 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130896+0000", 2023-11-05T08:32:58.565 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9905999999999999e-05 2023-11-05T08:32:58.566 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.566 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.566 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:58.566 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.143384+0000", 2023-11-05T08:32:58.566 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012488727999999999 2023-11-05T08:32:58.567 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.567 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.567 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:58.567 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.143412+0000", 2023-11-05T08:32:58.567 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7988999999999999e-05 2023-11-05T08:32:58.568 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.568 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.568 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:58.568 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.166186+0000", 2023-11-05T08:32:58.568 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022773787 2023-11-05T08:32:58.568 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.569 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.569 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:58.569 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.166273+0000", 2023-11-05T08:32:58.569 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.7237000000000005e-05 2023-11-05T08:32:58.569 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.569 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:58.570 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.570 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.570 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.570 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es2 41/30 ECSubWrite(tid=41, reqid=osd.2.0:40, at_version=0'0, trim_to=0'0, roll_forward_to=41'27))", 2023-11-05T08:32:58.570 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.148654+0000", 2023-11-05T08:32:58.571 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4.3193088289999997, 2023-11-05T08:32:58.571 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017394370999999999, 2023-11-05T08:32:58.571 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:58.571 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:58.571 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:58.572 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.572 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:58.572 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148654+0000", 2023-11-05T08:32:58.572 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.572 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.572 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.573 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:58.573 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148654+0000", 2023-11-05T08:32:58.573 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.573 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.573 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.573 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:58.574 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148656+0000", 2023-11-05T08:32:58.574 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.677e-06 2023-11-05T08:32:58.574 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.574 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.574 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:58.575 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148683+0000", 2023-11-05T08:32:58.575 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7246e-05 2023-11-05T08:32:58.575 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.575 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.575 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:58.576 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148715+0000", 2023-11-05T08:32:58.576 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2518999999999998e-05 2023-11-05T08:32:58.576 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.576 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.576 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:58.577 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148742+0000", 2023-11-05T08:32:58.577 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6876000000000002e-05 2023-11-05T08:32:58.577 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.577 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.577 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:58.578 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148890+0000", 2023-11-05T08:32:58.578 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014771599999999999 2023-11-05T08:32:58.578 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.578 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.579 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:58.579 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148915+0000", 2023-11-05T08:32:58.579 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5099000000000001e-05 2023-11-05T08:32:58.579 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.579 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.580 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:58.580 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.165914+0000", 2023-11-05T08:32:58.580 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.016999018000000001 2023-11-05T08:32:58.580 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.580 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.581 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:58.581 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.166048+0000", 2023-11-05T08:32:58.581 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013422 2023-11-05T08:32:58.581 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.582 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:58.582 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.582 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.582 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.582 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.2s5 41/40 ECSubWrite(tid=187, reqid=client.4691.0:292, at_version=41'9, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:32:58.583 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.162642+0000", 2023-11-05T08:32:58.583 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4.3053204300000001, 2023-11-05T08:32:58.583 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010859549, 2023-11-05T08:32:58.583 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:58.583 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:58.584 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:58.584 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.584 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:58.584 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.162642+0000", 2023-11-05T08:32:58.584 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.584 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.585 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.585 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:58.585 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.162642+0000", 2023-11-05T08:32:58.585 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.585 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.586 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.586 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:58.586 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.162644+0000", 2023-11-05T08:32:58.586 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.823e-06 2023-11-05T08:32:58.586 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.586 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.587 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:58.587 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.162700+0000", 2023-11-05T08:32:58.587 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5863999999999999e-05 2023-11-05T08:32:58.587 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.587 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.588 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:58.588 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.162731+0000", 2023-11-05T08:32:58.588 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0731000000000001e-05 2023-11-05T08:32:58.588 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.589 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.589 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:58.589 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.162754+0000", 2023-11-05T08:32:58.589 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3234000000000002e-05 2023-11-05T08:32:58.589 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.590 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.590 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:58.590 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.162988+0000", 2023-11-05T08:32:58.590 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00023393399999999999 2023-11-05T08:32:58.590 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.590 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.591 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:58.591 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.163016+0000", 2023-11-05T08:32:58.591 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8309e-05 2023-11-05T08:32:58.591 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.591 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.591 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:58.592 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.173420+0000", 2023-11-05T08:32:58.592 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010403802 2023-11-05T08:32:58.592 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.592 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.592 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:58.593 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.173502+0000", 2023-11-05T08:32:58.593 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.1852000000000002e-05 2023-11-05T08:32:58.593 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.593 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:58.593 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.593 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.594 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.594 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as2 41/33 ECSubWrite(tid=37, reqid=osd.7.0:36, at_version=0'0, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:32:58.594 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.167511+0000", 2023-11-05T08:32:58.594 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4.3004519920000002, 2023-11-05T08:32:58.594 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011732691, 2023-11-05T08:32:58.595 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:58.595 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:58.595 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:58.595 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.595 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:58.596 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167511+0000", 2023-11-05T08:32:58.596 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.596 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.596 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.596 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:58.597 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167511+0000", 2023-11-05T08:32:58.597 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.597 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.597 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.597 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:58.598 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167513+0000", 2023-11-05T08:32:58.598 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.0829999999999998e-06 2023-11-05T08:32:58.598 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.598 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.598 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:58.599 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167540+0000", 2023-11-05T08:32:58.599 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7095999999999999e-05 2023-11-05T08:32:58.599 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.599 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.599 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:58.599 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167572+0000", 2023-11-05T08:32:58.600 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2462000000000002e-05 2023-11-05T08:32:58.600 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.600 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.600 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:58.600 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167594+0000", 2023-11-05T08:32:58.600 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1316999999999999e-05 2023-11-05T08:32:58.601 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.601 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.601 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:58.601 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167791+0000", 2023-11-05T08:32:58.601 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00019748199999999999 2023-11-05T08:32:58.602 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.602 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.602 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:58.602 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167815+0000", 2023-11-05T08:32:58.602 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3536999999999998e-05 2023-11-05T08:32:58.603 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.603 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.603 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:58.603 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.179154+0000", 2023-11-05T08:32:58.603 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011339373999999999 2023-11-05T08:32:58.603 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.604 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.604 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:58.604 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.179244+0000", 2023-11-05T08:32:58.604 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9339999999999995e-05 2023-11-05T08:32:58.604 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.605 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:58.605 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.605 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.605 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.605 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.2s5 41/40 ECSubWrite(tid=188, reqid=osd.13.0:188, at_version=41'11, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:32:58.606 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.171651+0000", 2023-11-05T08:32:58.606 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4.296311405, 2023-11-05T08:32:58.606 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0076644390000000003, 2023-11-05T08:32:58.606 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:58.606 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:58.607 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:58.607 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.607 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:58.607 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171651+0000", 2023-11-05T08:32:58.608 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.608 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.608 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.608 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:58.608 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171651+0000", 2023-11-05T08:32:58.608 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.609 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.609 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.609 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:58.609 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171653+0000", 2023-11-05T08:32:58.609 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5039999999999999e-06 2023-11-05T08:32:58.610 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.610 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.610 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:58.610 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171700+0000", 2023-11-05T08:32:58.610 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.6989e-05 2023-11-05T08:32:58.611 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.611 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.611 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:58.611 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171731+0000", 2023-11-05T08:32:58.611 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1310999999999999e-05 2023-11-05T08:32:58.612 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.612 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.612 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:58.612 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171750+0000", 2023-11-05T08:32:58.612 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9188e-05 2023-11-05T08:32:58.612 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.613 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.613 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:58.613 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171900+0000", 2023-11-05T08:32:58.613 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00015010699999999999 2023-11-05T08:32:58.614 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.614 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.614 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:58.614 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171920+0000", 2023-11-05T08:32:58.614 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9626000000000001e-05 2023-11-05T08:32:58.615 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.615 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.615 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:58.615 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.179250+0000", 2023-11-05T08:32:58.615 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0073298900000000004 2023-11-05T08:32:58.615 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.616 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.616 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:58.616 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.179316+0000", 2023-11-05T08:32:58.616 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5823999999999999e-05 2023-11-05T08:32:58.617 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.617 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:58.617 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.617 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.617 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.617 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.2s5 41/40 ECSubWrite(tid=189, reqid=osd.13.0:188, at_version=0'0, trim_to=0'0, roll_forward_to=41'11))", 2023-11-05T08:32:58.618 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.183891+0000", 2023-11-05T08:32:58.618 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4.2840714039999996, 2023-11-05T08:32:58.618 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0045627280000000003, 2023-11-05T08:32:58.618 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:58.618 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:58.619 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:58.619 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.619 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:58.619 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.183891+0000", 2023-11-05T08:32:58.619 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.620 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.620 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.620 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:58.620 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.183891+0000", 2023-11-05T08:32:58.620 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.621 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.621 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.621 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:58.621 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.183892+0000", 2023-11-05T08:32:58.622 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7400000000000003e-07 2023-11-05T08:32:58.622 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.622 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.622 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:58.622 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.183909+0000", 2023-11-05T08:32:58.623 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6980999999999999e-05 2023-11-05T08:32:58.623 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.623 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.623 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:58.624 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.183926+0000", 2023-11-05T08:32:58.624 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7263000000000001e-05 2023-11-05T08:32:58.624 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.624 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.624 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:58.625 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.183942+0000", 2023-11-05T08:32:58.625 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5670000000000001e-05 2023-11-05T08:32:58.625 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.625 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.625 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:58.626 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.184091+0000", 2023-11-05T08:32:58.626 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014870399999999999 2023-11-05T08:32:58.626 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.626 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.626 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:58.627 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.184110+0000", 2023-11-05T08:32:58.627 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.933e-05 2023-11-05T08:32:58.627 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.627 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.627 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:58.628 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.188379+0000", 2023-11-05T08:32:58.628 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0042686269999999997 2023-11-05T08:32:58.628 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.628 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.628 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:58.629 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.188454+0000", 2023-11-05T08:32:58.629 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.5579000000000003e-05 2023-11-05T08:32:58.629 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.629 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:58.629 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.629 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.630 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.630 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.14s3 41/40 ECSubWrite(tid=17, reqid=client.4691.0:278, at_version=41'14, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:32:58.630 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.184891+0000", 2023-11-05T08:32:58.630 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4.2830721150000004, 2023-11-05T08:32:58.630 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0062390429999999997, 2023-11-05T08:32:58.631 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:58.631 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:58.631 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:58.631 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.631 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:58.632 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.184891+0000", 2023-11-05T08:32:58.632 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.632 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.632 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.632 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:58.632 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.184891+0000", 2023-11-05T08:32:58.633 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.633 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.633 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.633 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:58.633 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.184892+0000", 2023-11-05T08:32:58.633 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.497e-06 2023-11-05T08:32:58.634 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.634 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.634 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:58.634 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.185224+0000", 2023-11-05T08:32:58.634 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00033133899999999999 2023-11-05T08:32:58.635 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.635 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.635 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:58.635 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.185245+0000", 2023-11-05T08:32:58.635 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1185000000000001e-05 2023-11-05T08:32:58.636 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.636 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.636 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:58.636 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.185261+0000", 2023-11-05T08:32:58.636 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6067999999999999e-05 2023-11-05T08:32:58.636 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.637 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.637 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:58.637 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.185463+0000", 2023-11-05T08:32:58.637 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00020208800000000001 2023-11-05T08:32:58.637 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.638 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.638 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:58.638 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.185483+0000", 2023-11-05T08:32:58.638 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.0092e-05 2023-11-05T08:32:58.638 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.639 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.639 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:58.639 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.191053+0000", 2023-11-05T08:32:58.639 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0055700919999999996 2023-11-05T08:32:58.639 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.640 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.640 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:58.640 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.191130+0000", 2023-11-05T08:32:58.640 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.6681999999999995e-05 2023-11-05T08:32:58.640 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.641 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:58.641 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.641 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.641 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.641 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as2 42/33 ECSubWrite(tid=38, reqid=osd.7.0:38, at_version=42'16, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:32:58.642 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:55.121755+0000", 2023-11-05T08:32:58.642 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 3.3462074049999999, 2023-11-05T08:32:58.642 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011609566, 2023-11-05T08:32:58.642 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:58.643 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:58.643 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:58.643 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.643 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:58.643 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.121755+0000", 2023-11-05T08:32:58.643 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.644 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.644 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.644 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:58.644 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.121755+0000", 2023-11-05T08:32:58.644 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.645 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.645 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.645 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:58.645 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.121756+0000", 2023-11-05T08:32:58.645 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.0299999999999998e-07 2023-11-05T08:32:58.646 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.646 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.646 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:58.646 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.121810+0000", 2023-11-05T08:32:58.646 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3918000000000001e-05 2023-11-05T08:32:58.647 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.647 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.647 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:58.647 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.121861+0000", 2023-11-05T08:32:58.647 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.0846000000000001e-05 2023-11-05T08:32:58.648 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.648 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.648 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:58.648 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.121889+0000", 2023-11-05T08:32:58.648 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8359000000000001e-05 2023-11-05T08:32:58.649 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.649 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.649 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:58.649 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.122158+0000", 2023-11-05T08:32:58.649 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00026898800000000001 2023-11-05T08:32:58.650 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.650 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.650 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:58.650 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.122192+0000", 2023-11-05T08:32:58.650 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3259000000000001e-05 2023-11-05T08:32:58.651 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.651 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.651 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:58.651 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.133269+0000", 2023-11-05T08:32:58.651 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011076979000000001 2023-11-05T08:32:58.652 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.652 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.652 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:58.652 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.133365+0000", 2023-11-05T08:32:58.652 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6414000000000002e-05 2023-11-05T08:32:58.653 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.653 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:58.653 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.653 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.653 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.653 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as2 42/33 ECSubWrite(tid=39, reqid=osd.7.0:39, at_version=42'18, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:32:58.654 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:55.124648+0000", 2023-11-05T08:32:58.654 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 3.3433145450000001, 2023-11-05T08:32:58.654 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.025426339999999999, 2023-11-05T08:32:58.654 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:58.654 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:58.655 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:58.655 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.655 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:58.655 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.124648+0000", 2023-11-05T08:32:58.655 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.655 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.656 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.656 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:58.656 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.124648+0000", 2023-11-05T08:32:58.656 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.656 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.656 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.657 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:58.657 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.124650+0000", 2023-11-05T08:32:58.657 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.463e-06 2023-11-05T08:32:58.657 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.657 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.658 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:58.658 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.124710+0000", 2023-11-05T08:32:58.658 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0001999999999999e-05 2023-11-05T08:32:58.658 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.658 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.659 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:58.659 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.124739+0000", 2023-11-05T08:32:58.659 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9278999999999999e-05 2023-11-05T08:32:58.659 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.659 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.659 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:58.660 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.124763+0000", 2023-11-05T08:32:58.660 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4036000000000002e-05 2023-11-05T08:32:58.660 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.660 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.660 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:58.661 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.125122+0000", 2023-11-05T08:32:58.661 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00035926900000000001 2023-11-05T08:32:58.661 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.661 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.661 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:58.661 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.125149+0000", 2023-11-05T08:32:58.662 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7192999999999999e-05 2023-11-05T08:32:58.662 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.662 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.662 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:58.662 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.149969+0000", 2023-11-05T08:32:58.663 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.024819074 2023-11-05T08:32:58.663 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.663 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.663 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:58.663 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.150075+0000", 2023-11-05T08:32:58.664 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000106024 2023-11-05T08:32:58.664 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.664 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:58.664 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.664 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.665 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.665 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es2 42/30 ECSubWrite(tid=43, reqid=osd.2.0:43, at_version=42'29, trim_to=0'0, roll_forward_to=41'27))", 2023-11-05T08:32:58.665 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:55.134544+0000", 2023-11-05T08:32:58.665 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 3.3334186250000002, 2023-11-05T08:32:58.666 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.02091324, 2023-11-05T08:32:58.666 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:58.666 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:58.666 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:58.666 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.667 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:58.667 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134544+0000", 2023-11-05T08:32:58.667 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.667 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.667 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.668 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:58.668 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134544+0000", 2023-11-05T08:32:58.668 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.668 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.668 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.669 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:58.669 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134545+0000", 2023-11-05T08:32:58.669 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3499999999999996e-07 2023-11-05T08:32:58.669 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.669 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.670 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:58.670 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134593+0000", 2023-11-05T08:32:58.670 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.8538999999999997e-05 2023-11-05T08:32:58.670 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.671 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.671 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:58.671 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134621+0000", 2023-11-05T08:32:58.671 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7939000000000001e-05 2023-11-05T08:32:58.671 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.671 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.672 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:58.672 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134641+0000", 2023-11-05T08:32:58.672 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9352999999999999e-05 2023-11-05T08:32:58.672 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.672 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.673 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:58.673 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134788+0000", 2023-11-05T08:32:58.673 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000147806 2023-11-05T08:32:58.673 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.673 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.674 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:58.674 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134850+0000", 2023-11-05T08:32:58.674 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1614999999999999e-05 2023-11-05T08:32:58.674 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.674 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.675 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:58.675 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.155400+0000", 2023-11-05T08:32:58.675 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.020550023000000001 2023-11-05T08:32:58.675 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.675 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.676 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:58.676 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.155457+0000", 2023-11-05T08:32:58.676 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7330000000000002e-05 2023-11-05T08:32:58.676 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.676 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:58.677 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.677 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.677 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.677 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.2s5 42/40 ECSubWrite(tid=192, reqid=osd.13.0:192, at_version=42'13, trim_to=0'0, roll_forward_to=41'11))", 2023-11-05T08:32:58.677 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:55.150316+0000", 2023-11-05T08:32:58.677 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 3.317647129, 2023-11-05T08:32:58.678 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0088926939999999996, 2023-11-05T08:32:58.678 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:58.678 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:58.678 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:58.678 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.679 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:58.679 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.150316+0000", 2023-11-05T08:32:58.679 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.679 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.679 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.680 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:58.680 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.150316+0000", 2023-11-05T08:32:58.680 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.680 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.680 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.681 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:58.681 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.150318+0000", 2023-11-05T08:32:58.681 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.0140000000000001e-06 2023-11-05T08:32:58.681 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.681 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.682 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:58.682 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.150377+0000", 2023-11-05T08:32:58.682 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9231000000000003e-05 2023-11-05T08:32:58.682 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.682 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.683 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:58.683 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.150411+0000", 2023-11-05T08:32:58.683 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4279000000000002e-05 2023-11-05T08:32:58.683 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.683 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.684 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:58.684 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.150435+0000", 2023-11-05T08:32:58.684 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4165999999999999e-05 2023-11-05T08:32:58.684 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.685 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.685 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:58.685 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.152989+0000", 2023-11-05T08:32:58.685 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0025534889999999999 2023-11-05T08:32:58.685 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.685 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.686 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:58.686 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.153035+0000", 2023-11-05T08:32:58.686 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.6078000000000002e-05 2023-11-05T08:32:58.686 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.686 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.687 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:58.687 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.159134+0000", 2023-11-05T08:32:58.687 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0060986069999999998 2023-11-05T08:32:58.687 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.687 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.688 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:58.688 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.159208+0000", 2023-11-05T08:32:58.688 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.483e-05 2023-11-05T08:32:58.688 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.688 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:58.689 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.689 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.689 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.689 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as2 42/33 ECSubWrite(tid=40, reqid=osd.7.0:39, at_version=0'0, trim_to=0'0, roll_forward_to=42'18))", 2023-11-05T08:32:58.690 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:55.151263+0000", 2023-11-05T08:32:58.690 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 3.3166996960000001, 2023-11-05T08:32:58.690 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0080059020000000005, 2023-11-05T08:32:58.690 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:58.690 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:58.691 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:58.691 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.691 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:58.691 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.151263+0000", 2023-11-05T08:32:58.691 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.691 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.693 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.693 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:58.693 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.151263+0000", 2023-11-05T08:32:58.694 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.694 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.694 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.694 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:58.694 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.151265+0000", 2023-11-05T08:32:58.695 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9939999999999999e-06 2023-11-05T08:32:58.695 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.695 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.695 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:58.695 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.151295+0000", 2023-11-05T08:32:58.696 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0355e-05 2023-11-05T08:32:58.696 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.696 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.696 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:58.696 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.151328+0000", 2023-11-05T08:32:58.696 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2246000000000003e-05 2023-11-05T08:32:58.697 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.697 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.697 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:58.697 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.151359+0000", 2023-11-05T08:32:58.697 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1458999999999998e-05 2023-11-05T08:32:58.697 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.698 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.698 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:58.698 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.153854+0000", 2023-11-05T08:32:58.698 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002495115 2023-11-05T08:32:58.698 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.699 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.699 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:58.699 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.153879+0000", 2023-11-05T08:32:58.699 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5065000000000001e-05 2023-11-05T08:32:58.699 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.700 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.700 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:58.700 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.159215+0000", 2023-11-05T08:32:58.700 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0053359690000000003 2023-11-05T08:32:58.700 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.701 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.701 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:58.701 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.159269+0000", 2023-11-05T08:32:58.701 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3699000000000002e-05 2023-11-05T08:32:58.701 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.702 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:58.702 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.702 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.702 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.702 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es2 42/30 ECSubWrite(tid=44, reqid=osd.2.0:43, at_version=0'0, trim_to=0'0, roll_forward_to=42'29))", 2023-11-05T08:32:58.703 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:55.156340+0000", 2023-11-05T08:32:58.703 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 3.3116223360000001, 2023-11-05T08:32:58.703 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0046627070000000003, 2023-11-05T08:32:58.703 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:58.704 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:58.704 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:58.704 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.704 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:58.704 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.156340+0000", 2023-11-05T08:32:58.704 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.705 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.705 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.705 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:58.705 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.156340+0000", 2023-11-05T08:32:58.705 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.706 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.706 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.706 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:58.706 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.156342+0000", 2023-11-05T08:32:58.706 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7710000000000001e-06 2023-11-05T08:32:58.707 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.707 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.707 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:58.707 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.156368+0000", 2023-11-05T08:32:58.707 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5273999999999999e-05 2023-11-05T08:32:58.708 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.708 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.708 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:58.708 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.156414+0000", 2023-11-05T08:32:58.708 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.6238e-05 2023-11-05T08:32:58.709 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.709 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.709 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:58.709 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.156452+0000", 2023-11-05T08:32:58.709 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8516999999999997e-05 2023-11-05T08:32:58.710 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.710 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.710 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:58.710 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.156693+0000", 2023-11-05T08:32:58.711 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00024051500000000001 2023-11-05T08:32:58.711 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.711 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.711 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:58.711 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.156740+0000", 2023-11-05T08:32:58.711 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7719000000000002e-05 2023-11-05T08:32:58.712 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.712 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.712 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:58.712 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.160927+0000", 2023-11-05T08:32:58.713 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0041866780000000001 2023-11-05T08:32:58.713 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.713 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.713 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:58.713 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.161003+0000", 2023-11-05T08:32:58.714 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.5995000000000007e-05 2023-11-05T08:32:58.714 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.714 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:58.714 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.714 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.714 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:58.715 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:32:58.716 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-11-05T08:32:58.717 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:32:58.715+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:32:58.817 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:32:58.815+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:32:58.849 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:32:58.850 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2023-11-05T08:32:58.850 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.850 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:317 3.17s0 3.56317c97 (undecoded) ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e45)", 2023-11-05T08:32:58.850 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:58.238377+0000", 2023-11-05T08:32:58.850 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.611074428, 2023-11-05T08:32:58.851 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.61111891399999996, 2023-11-05T08:32:58.851 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:58.851 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "waiting for peered", 2023-11-05T08:32:58.851 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:32:58.852 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:32:58.852 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:32:58.852 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 317 2023-11-05T08:32:58.852 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.852 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:58.853 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.853 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:58.853 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:58.238377+0000", 2023-11-05T08:32:58.853 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.853 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.853 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.854 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:58.854 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:58.238377+0000", 2023-11-05T08:32:58.854 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:58.854 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.854 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.855 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:58.855 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:58.238379+0000", 2023-11-05T08:32:58.855 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9460000000000001e-06 2023-11-05T08:32:58.855 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.855 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.855 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:58.856 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:58.238390+0000", 2023-11-05T08:32:58.856 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1065e-05 2023-11-05T08:32:58.856 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.856 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.856 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:58.856 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:58.238412+0000", 2023-11-05T08:32:58.857 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.2529999999999999e-05 2023-11-05T08:32:58.857 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.857 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.857 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:58.857 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:58.238425+0000", 2023-11-05T08:32:58.857 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3077e-05 2023-11-05T08:32:58.858 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.858 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.858 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:58.858 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:58.238536+0000", 2023-11-05T08:32:58.858 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00011045899999999999 2023-11-05T08:32:58.859 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:58.859 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:58.859 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2023-11-05T08:32:58.859 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:58.238541+0000", 2023-11-05T08:32:58.859 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9710000000000003e-06 2023-11-05T08:32:58.859 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.860 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:58.860 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.860 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:58.860 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2023-11-05T08:32:58.860 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 1 2023-11-05T08:32:58.861 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:32:58.864 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-11-05T08:32:58.918 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:32:58.913+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:32:58.989 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:32:58.990 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2023-11-05T08:32:58.990 INFO:teuthology.orchestra.run.smithi053.stdout: "complaint_time": 30, 2023-11-05T08:32:58.990 INFO:teuthology.orchestra.run.smithi053.stdout: "num_blocked_ops": 0 2023-11-05T08:32:58.990 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:32:59.004 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-11-05T08:32:59.007 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op export --pgid 2.2 --file /var/log/ceph/exp.2.2.3 2023-11-05T08:32:59.046 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:32:59.045+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:32:59.120 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:32:59.117+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:32:59.180 INFO:teuthology.orchestra.run.smithi002.stderr:noscrub is unset 2023-11-05T08:32:59.187 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:32:59.187 INFO:teuthology.orchestra.run.smithi053.stdout: "size": 20, 2023-11-05T08:32:59.187 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 600, 2023-11-05T08:32:59.187 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2023-11-05T08:32:59.188 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.188 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.3s5 35/30 ECSubWrite(tid=179, reqid=client.4691.0:218, at_version=35'8, trim_to=0'0, roll_forward_to=33'6))", 2023-11-05T08:32:59.188 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:48.054791+0000", 2023-11-05T08:32:59.188 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 11.12505964, 2023-11-05T08:32:59.189 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.003742544, 2023-11-05T08:32:59.189 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:59.189 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:59.189 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:59.189 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.190 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:59.190 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.054791+0000", 2023-11-05T08:32:59.190 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.191 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.191 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.191 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:59.191 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.054791+0000", 2023-11-05T08:32:59.192 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.192 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.192 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.192 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:59.193 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.054792+0000", 2023-11-05T08:32:59.193 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.13e-07 2023-11-05T08:32:59.193 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.193 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.194 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:59.194 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.054840+0000", 2023-11-05T08:32:59.194 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.8112000000000003e-05 2023-11-05T08:32:59.194 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.194 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.195 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:59.195 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.054862+0000", 2023-11-05T08:32:59.195 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.2489000000000001e-05 2023-11-05T08:32:59.195 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.196 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.196 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:59.196 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.054878+0000", 2023-11-05T08:32:59.196 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6116e-05 2023-11-05T08:32:59.196 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.197 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.197 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:59.197 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.055095+0000", 2023-11-05T08:32:59.197 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00021669300000000001 2023-11-05T08:32:59.198 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.198 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.198 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:59.198 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.055111+0000", 2023-11-05T08:32:59.198 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6005999999999999e-05 2023-11-05T08:32:59.199 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.199 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.199 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:59.199 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.058420+0000", 2023-11-05T08:32:59.199 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0033094019999999999 2023-11-05T08:32:59.200 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.200 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.200 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:59.200 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.058533+0000", 2023-11-05T08:32:59.200 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000113113 2023-11-05T08:32:59.200 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.201 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:59.201 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.201 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.201 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.201 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.3s5 35/30 ECSubWrite(tid=180, reqid=client.4691.0:219, at_version=35'9, trim_to=0'0, roll_forward_to=33'6))", 2023-11-05T08:32:59.202 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:48.057317+0000", 2023-11-05T08:32:59.202 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 11.122533315, 2023-11-05T08:32:59.202 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0029650029999999999, 2023-11-05T08:32:59.202 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:59.202 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:59.203 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:59.203 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.203 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:59.203 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.057317+0000", 2023-11-05T08:32:59.204 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.204 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.204 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.204 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:59.204 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.057317+0000", 2023-11-05T08:32:59.205 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.205 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.205 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.205 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:59.205 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.057319+0000", 2023-11-05T08:32:59.206 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7370000000000001e-06 2023-11-05T08:32:59.206 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.206 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.206 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:59.207 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.057367+0000", 2023-11-05T08:32:59.207 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.8273999999999999e-05 2023-11-05T08:32:59.207 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.207 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.207 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:59.208 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.057405+0000", 2023-11-05T08:32:59.208 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.7805999999999997e-05 2023-11-05T08:32:59.208 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.208 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.208 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:59.209 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.057434+0000", 2023-11-05T08:32:59.209 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9026e-05 2023-11-05T08:32:59.209 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.209 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.209 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:59.210 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.057596+0000", 2023-11-05T08:32:59.210 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00016196 2023-11-05T08:32:59.210 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.210 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.211 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:59.211 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.057614+0000", 2023-11-05T08:32:59.211 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7897000000000002e-05 2023-11-05T08:32:59.211 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.212 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.212 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:59.212 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.060168+0000", 2023-11-05T08:32:59.212 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002554317 2023-11-05T08:32:59.212 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.213 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.213 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:59.213 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:48.060282+0000", 2023-11-05T08:32:59.213 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000113986 2023-11-05T08:32:59.213 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.213 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:59.214 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.214 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.214 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.214 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.3s5 36/30 ECSubWrite(tid=183, reqid=osd.13.0:183, at_version=36'12, trim_to=0'0, roll_forward_to=35'10))", 2023-11-05T08:32:59.214 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:49.070063+0000", 2023-11-05T08:32:59.215 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 10.109787216000001, 2023-11-05T08:32:59.215 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.003714321, 2023-11-05T08:32:59.215 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:59.215 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:59.215 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:59.216 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.216 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:59.216 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.070063+0000", 2023-11-05T08:32:59.216 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.217 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.217 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.217 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:59.217 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.070063+0000", 2023-11-05T08:32:59.217 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.218 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.218 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.218 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:59.218 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.070064+0000", 2023-11-05T08:32:59.219 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.2900000000000002e-07 2023-11-05T08:32:59.219 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.219 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.219 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:59.219 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.070123+0000", 2023-11-05T08:32:59.220 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8941999999999999e-05 2023-11-05T08:32:59.220 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.220 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.220 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:59.220 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.070192+0000", 2023-11-05T08:32:59.221 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9181000000000001e-05 2023-11-05T08:32:59.221 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.221 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.221 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:59.221 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.070222+0000", 2023-11-05T08:32:59.222 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9128999999999998e-05 2023-11-05T08:32:59.222 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.222 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.222 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:59.222 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.070429+0000", 2023-11-05T08:32:59.223 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000207197 2023-11-05T08:32:59.223 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.223 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.223 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:59.224 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.070456+0000", 2023-11-05T08:32:59.224 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7695000000000002e-05 2023-11-05T08:32:59.224 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.224 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.224 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:59.225 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.073701+0000", 2023-11-05T08:32:59.225 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.003244884 2023-11-05T08:32:59.225 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.225 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.226 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:59.226 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.073778+0000", 2023-11-05T08:32:59.226 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.6364000000000003e-05 2023-11-05T08:32:59.226 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.227 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:59.227 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.227 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.227 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.227 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.3s5 36/30 ECSubWrite(tid=184, reqid=osd.13.0:183, at_version=0'0, trim_to=0'0, roll_forward_to=36'12))", 2023-11-05T08:32:59.228 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:49.079793+0000", 2023-11-05T08:32:59.228 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 10.100057998, 2023-11-05T08:32:59.228 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0039516839999999996, 2023-11-05T08:32:59.229 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:59.229 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:59.229 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:59.229 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.229 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:59.229 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.079793+0000", 2023-11-05T08:32:59.230 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.230 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.230 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.230 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:59.231 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.079793+0000", 2023-11-05T08:32:59.231 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.231 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.231 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.231 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:59.232 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.079795+0000", 2023-11-05T08:32:59.232 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9630000000000001e-06 2023-11-05T08:32:59.232 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.232 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.233 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:59.233 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.079823+0000", 2023-11-05T08:32:59.233 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8235999999999999e-05 2023-11-05T08:32:59.233 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.234 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.234 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:59.234 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.079855+0000", 2023-11-05T08:32:59.234 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1974999999999998e-05 2023-11-05T08:32:59.235 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.235 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.235 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:59.235 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.079909+0000", 2023-11-05T08:32:59.235 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3939e-05 2023-11-05T08:32:59.236 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.236 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.236 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:59.236 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.080164+0000", 2023-11-05T08:32:59.236 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00025553099999999998 2023-11-05T08:32:59.237 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.237 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.237 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:59.237 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.080193+0000", 2023-11-05T08:32:59.238 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8359000000000001e-05 2023-11-05T08:32:59.238 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.238 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.238 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:59.238 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.083646+0000", 2023-11-05T08:32:59.239 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0034532209999999998 2023-11-05T08:32:59.239 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.239 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.239 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:59.239 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.083744+0000", 2023-11-05T08:32:59.240 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.8460000000000003e-05 2023-11-05T08:32:59.240 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.240 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:59.240 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.241 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.241 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.241 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 36/30 ECSubWriteReply(tid=23, last_complete=36'15, committed=1, applied=1))", 2023-11-05T08:32:59.241 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:49.086812+0000", 2023-11-05T08:32:59.241 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 10.093038143999999, 2023-11-05T08:32:59.242 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0030445369999999999, 2023-11-05T08:32:59.242 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:59.242 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:59.242 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:59.243 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.243 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:59.243 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.086812+0000", 2023-11-05T08:32:59.243 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.244 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.244 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.244 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:59.244 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.086812+0000", 2023-11-05T08:32:59.244 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.244 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.245 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.245 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:59.245 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.086814+0000", 2023-11-05T08:32:59.245 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7099999999999999e-06 2023-11-05T08:32:59.246 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.246 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.246 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:59.246 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.086823+0000", 2023-11-05T08:32:59.247 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.799e-06 2023-11-05T08:32:59.247 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.247 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.247 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:59.247 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.086853+0000", 2023-11-05T08:32:59.248 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9916e-05 2023-11-05T08:32:59.248 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.248 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.248 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:59.248 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.086874+0000", 2023-11-05T08:32:59.249 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.0696e-05 2023-11-05T08:32:59.249 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.249 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.249 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:59.249 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.087043+0000", 2023-11-05T08:32:59.250 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00016991400000000001 2023-11-05T08:32:59.250 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.250 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.250 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:59.250 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.089857+0000", 2023-11-05T08:32:59.251 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0028135019999999998 2023-11-05T08:32:59.251 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.251 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:59.251 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.251 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.251 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.252 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:271 3.18s0 3:1aa38538:::smithi09039252-11:5 [read 0~1097728,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:32:59.252 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:53.109072+0000", 2023-11-05T08:32:59.252 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 6.0707788269999998, 2023-11-05T08:32:59.252 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.069197392, 2023-11-05T08:32:59.252 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:59.252 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2023-11-05T08:32:59.253 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:32:59.253 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:32:59.253 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:32:59.253 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 271 2023-11-05T08:32:59.253 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.254 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:59.254 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.254 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:59.254 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109072+0000", 2023-11-05T08:32:59.254 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.255 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.255 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.255 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:59.255 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109072+0000", 2023-11-05T08:32:59.255 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.255 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.256 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.256 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:59.256 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109076+0000", 2023-11-05T08:32:59.256 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.9639999999999997e-06 2023-11-05T08:32:59.257 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.257 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.257 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:59.257 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109094+0000", 2023-11-05T08:32:59.257 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7995e-05 2023-11-05T08:32:59.257 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.258 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.258 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:59.258 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109134+0000", 2023-11-05T08:32:59.258 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.0729000000000003e-05 2023-11-05T08:32:59.258 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.258 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.259 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:59.259 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109160+0000", 2023-11-05T08:32:59.259 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5276e-05 2023-11-05T08:32:59.259 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.259 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.259 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:59.260 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.134337+0000", 2023-11-05T08:32:59.260 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.025177609 2023-11-05T08:32:59.260 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.260 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.260 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2023-11-05T08:32:59.261 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.134667+0000", 2023-11-05T08:32:59.261 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00032950600000000003 2023-11-05T08:32:59.261 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.261 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.261 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:59.262 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.166070+0000", 2023-11-05T08:32:59.262 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0314031800000001 2023-11-05T08:32:59.262 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.262 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.262 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:32:59.262 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.166695+0000", 2023-11-05T08:32:59.263 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000625137 2023-11-05T08:32:59.263 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.263 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.263 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:59.263 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.178269+0000", 2023-11-05T08:32:59.264 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011573996 2023-11-05T08:32:59.264 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.264 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:59.264 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.264 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.265 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.265 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:272 3.18s0 3:1aa38538:::smithi09039252-11:5 [read 0~1097728,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:32:59.265 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:53.109221+0000", 2023-11-05T08:32:59.265 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 6.0706292150000003, 2023-11-05T08:32:59.265 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0781554369999999, 2023-11-05T08:32:59.266 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:59.266 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2023-11-05T08:32:59.266 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:32:59.266 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:32:59.267 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:32:59.267 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 272 2023-11-05T08:32:59.267 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.267 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:59.267 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.268 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:59.268 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109221+0000", 2023-11-05T08:32:59.268 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.268 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.269 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.269 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:59.269 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109221+0000", 2023-11-05T08:32:59.269 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.270 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.270 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.270 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:59.270 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109223+0000", 2023-11-05T08:32:59.271 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4330000000000001e-06 2023-11-05T08:32:59.271 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.271 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.271 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:59.271 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109232+0000", 2023-11-05T08:32:59.272 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.234e-06 2023-11-05T08:32:59.272 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.272 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.272 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:59.272 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109254+0000", 2023-11-05T08:32:59.272 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1576000000000001e-05 2023-11-05T08:32:59.273 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.273 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.273 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:59.273 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109270+0000", 2023-11-05T08:32:59.274 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6396e-05 2023-11-05T08:32:59.274 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.274 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.275 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:59.275 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.134760+0000", 2023-11-05T08:32:59.275 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.025490292000000001 2023-11-05T08:32:59.275 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.275 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.276 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2023-11-05T08:32:59.276 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.134764+0000", 2023-11-05T08:32:59.276 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5470000000000001e-06 2023-11-05T08:32:59.276 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.276 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.277 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:59.277 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.178397+0000", 2023-11-05T08:32:59.277 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.04363341 2023-11-05T08:32:59.277 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.277 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.278 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:32:59.278 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.178606+0000", 2023-11-05T08:32:59.278 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00020828200000000001 2023-11-05T08:32:59.278 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.279 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.279 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:59.279 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.187377+0000", 2023-11-05T08:32:59.279 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0087712669999999993 2023-11-05T08:32:59.280 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.280 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:59.280 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.280 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.280 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.281 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:273 3.18s0 3:1aa38538:::smithi09039252-11:5 [read 0~1097728,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:32:59.281 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:53.109291+0000", 2023-11-05T08:32:59.281 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 6.0705594439999997, 2023-11-05T08:32:59.281 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.08037483, 2023-11-05T08:32:59.282 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:59.282 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2023-11-05T08:32:59.282 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:32:59.282 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:32:59.283 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:32:59.283 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 273 2023-11-05T08:32:59.283 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.283 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:59.283 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.284 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:59.284 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109291+0000", 2023-11-05T08:32:59.284 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.284 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.284 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.285 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:59.285 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109291+0000", 2023-11-05T08:32:59.285 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.285 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.286 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.286 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:59.286 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109292+0000", 2023-11-05T08:32:59.286 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.054e-06 2023-11-05T08:32:59.286 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.287 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.287 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:59.287 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109299+0000", 2023-11-05T08:32:59.288 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.7700000000000004e-06 2023-11-05T08:32:59.288 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.288 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.288 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:59.288 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109317+0000", 2023-11-05T08:32:59.289 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.8529000000000002e-05 2023-11-05T08:32:59.289 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.289 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.289 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:59.290 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109332+0000", 2023-11-05T08:32:59.290 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4905999999999999e-05 2023-11-05T08:32:59.290 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.290 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.291 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:59.291 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.134830+0000", 2023-11-05T08:32:59.291 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.025497912000000001 2023-11-05T08:32:59.291 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.291 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.292 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2023-11-05T08:32:59.292 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.134832+0000", 2023-11-05T08:32:59.292 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1399999999999998e-06 2023-11-05T08:32:59.292 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.292 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.293 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:59.293 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.179990+0000", 2023-11-05T08:32:59.293 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0451579209999999 2023-11-05T08:32:59.293 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.293 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.294 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:32:59.294 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.180250+0000", 2023-11-05T08:32:59.294 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000259595 2023-11-05T08:32:59.294 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.295 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.295 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:59.295 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.189666+0000", 2023-11-05T08:32:59.295 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0094160029999999992 2023-11-05T08:32:59.296 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.296 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:59.296 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.296 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.297 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.297 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpRead(3.18s0 41/40 ECSubRead(tid=25, to_read={3:1aa38538:::smithi09039252-11:head=0,274432,0}, subchunks={3:1aa38538:::smithi09039252-11:head=[0,1]}, attrs_to_read=))", 2023-11-05T08:32:59.297 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.169004+0000", 2023-11-05T08:32:59.297 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 5.010846667, 2023-11-05T08:32:59.297 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0053158040000000004, 2023-11-05T08:32:59.298 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:59.298 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:59.298 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:59.298 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.299 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:59.299 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.169004+0000", 2023-11-05T08:32:59.299 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.299 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.300 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.300 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:59.300 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.169004+0000", 2023-11-05T08:32:59.300 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.300 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.301 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.301 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:59.301 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.169004+0000", 2023-11-05T08:32:59.301 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.301 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.302 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.302 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:59.302 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.169004+0000", 2023-11-05T08:32:59.302 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.302 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.303 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.303 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:59.303 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.169294+0000", 2023-11-05T08:32:59.303 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00029003200000000002 2023-11-05T08:32:59.303 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.304 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.304 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:59.304 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.169317+0000", 2023-11-05T08:32:59.304 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.2620999999999999e-05 2023-11-05T08:32:59.305 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.305 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.305 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:59.305 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.173103+0000", 2023-11-05T08:32:59.305 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0037861959999999999 2023-11-05T08:32:59.306 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.306 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.306 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:59.306 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.174320+0000", 2023-11-05T08:32:59.307 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001216955 2023-11-05T08:32:59.307 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.307 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:59.307 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.307 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.308 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.308 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpReadReply(3.18s0 41/40 ECSubReadReply(tid=25, attrs_read=0))", 2023-11-05T08:32:59.308 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.170677+0000", 2023-11-05T08:32:59.308 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 5.0091740500000004, 2023-11-05T08:32:59.309 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0037705830000000001, 2023-11-05T08:32:59.309 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:59.309 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:59.309 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:59.309 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.310 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:59.310 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.170677+0000", 2023-11-05T08:32:59.310 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.310 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.311 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.311 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:59.311 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.170677+0000", 2023-11-05T08:32:59.311 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.311 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.312 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.312 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:59.312 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.170678+0000", 2023-11-05T08:32:59.312 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.637e-06 2023-11-05T08:32:59.312 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.313 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.313 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:59.313 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171184+0000", 2023-11-05T08:32:59.313 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00050578500000000002 2023-11-05T08:32:59.313 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.314 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.314 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:59.314 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171445+0000", 2023-11-05T08:32:59.314 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00026120000000000001 2023-11-05T08:32:59.314 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.314 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.315 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:59.315 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171463+0000", 2023-11-05T08:32:59.315 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.8017e-05 2023-11-05T08:32:59.315 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.315 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.316 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:59.316 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.174371+0000", 2023-11-05T08:32:59.316 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0029081160000000001 2023-11-05T08:32:59.316 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.316 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.317 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:59.317 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.174447+0000", 2023-11-05T08:32:59.317 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.5828000000000007e-05 2023-11-05T08:32:59.317 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.317 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:59.318 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.318 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.318 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.318 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpReadReply(3.18s0 41/40 ECSubReadReply(tid=25, attrs_read=0))", 2023-11-05T08:32:59.318 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.171585+0000", 2023-11-05T08:32:59.318 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 5.0082652049999998, 2023-11-05T08:32:59.319 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0029934340000000001, 2023-11-05T08:32:59.319 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:59.319 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:59.319 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:59.319 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.319 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:59.320 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171585+0000", 2023-11-05T08:32:59.320 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.320 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.320 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.320 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:59.321 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171585+0000", 2023-11-05T08:32:59.321 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.321 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.321 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.321 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:59.321 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171587+0000", 2023-11-05T08:32:59.322 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5999999999999999e-06 2023-11-05T08:32:59.322 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.322 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.322 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:59.322 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.172296+0000", 2023-11-05T08:32:59.323 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00070894600000000001 2023-11-05T08:32:59.323 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.323 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.323 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:59.324 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.172325+0000", 2023-11-05T08:32:59.324 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8628000000000001e-05 2023-11-05T08:32:59.324 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.324 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.324 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:59.325 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.172341+0000", 2023-11-05T08:32:59.325 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6124999999999999e-05 2023-11-05T08:32:59.325 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.325 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.326 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:59.326 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.174511+0000", 2023-11-05T08:32:59.326 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0021707269999999999 2023-11-05T08:32:59.326 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.326 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.327 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:59.327 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.174579+0000", 2023-11-05T08:32:59.327 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.7408000000000002e-05 2023-11-05T08:32:59.327 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.327 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:59.328 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.328 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.328 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.328 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpReadReply(3.18s0 41/40 ECSubReadReply(tid=25, attrs_read=0))", 2023-11-05T08:32:59.329 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.173059+0000", 2023-11-05T08:32:59.329 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 5.0067914870000001, 2023-11-05T08:32:59.329 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0069229169999999998, 2023-11-05T08:32:59.329 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:59.330 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:59.330 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:59.330 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.330 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:59.331 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.173059+0000", 2023-11-05T08:32:59.331 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.331 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.331 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.331 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:59.332 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.173059+0000", 2023-11-05T08:32:59.332 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.332 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.332 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.332 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:59.333 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.173061+0000", 2023-11-05T08:32:59.333 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4720000000000001e-06 2023-11-05T08:32:59.333 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.333 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.333 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:59.333 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.174176+0000", 2023-11-05T08:32:59.334 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001115904 2023-11-05T08:32:59.334 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.334 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.334 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:59.334 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.174201+0000", 2023-11-05T08:32:59.335 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4281e-05 2023-11-05T08:32:59.335 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.335 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.335 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:59.335 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.174215+0000", 2023-11-05T08:32:59.336 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4575e-05 2023-11-05T08:32:59.336 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.336 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.336 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:59.336 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.174738+0000", 2023-11-05T08:32:59.336 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000523122 2023-11-05T08:32:59.337 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.337 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.337 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:59.337 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.179982+0000", 2023-11-05T08:32:59.337 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0052435629999999997 2023-11-05T08:32:59.337 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.338 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:59.338 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.338 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.338 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.338 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.11s3 41/40 ECSubWrite(tid=65, reqid=client.4691.0:287, at_version=41'13, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:32:59.339 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.173108+0000", 2023-11-05T08:32:59.339 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 5.0067421540000003, 2023-11-05T08:32:59.339 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012254233999999999, 2023-11-05T08:32:59.339 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:59.339 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:59.339 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:59.340 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.340 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:59.340 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.173108+0000", 2023-11-05T08:32:59.340 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.340 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.341 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.341 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:59.341 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.173108+0000", 2023-11-05T08:32:59.341 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.341 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.341 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.342 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:59.342 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.173109+0000", 2023-11-05T08:32:59.342 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.7700000000000004e-07 2023-11-05T08:32:59.342 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.343 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.343 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:59.343 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.174570+0000", 2023-11-05T08:32:59.343 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00146053 2023-11-05T08:32:59.343 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.344 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.344 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:59.344 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.174598+0000", 2023-11-05T08:32:59.344 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8404000000000001e-05 2023-11-05T08:32:59.345 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.345 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.345 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:59.345 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.174614+0000", 2023-11-05T08:32:59.345 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6436999999999999e-05 2023-11-05T08:32:59.346 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.346 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.346 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:59.346 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.174752+0000", 2023-11-05T08:32:59.346 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013757799999999999 2023-11-05T08:32:59.347 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.347 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.347 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:59.347 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.174775+0000", 2023-11-05T08:32:59.348 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.287e-05 2023-11-05T08:32:59.348 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.348 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.348 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:59.348 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.185293+0000", 2023-11-05T08:32:59.349 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010518192000000001 2023-11-05T08:32:59.349 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.349 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.349 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:59.349 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.185363+0000", 2023-11-05T08:32:59.350 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9546000000000001e-05 2023-11-05T08:32:59.350 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.350 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:59.350 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.350 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.351 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.351 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.11s3 41/40 ECSubWrite(tid=66, reqid=client.4691.0:288, at_version=41'14, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:32:59.351 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.177594+0000", 2023-11-05T08:32:59.351 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 5.0022569580000003, 2023-11-05T08:32:59.351 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0090366969999999998, 2023-11-05T08:32:59.352 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:59.352 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:59.352 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:59.352 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.352 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:59.353 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.177594+0000", 2023-11-05T08:32:59.353 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.353 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.353 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.354 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:59.354 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.177594+0000", 2023-11-05T08:32:59.354 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.354 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.354 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.355 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:59.355 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.177595+0000", 2023-11-05T08:32:59.355 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6599999999999994e-07 2023-11-05T08:32:59.355 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.355 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.356 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:59.356 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.180867+0000", 2023-11-05T08:32:59.356 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.003272264 2023-11-05T08:32:59.356 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.356 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.357 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:59.357 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.180893+0000", 2023-11-05T08:32:59.357 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5956e-05 2023-11-05T08:32:59.357 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.357 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.358 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:59.358 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.180915+0000", 2023-11-05T08:32:59.358 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1756999999999999e-05 2023-11-05T08:32:59.358 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.358 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.358 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:59.359 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.181066+0000", 2023-11-05T08:32:59.359 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000151213 2023-11-05T08:32:59.359 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.359 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.359 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:59.360 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.181085+0000", 2023-11-05T08:32:59.360 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9389999999999999e-05 2023-11-05T08:32:59.360 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.360 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.361 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:59.361 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.186564+0000", 2023-11-05T08:32:59.361 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0054784569999999999 2023-11-05T08:32:59.361 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.361 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.362 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:59.362 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.186630+0000", 2023-11-05T08:32:59.362 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.6693999999999995e-05 2023-11-05T08:32:59.362 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.362 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:59.362 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.363 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.363 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.363 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.11s3 41/40 ECSubWrite(tid=67, reqid=client.4691.0:289, at_version=41'15, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:32:59.363 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.182510+0000", 2023-11-05T08:32:59.364 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4.9973406650000003, 2023-11-05T08:32:59.364 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.004164063, 2023-11-05T08:32:59.364 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:59.364 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:59.364 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:59.364 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.365 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:59.365 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.182510+0000", 2023-11-05T08:32:59.365 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.365 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.365 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.366 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:59.366 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.182510+0000", 2023-11-05T08:32:59.366 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.366 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.367 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.367 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:59.367 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.182512+0000", 2023-11-05T08:32:59.367 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9439999999999999e-06 2023-11-05T08:32:59.368 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.368 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.368 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:59.368 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.182544+0000", 2023-11-05T08:32:59.368 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2527999999999998e-05 2023-11-05T08:32:59.369 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.369 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.369 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:59.369 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.182568+0000", 2023-11-05T08:32:59.370 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3672e-05 2023-11-05T08:32:59.370 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.370 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.370 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:59.370 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.182584+0000", 2023-11-05T08:32:59.371 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5773999999999998e-05 2023-11-05T08:32:59.371 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.371 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.371 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:59.372 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.182702+0000", 2023-11-05T08:32:59.372 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000118128 2023-11-05T08:32:59.372 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.372 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.372 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:59.373 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.182721+0000", 2023-11-05T08:32:59.373 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.8658999999999999e-05 2023-11-05T08:32:59.373 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.373 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.374 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:59.374 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.186637+0000", 2023-11-05T08:32:59.374 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0039165449999999996 2023-11-05T08:32:59.374 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.374 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.375 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:59.375 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.186674+0000", 2023-11-05T08:32:59.375 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.6813000000000001e-05 2023-11-05T08:32:59.375 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.375 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:59.376 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.376 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.376 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.376 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpReadReply(3.18s0 41/40 ECSubReadReply(tid=27, attrs_read=0))", 2023-11-05T08:32:59.376 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.183736+0000", 2023-11-05T08:32:59.376 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4.996114704, 2023-11-05T08:32:59.377 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0038250010000000002, 2023-11-05T08:32:59.377 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:59.377 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:59.377 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:59.377 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.378 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:59.378 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.183736+0000", 2023-11-05T08:32:59.378 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.378 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.379 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.379 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:59.379 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.183736+0000", 2023-11-05T08:32:59.379 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.379 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.380 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.380 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:59.380 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.183737+0000", 2023-11-05T08:32:59.380 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.7199999999999997e-07 2023-11-05T08:32:59.380 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.381 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.381 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:59.381 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.183956+0000", 2023-11-05T08:32:59.381 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000219302 2023-11-05T08:32:59.381 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.382 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.382 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:59.382 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.183973+0000", 2023-11-05T08:32:59.382 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7198e-05 2023-11-05T08:32:59.382 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.383 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.383 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:59.383 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.183987+0000", 2023-11-05T08:32:59.383 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3365e-05 2023-11-05T08:32:59.384 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.384 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.384 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:59.384 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.187480+0000", 2023-11-05T08:32:59.384 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0034932219999999998 2023-11-05T08:32:59.385 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.385 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.385 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:59.385 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.187561+0000", 2023-11-05T08:32:59.385 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.0941999999999999e-05 2023-11-05T08:32:59.386 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.386 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:59.386 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.386 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.387 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.387 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpReadReply(3.18s0 41/40 ECSubReadReply(tid=26, attrs_read=0))", 2023-11-05T08:32:59.387 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.184054+0000", 2023-11-05T08:32:59.387 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4.9957969250000005, 2023-11-05T08:32:59.388 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0033722000000000001, 2023-11-05T08:32:59.388 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:59.388 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:59.388 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:59.388 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.389 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:59.389 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.184054+0000", 2023-11-05T08:32:59.389 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.389 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.389 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.390 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:59.390 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.184054+0000", 2023-11-05T08:32:59.390 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.390 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.390 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.391 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:59.391 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.184055+0000", 2023-11-05T08:32:59.391 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0419999999999999e-06 2023-11-05T08:32:59.391 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.391 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.391 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:59.392 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.184345+0000", 2023-11-05T08:32:59.392 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000290692 2023-11-05T08:32:59.392 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.392 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.392 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:59.393 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.184364+0000", 2023-11-05T08:32:59.393 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.8987e-05 2023-11-05T08:32:59.393 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.393 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.393 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:59.393 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.184376+0000", 2023-11-05T08:32:59.394 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1642e-05 2023-11-05T08:32:59.394 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.394 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.394 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:59.395 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.184709+0000", 2023-11-05T08:32:59.395 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00033334100000000002 2023-11-05T08:32:59.395 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.395 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.395 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:59.396 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.187426+0000", 2023-11-05T08:32:59.396 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0027164960000000001 2023-11-05T08:32:59.396 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.396 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:59.396 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.397 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.397 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.397 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpReadReply(3.18s0 41/40 ECSubReadReply(tid=27, attrs_read=0))", 2023-11-05T08:32:59.397 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.184397+0000", 2023-11-05T08:32:59.397 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4.9954538849999999, 2023-11-05T08:32:59.398 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0053463690000000001, 2023-11-05T08:32:59.398 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:59.398 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:59.398 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:59.398 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.398 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:59.399 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.184397+0000", 2023-11-05T08:32:59.399 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.399 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.399 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.400 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:59.400 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.184397+0000", 2023-11-05T08:32:59.400 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.400 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.400 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.401 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:59.401 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.184397+0000", 2023-11-05T08:32:59.401 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.82e-07 2023-11-05T08:32:59.401 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.401 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.402 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:59.402 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.184675+0000", 2023-11-05T08:32:59.402 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00027796 2023-11-05T08:32:59.402 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.402 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.403 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:59.403 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.184691+0000", 2023-11-05T08:32:59.403 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5372000000000001e-05 2023-11-05T08:32:59.403 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.403 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.404 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:59.404 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.184700+0000", 2023-11-05T08:32:59.404 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6719999999999999e-06 2023-11-05T08:32:59.404 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.404 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.405 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:59.405 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.187651+0000", 2023-11-05T08:32:59.405 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0029503810000000002 2023-11-05T08:32:59.405 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.405 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.406 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:59.406 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.189743+0000", 2023-11-05T08:32:59.406 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0020924020000000002 2023-11-05T08:32:59.406 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.407 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:59.407 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.407 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.407 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.407 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.11s3 41/40 ECSubWrite(tid=68, reqid=client.4691.0:289, at_version=0'0, trim_to=0'0, roll_forward_to=41'15))", 2023-11-05T08:32:59.408 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.194427+0000", 2023-11-05T08:32:59.408 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4.9854233450000001, 2023-11-05T08:32:59.408 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0033829469999999999, 2023-11-05T08:32:59.408 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:59.409 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:59.409 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:59.409 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.409 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:59.409 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.194427+0000", 2023-11-05T08:32:59.410 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.410 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.410 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.410 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:59.410 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.194427+0000", 2023-11-05T08:32:59.411 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.411 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.411 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.411 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:59.412 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.194433+0000", 2023-11-05T08:32:59.412 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0639999999999999e-06 2023-11-05T08:32:59.412 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.412 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.412 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:59.413 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.194451+0000", 2023-11-05T08:32:59.413 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7944e-05 2023-11-05T08:32:59.413 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.413 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.413 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:59.414 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.194490+0000", 2023-11-05T08:32:59.414 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.9165000000000002e-05 2023-11-05T08:32:59.414 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.414 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.415 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:59.415 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.194501+0000", 2023-11-05T08:32:59.415 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0721e-05 2023-11-05T08:32:59.415 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.415 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.416 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:59.416 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.194635+0000", 2023-11-05T08:32:59.416 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013404399999999999 2023-11-05T08:32:59.416 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.416 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.417 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:59.417 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.194659+0000", 2023-11-05T08:32:59.417 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3385000000000001e-05 2023-11-05T08:32:59.417 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.417 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.418 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:59.418 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.197731+0000", 2023-11-05T08:32:59.418 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0030725370000000002 2023-11-05T08:32:59.418 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.418 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.419 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:59.419 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.197810+0000", 2023-11-05T08:32:59.419 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.9086999999999997e-05 2023-11-05T08:32:59.419 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.419 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:59.419 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.420 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.420 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.420 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.13s5 41/40 ECSubWrite(tid=191, reqid=osd.13.0:190, at_version=0'0, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:32:59.420 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.194852+0000", 2023-11-05T08:32:59.420 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4.9849988710000002, 2023-11-05T08:32:59.421 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0038799720000000002, 2023-11-05T08:32:59.421 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:59.421 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:59.421 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:59.421 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.421 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:59.422 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.194852+0000", 2023-11-05T08:32:59.422 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.422 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.422 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.423 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:59.423 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.194852+0000", 2023-11-05T08:32:59.423 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.423 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.423 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.424 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:59.424 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.194853+0000", 2023-11-05T08:32:59.424 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.254e-06 2023-11-05T08:32:59.424 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.425 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.425 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:59.425 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.194873+0000", 2023-11-05T08:32:59.425 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9732e-05 2023-11-05T08:32:59.426 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.426 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.426 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:59.426 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.194894+0000", 2023-11-05T08:32:59.426 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1101e-05 2023-11-05T08:32:59.427 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.427 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.427 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:59.427 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.194909+0000", 2023-11-05T08:32:59.428 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5447999999999998e-05 2023-11-05T08:32:59.428 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.428 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.428 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:59.428 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.195021+0000", 2023-11-05T08:32:59.429 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000112 2023-11-05T08:32:59.429 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.429 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.429 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:59.430 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.195038+0000", 2023-11-05T08:32:59.430 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6684000000000002e-05 2023-11-05T08:32:59.430 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.430 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.430 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:59.431 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.198682+0000", 2023-11-05T08:32:59.431 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0036440209999999999 2023-11-05T08:32:59.431 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.431 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.432 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:59.432 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.198732+0000", 2023-11-05T08:32:59.432 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9731999999999998e-05 2023-11-05T08:32:59.432 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.432 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:59.433 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.433 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.433 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:59.433 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:32:59.435 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2023-11-05T08:32:59.436 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:32:59.435+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:32:59.535 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:32:59.535+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:32:59.615 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:32:59.616 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2023-11-05T08:32:59.616 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 0 2023-11-05T08:32:59.616 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:32:59.631 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2023-11-05T08:32:59.636 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:32:59.635+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:32:59.737 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:32:59.733+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:32:59.756 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:32:59.756 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2023-11-05T08:32:59.757 INFO:teuthology.orchestra.run.smithi053.stdout: "complaint_time": 30, 2023-11-05T08:32:59.757 INFO:teuthology.orchestra.run.smithi053.stdout: "num_blocked_ops": 0 2023-11-05T08:32:59.757 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:32:59.771 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2023-11-05T08:32:59.858 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:32:59.857+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:32:59.939 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:32:59.939+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:32:59.942 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:32:59.942 INFO:teuthology.orchestra.run.smithi053.stdout: "size": 20, 2023-11-05T08:32:59.943 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 600, 2023-11-05T08:32:59.943 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2023-11-05T08:32:59.943 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.943 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es1 36/30 ECSubWrite(tid=33, reqid=osd.2.0:33, at_version=36'21, trim_to=0'0, roll_forward_to=34'19))", 2023-11-05T08:32:59.943 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:49.081168+0000", 2023-11-05T08:32:59.944 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 10.858343916999999, 2023-11-05T08:32:59.944 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0025210570000000002, 2023-11-05T08:32:59.944 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:59.944 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:59.944 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:59.945 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.945 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:59.945 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.081168+0000", 2023-11-05T08:32:59.945 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.945 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.946 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.946 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:59.946 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.081168+0000", 2023-11-05T08:32:59.946 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.946 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.947 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.947 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:59.947 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.081169+0000", 2023-11-05T08:32:59.947 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5899999999999996e-07 2023-11-05T08:32:59.947 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.948 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.948 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:59.948 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.081213+0000", 2023-11-05T08:32:59.948 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3769000000000001e-05 2023-11-05T08:32:59.949 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.949 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.949 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:59.949 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.081246+0000", 2023-11-05T08:32:59.949 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2712000000000001e-05 2023-11-05T08:32:59.950 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.950 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.950 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:59.950 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.081271+0000", 2023-11-05T08:32:59.951 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5199e-05 2023-11-05T08:32:59.951 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.951 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.951 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:59.951 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.081453+0000", 2023-11-05T08:32:59.952 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00018175099999999999 2023-11-05T08:32:59.952 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.952 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.952 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:59.952 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.081481+0000", 2023-11-05T08:32:59.953 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8549000000000001e-05 2023-11-05T08:32:59.953 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.953 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.953 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:59.953 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.083591+0000", 2023-11-05T08:32:59.954 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0021102040000000001 2023-11-05T08:32:59.954 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.954 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.954 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:59.954 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.083690+0000", 2023-11-05T08:32:59.955 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.8214000000000005e-05 2023-11-05T08:32:59.955 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.955 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:59.955 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.955 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.956 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.956 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es1 36/30 ECSubWrite(tid=34, reqid=osd.2.0:33, at_version=0'0, trim_to=0'0, roll_forward_to=36'21))", 2023-11-05T08:32:59.956 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:49.088134+0000", 2023-11-05T08:32:59.956 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 10.851378535, 2023-11-05T08:32:59.956 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0028884520000000001, 2023-11-05T08:32:59.957 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:59.957 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:59.957 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:59.957 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.957 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:59.958 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.088134+0000", 2023-11-05T08:32:59.958 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.958 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.958 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.958 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:59.959 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.088134+0000", 2023-11-05T08:32:59.959 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.959 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.959 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.959 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:59.959 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.088136+0000", 2023-11-05T08:32:59.960 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7430000000000001e-06 2023-11-05T08:32:59.960 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.960 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.960 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:59.960 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.088165+0000", 2023-11-05T08:32:59.961 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8912e-05 2023-11-05T08:32:59.961 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.961 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.961 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:59.961 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.088206+0000", 2023-11-05T08:32:59.962 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1387999999999997e-05 2023-11-05T08:32:59.962 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.962 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.962 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:59.962 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.088234+0000", 2023-11-05T08:32:59.962 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8376000000000001e-05 2023-11-05T08:32:59.963 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.963 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.963 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:59.963 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.088453+0000", 2023-11-05T08:32:59.963 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00021906599999999999 2023-11-05T08:32:59.964 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.964 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.964 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:59.964 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.088480+0000", 2023-11-05T08:32:59.964 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6693e-05 2023-11-05T08:32:59.965 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.965 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.965 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:59.965 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.090852+0000", 2023-11-05T08:32:59.965 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002372103 2023-11-05T08:32:59.965 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.966 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.966 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:59.966 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.091022+0000", 2023-11-05T08:32:59.966 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00017017099999999999 2023-11-05T08:32:59.966 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.967 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:59.967 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.967 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.967 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.967 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es1 40/30 ECSubWrite(tid=35, reqid=client.4691.0:274, at_version=40'23, trim_to=0'0, roll_forward_to=36'21))", 2023-11-05T08:32:59.968 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:53.137984+0000", 2023-11-05T08:32:59.968 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 6.8015284600000001, 2023-11-05T08:32:59.968 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0042231669999999999, 2023-11-05T08:32:59.968 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:59.968 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:59.968 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:59.969 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.969 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:59.969 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.137984+0000", 2023-11-05T08:32:59.969 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.969 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.970 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.970 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:59.970 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.137984+0000", 2023-11-05T08:32:59.970 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.970 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.970 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.971 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:59.971 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.137985+0000", 2023-11-05T08:32:59.971 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4600000000000004e-07 2023-11-05T08:32:59.971 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.971 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.972 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:59.972 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.138720+0000", 2023-11-05T08:32:59.972 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00073507800000000001 2023-11-05T08:32:59.972 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.972 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.973 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:59.973 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.138759+0000", 2023-11-05T08:32:59.973 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.9104999999999999e-05 2023-11-05T08:32:59.973 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.973 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.973 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:59.974 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.138782+0000", 2023-11-05T08:32:59.974 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.2767000000000001e-05 2023-11-05T08:32:59.974 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.974 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.975 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:59.975 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.138933+0000", 2023-11-05T08:32:59.976 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000151559 2023-11-05T08:32:59.976 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.976 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.976 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:59.976 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.138948+0000", 2023-11-05T08:32:59.977 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4674e-05 2023-11-05T08:32:59.977 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.977 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.977 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:59.977 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.142110+0000", 2023-11-05T08:32:59.978 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0031620680000000001 2023-11-05T08:32:59.978 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.978 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.978 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:59.978 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.142207+0000", 2023-11-05T08:32:59.979 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.7269999999999995e-05 2023-11-05T08:32:59.979 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.979 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:59.979 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.979 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.980 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.980 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es1 40/30 ECSubWrite(tid=36, reqid=client.4691.0:275, at_version=40'24, trim_to=0'0, roll_forward_to=36'21))", 2023-11-05T08:32:59.980 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:53.144705+0000", 2023-11-05T08:32:59.980 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 6.7948069039999996, 2023-11-05T08:32:59.980 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.003328119, 2023-11-05T08:32:59.980 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:59.981 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:59.981 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:59.981 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.981 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:59.981 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.144705+0000", 2023-11-05T08:32:59.982 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.982 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.982 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.982 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:59.982 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.144705+0000", 2023-11-05T08:32:59.983 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.983 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.983 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.983 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:59.983 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.144707+0000", 2023-11-05T08:32:59.984 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9999999999999999e-06 2023-11-05T08:32:59.984 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.984 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.984 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:59.984 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.144887+0000", 2023-11-05T08:32:59.985 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00017945799999999999 2023-11-05T08:32:59.985 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.985 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.985 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:59.985 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.144918+0000", 2023-11-05T08:32:59.986 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0908999999999998e-05 2023-11-05T08:32:59.986 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.986 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.986 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:59.986 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.144933+0000", 2023-11-05T08:32:59.987 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4776e-05 2023-11-05T08:32:59.987 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.987 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.987 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:59.987 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.145080+0000", 2023-11-05T08:32:59.988 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014751 2023-11-05T08:32:59.988 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.988 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.988 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:32:59.988 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.145104+0000", 2023-11-05T08:32:59.989 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3708000000000001e-05 2023-11-05T08:32:59.989 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.989 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.989 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:32:59.989 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.147947+0000", 2023-11-05T08:32:59.989 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002843141 2023-11-05T08:32:59.990 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.990 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.990 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:32:59.990 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.148034+0000", 2023-11-05T08:32:59.990 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.6617000000000001e-05 2023-11-05T08:32:59.991 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.991 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:32:59.991 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:32:59.991 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.991 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.992 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as3 40/33 ECSubWrite(tid=30, reqid=client.4691.0:283, at_version=40'8, trim_to=0'0, roll_forward_to=39'6))", 2023-11-05T08:32:59.992 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:53.168454+0000", 2023-11-05T08:32:59.992 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 6.7710586160000004, 2023-11-05T08:32:59.992 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00403799, 2023-11-05T08:32:59.992 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:32:59.992 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:32:59.993 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:32:59.993 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.993 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:32:59.993 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.168454+0000", 2023-11-05T08:32:59.993 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.994 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.994 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.994 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:32:59.994 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.168454+0000", 2023-11-05T08:32:59.994 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:32:59.995 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.995 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.995 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:32:59.995 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.168454+0000", 2023-11-05T08:32:59.995 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2000000000000001e-07 2023-11-05T08:32:59.995 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.996 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.996 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:32:59.996 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.168829+0000", 2023-11-05T08:32:59.996 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00037478899999999999 2023-11-05T08:32:59.996 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.997 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.997 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:32:59.997 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.168855+0000", 2023-11-05T08:32:59.997 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6557e-05 2023-11-05T08:32:59.997 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.998 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.998 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:32:59.998 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.168870+0000", 2023-11-05T08:32:59.998 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4365000000000001e-05 2023-11-05T08:32:59.998 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:32:59.999 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:32:59.999 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:32:59.999 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.169014+0000", 2023-11-05T08:32:59.999 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014451000000000001 2023-11-05T08:32:59.999 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.000 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.000 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:00.000 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.169028+0000", 2023-11-05T08:33:00.000 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3671e-05 2023-11-05T08:33:00.000 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.001 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.001 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:00.001 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.172412+0000", 2023-11-05T08:33:00.001 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0033835229999999998 2023-11-05T08:33:00.001 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.002 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.002 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:00.002 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.172492+0000", 2023-11-05T08:33:00.002 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.0254999999999996e-05 2023-11-05T08:33:00.002 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.003 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:00.003 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.003 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.003 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.003 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as3 40/33 ECSubWrite(tid=31, reqid=client.4691.0:284, at_version=40'9, trim_to=0'0, roll_forward_to=39'6))", 2023-11-05T08:33:00.004 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:53.173681+0000", 2023-11-05T08:33:00.004 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 6.7658318929999997, 2023-11-05T08:33:00.004 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002892379, 2023-11-05T08:33:00.004 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:00.004 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:00.004 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:00.005 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.005 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:00.005 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.173681+0000", 2023-11-05T08:33:00.005 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.005 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.006 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.006 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:00.006 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.173681+0000", 2023-11-05T08:33:00.006 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.006 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.006 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.007 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:00.007 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.173682+0000", 2023-11-05T08:33:00.007 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.795e-06 2023-11-05T08:33:00.007 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.007 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.008 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:00.008 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.173792+0000", 2023-11-05T08:33:00.008 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00011013100000000001 2023-11-05T08:33:00.008 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.008 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.009 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:00.009 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.173824+0000", 2023-11-05T08:33:00.009 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1783999999999999e-05 2023-11-05T08:33:00.009 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.009 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.010 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:00.010 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.173839+0000", 2023-11-05T08:33:00.010 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4871e-05 2023-11-05T08:33:00.010 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.010 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.011 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:00.011 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.173981+0000", 2023-11-05T08:33:00.011 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000141915 2023-11-05T08:33:00.011 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.011 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.012 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:00.012 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.174008+0000", 2023-11-05T08:33:00.012 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7461999999999999e-05 2023-11-05T08:33:00.012 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.012 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.012 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:00.013 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.176491+0000", 2023-11-05T08:33:00.013 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0024823200000000001 2023-11-05T08:33:00.013 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.013 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.013 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:00.014 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.176573+0000", 2023-11-05T08:33:00.014 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.2101000000000006e-05 2023-11-05T08:33:00.014 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.014 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:00.015 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.015 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.015 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.015 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es1 41/30 ECSubWrite(tid=40, reqid=osd.2.0:40, at_version=41'27, trim_to=0'0, roll_forward_to=40'25))", 2023-11-05T08:33:00.015 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.124852+0000", 2023-11-05T08:33:00.015 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 5.8146606670000001, 2023-11-05T08:33:00.016 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0054426939999999997, 2023-11-05T08:33:00.016 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:00.016 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:00.016 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:00.016 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.017 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:00.017 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.124852+0000", 2023-11-05T08:33:00.017 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.017 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.017 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.018 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:00.018 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.124852+0000", 2023-11-05T08:33:00.018 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.018 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.018 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.019 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:00.019 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.124852+0000", 2023-11-05T08:33:00.019 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0999999999999998e-07 2023-11-05T08:33:00.019 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.019 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.020 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:00.020 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.124913+0000", 2023-11-05T08:33:00.020 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0405000000000002e-05 2023-11-05T08:33:00.020 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.020 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.021 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:00.021 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.124945+0000", 2023-11-05T08:33:00.021 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2261999999999997e-05 2023-11-05T08:33:00.021 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.021 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.022 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:00.022 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.124969+0000", 2023-11-05T08:33:00.022 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4263000000000001e-05 2023-11-05T08:33:00.022 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.022 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.022 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:00.023 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.125364+0000", 2023-11-05T08:33:00.023 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00039512199999999998 2023-11-05T08:33:00.023 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.023 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.023 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:00.024 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.125393+0000", 2023-11-05T08:33:00.024 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8969e-05 2023-11-05T08:33:00.024 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.024 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.024 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:00.025 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130209+0000", 2023-11-05T08:33:00.025 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0048159149999999996 2023-11-05T08:33:00.025 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.025 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.025 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:00.026 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130294+0000", 2023-11-05T08:33:00.026 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.5148000000000005e-05 2023-11-05T08:33:00.026 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.026 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:00.026 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.027 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.027 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.027 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as3 41/33 ECSubWrite(tid=35, reqid=osd.7.0:35, at_version=41'12, trim_to=0'0, roll_forward_to=40'10))", 2023-11-05T08:33:00.027 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.129446+0000", 2023-11-05T08:33:00.027 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 5.8100666829999996, 2023-11-05T08:33:00.028 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010453017, 2023-11-05T08:33:00.028 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:00.028 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:00.028 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:00.028 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.029 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:00.029 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.129446+0000", 2023-11-05T08:33:00.029 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.029 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.029 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.030 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:00.030 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.129446+0000", 2023-11-05T08:33:00.030 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.030 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.030 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.031 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:00.031 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.129446+0000", 2023-11-05T08:33:00.031 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.5000000000000002e-07 2023-11-05T08:33:00.031 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.031 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.032 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:00.032 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.129503+0000", 2023-11-05T08:33:00.032 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6286000000000003e-05 2023-11-05T08:33:00.032 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.032 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.033 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:00.033 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.129536+0000", 2023-11-05T08:33:00.033 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3036e-05 2023-11-05T08:33:00.033 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.033 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.033 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:00.034 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.129559+0000", 2023-11-05T08:33:00.034 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3091999999999998e-05 2023-11-05T08:33:00.034 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.034 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.034 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:00.035 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.129756+0000", 2023-11-05T08:33:00.035 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000196918 2023-11-05T08:33:00.035 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.035 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.035 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:00.036 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.129782+0000", 2023-11-05T08:33:00.036 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6013e-05 2023-11-05T08:33:00.036 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.036 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.036 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:00.037 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.139822+0000", 2023-11-05T08:33:00.037 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010040261 2023-11-05T08:33:00.037 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.037 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.037 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:00.038 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.139899+0000", 2023-11-05T08:33:00.038 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.6661000000000003e-05 2023-11-05T08:33:00.038 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.038 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:00.038 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.039 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.039 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.039 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as3 41/33 ECSubWrite(tid=36, reqid=osd.7.0:36, at_version=41'14, trim_to=0'0, roll_forward_to=40'10))", 2023-11-05T08:33:00.039 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.131534+0000", 2023-11-05T08:33:00.039 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 5.8079782590000004, 2023-11-05T08:33:00.039 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.016827443000000001, 2023-11-05T08:33:00.040 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:00.040 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:00.040 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:00.040 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.040 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:00.041 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.131534+0000", 2023-11-05T08:33:00.041 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.041 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.041 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.041 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:00.042 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.131534+0000", 2023-11-05T08:33:00.042 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.042 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.042 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.042 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:00.043 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.131536+0000", 2023-11-05T08:33:00.043 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3769999999999999e-06 2023-11-05T08:33:00.043 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.043 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.043 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:00.044 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.131586+0000", 2023-11-05T08:33:00.044 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.0778000000000001e-05 2023-11-05T08:33:00.044 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.044 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.044 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:00.045 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.131619+0000", 2023-11-05T08:33:00.045 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2793000000000003e-05 2023-11-05T08:33:00.045 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.045 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.045 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:00.046 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.131642+0000", 2023-11-05T08:33:00.046 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3241999999999999e-05 2023-11-05T08:33:00.046 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.046 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.046 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:00.047 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.131806+0000", 2023-11-05T08:33:00.047 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000163843 2023-11-05T08:33:00.047 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.047 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.047 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:00.047 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.131832+0000", 2023-11-05T08:33:00.048 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5689000000000001e-05 2023-11-05T08:33:00.048 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.048 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.048 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:00.048 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148268+0000", 2023-11-05T08:33:00.049 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.016435924000000001 2023-11-05T08:33:00.049 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.049 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.049 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:00.049 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148362+0000", 2023-11-05T08:33:00.050 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.3796999999999997e-05 2023-11-05T08:33:00.050 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.050 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:00.050 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.050 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.051 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.051 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es1 41/30 ECSubWrite(tid=41, reqid=osd.2.0:40, at_version=0'0, trim_to=0'0, roll_forward_to=41'27))", 2023-11-05T08:33:00.051 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.148780+0000", 2023-11-05T08:33:00.051 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 5.7907326540000001, 2023-11-05T08:33:00.051 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0090092869999999995, 2023-11-05T08:33:00.051 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:00.052 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:00.052 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:00.052 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.052 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:00.052 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148780+0000", 2023-11-05T08:33:00.052 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.053 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.053 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.053 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:00.053 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148780+0000", 2023-11-05T08:33:00.053 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.053 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.054 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.054 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:00.054 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148789+0000", 2023-11-05T08:33:00.054 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.2029999999999998e-06 2023-11-05T08:33:00.054 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.055 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.055 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:00.055 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148816+0000", 2023-11-05T08:33:00.055 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7291999999999999e-05 2023-11-05T08:33:00.055 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.055 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.056 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:00.056 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148847+0000", 2023-11-05T08:33:00.056 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1118999999999998e-05 2023-11-05T08:33:00.056 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.056 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.056 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:00.057 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148871+0000", 2023-11-05T08:33:00.057 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3448999999999999e-05 2023-11-05T08:33:00.057 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.057 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.057 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:00.058 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.149090+0000", 2023-11-05T08:33:00.058 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00021928799999999999 2023-11-05T08:33:00.058 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.058 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.058 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:00.059 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.149116+0000", 2023-11-05T08:33:00.059 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6239000000000001e-05 2023-11-05T08:33:00.059 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.059 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.059 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:00.060 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.157693+0000", 2023-11-05T08:33:00.060 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0085768340000000002 2023-11-05T08:33:00.060 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.060 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.060 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:00.060 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.157789+0000", 2023-11-05T08:33:00.061 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.5863e-05 2023-11-05T08:33:00.061 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.061 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:00.061 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.061 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.062 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.062 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as3 41/33 ECSubWrite(tid=37, reqid=osd.7.0:36, at_version=0'0, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:33:00.062 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.167660+0000", 2023-11-05T08:33:00.062 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 5.7718523729999998, 2023-11-05T08:33:00.062 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0022408739999999999, 2023-11-05T08:33:00.063 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:00.063 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:00.063 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:00.063 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.063 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:00.064 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167660+0000", 2023-11-05T08:33:00.064 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.064 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.064 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.064 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:00.064 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167660+0000", 2023-11-05T08:33:00.065 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.065 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.065 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.065 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:00.065 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167662+0000", 2023-11-05T08:33:00.066 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7e-06 2023-11-05T08:33:00.066 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.066 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.066 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:00.067 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167689+0000", 2023-11-05T08:33:00.067 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7056999999999999e-05 2023-11-05T08:33:00.067 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.067 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.067 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:00.067 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167725+0000", 2023-11-05T08:33:00.068 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.6510000000000001e-05 2023-11-05T08:33:00.068 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.068 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.068 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:00.068 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167750+0000", 2023-11-05T08:33:00.069 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4719000000000002e-05 2023-11-05T08:33:00.069 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.069 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.069 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:00.069 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167913+0000", 2023-11-05T08:33:00.070 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000163206 2023-11-05T08:33:00.070 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.070 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.070 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:00.070 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167938+0000", 2023-11-05T08:33:00.071 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4739000000000001e-05 2023-11-05T08:33:00.071 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.071 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.071 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:00.071 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.169804+0000", 2023-11-05T08:33:00.072 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001866038 2023-11-05T08:33:00.072 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.072 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.072 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:00.072 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.169901+0000", 2023-11-05T08:33:00.073 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6904999999999995e-05 2023-11-05T08:33:00.073 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.073 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:00.073 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.073 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.073 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.074 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.11s2 41/40 ECSubWrite(tid=65, reqid=client.4691.0:287, at_version=41'13, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:00.074 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.173568+0000", 2023-11-05T08:33:00.074 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 5.7659445570000001, 2023-11-05T08:33:00.074 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0078777859999999995, 2023-11-05T08:33:00.074 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:00.075 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:00.075 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:00.075 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.076 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:00.076 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.173568+0000", 2023-11-05T08:33:00.076 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.076 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.076 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.077 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:00.077 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.173568+0000", 2023-11-05T08:33:00.077 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.077 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.077 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.078 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:00.078 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.173570+0000", 2023-11-05T08:33:00.078 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9460000000000001e-06 2023-11-05T08:33:00.078 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.078 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.079 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:00.079 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.174495+0000", 2023-11-05T08:33:00.079 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00092541300000000004 2023-11-05T08:33:00.079 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.079 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.079 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:00.080 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.174540+0000", 2023-11-05T08:33:00.080 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5170999999999998e-05 2023-11-05T08:33:00.080 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.080 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.080 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:00.080 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.174564+0000", 2023-11-05T08:33:00.081 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3159e-05 2023-11-05T08:33:00.081 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.081 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.081 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:00.081 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.175041+0000", 2023-11-05T08:33:00.082 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00047766399999999999 2023-11-05T08:33:00.082 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.082 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.082 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:00.082 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.175062+0000", 2023-11-05T08:33:00.083 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1064999999999999e-05 2023-11-05T08:33:00.083 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.083 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.083 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:00.083 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.181334+0000", 2023-11-05T08:33:00.084 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0062722159999999997 2023-11-05T08:33:00.084 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.084 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.084 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:00.084 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.181446+0000", 2023-11-05T08:33:00.085 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000111152 2023-11-05T08:33:00.085 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.085 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:00.085 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.085 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.085 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.086 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.11s2 41/40 ECSubWrite(tid=66, reqid=client.4691.0:288, at_version=41'14, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:00.086 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.177422+0000", 2023-11-05T08:33:00.086 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 5.7620902379999999, 2023-11-05T08:33:00.086 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0054156949999999999, 2023-11-05T08:33:00.086 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:00.087 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:00.087 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:00.087 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.087 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:00.087 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.177422+0000", 2023-11-05T08:33:00.088 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.088 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.088 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.088 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:00.088 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.177422+0000", 2023-11-05T08:33:00.088 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.089 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.089 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.089 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:00.089 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.177473+0000", 2023-11-05T08:33:00.089 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.0621999999999997e-05 2023-11-05T08:33:00.089 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.090 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.090 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:00.090 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.177617+0000", 2023-11-05T08:33:00.090 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014379799999999999 2023-11-05T08:33:00.090 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.091 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.091 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:00.091 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.177647+0000", 2023-11-05T08:33:00.091 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0340000000000001e-05 2023-11-05T08:33:00.091 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.091 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.092 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:00.092 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.177667+0000", 2023-11-05T08:33:00.092 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.0407999999999998e-05 2023-11-05T08:33:00.092 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.092 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.093 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:00.093 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.177812+0000", 2023-11-05T08:33:00.093 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014462399999999999 2023-11-05T08:33:00.093 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.093 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.094 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:00.094 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.177831+0000", 2023-11-05T08:33:00.094 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9069e-05 2023-11-05T08:33:00.094 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.094 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.094 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:00.095 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.182773+0000", 2023-11-05T08:33:00.095 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0049419210000000002 2023-11-05T08:33:00.095 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.095 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.095 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:00.096 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.182838+0000", 2023-11-05T08:33:00.096 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4912999999999993e-05 2023-11-05T08:33:00.096 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.096 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:00.096 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.096 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.097 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.097 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.11s2 41/40 ECSubWrite(tid=67, reqid=client.4691.0:289, at_version=41'15, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:00.097 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.179735+0000", 2023-11-05T08:33:00.097 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 5.7597773190000003, 2023-11-05T08:33:00.097 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00376436, 2023-11-05T08:33:00.098 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:00.098 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:00.098 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:00.098 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.098 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:00.099 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.179735+0000", 2023-11-05T08:33:00.099 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.099 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.099 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.099 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:00.100 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.179735+0000", 2023-11-05T08:33:00.100 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.100 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.100 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.100 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:00.100 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.179737+0000", 2023-11-05T08:33:00.101 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6470000000000001e-06 2023-11-05T08:33:00.101 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.101 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.101 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:00.101 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.179775+0000", 2023-11-05T08:33:00.102 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8012000000000001e-05 2023-11-05T08:33:00.102 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.102 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.102 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:00.103 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.179802+0000", 2023-11-05T08:33:00.103 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7081e-05 2023-11-05T08:33:00.103 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.103 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.103 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:00.104 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.179820+0000", 2023-11-05T08:33:00.104 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.8465e-05 2023-11-05T08:33:00.104 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.104 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.104 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:00.105 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.179993+0000", 2023-11-05T08:33:00.105 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00017233299999999999 2023-11-05T08:33:00.105 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.105 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.105 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:00.106 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.180012+0000", 2023-11-05T08:33:00.106 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9541000000000001e-05 2023-11-05T08:33:00.106 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.106 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.106 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:00.107 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.183422+0000", 2023-11-05T08:33:00.107 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.003409937 2023-11-05T08:33:00.107 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.107 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.107 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:00.108 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.183499+0000", 2023-11-05T08:33:00.108 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.7343999999999997e-05 2023-11-05T08:33:00.108 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.108 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:00.108 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.108 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.109 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.109 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as3 42/33 ECSubWrite(tid=38, reqid=osd.7.0:38, at_version=42'16, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:33:00.109 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:55.126927+0000", 2023-11-05T08:33:00.109 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4.812585672, 2023-11-05T08:33:00.109 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.013340393000000001, 2023-11-05T08:33:00.110 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:00.110 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:00.110 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:00.110 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.110 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:00.110 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.126927+0000", 2023-11-05T08:33:00.111 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.111 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.111 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.111 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:00.111 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.126927+0000", 2023-11-05T08:33:00.112 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.112 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.112 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.112 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:00.112 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.126927+0000", 2023-11-05T08:33:00.113 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7e-07 2023-11-05T08:33:00.113 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.113 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.113 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:00.113 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.127040+0000", 2023-11-05T08:33:00.113 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00011244299999999999 2023-11-05T08:33:00.114 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.114 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.114 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:00.114 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.127069+0000", 2023-11-05T08:33:00.114 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9255999999999999e-05 2023-11-05T08:33:00.115 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.115 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.115 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:00.115 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.127093+0000", 2023-11-05T08:33:00.115 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3839e-05 2023-11-05T08:33:00.116 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.116 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.116 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:00.116 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.129369+0000", 2023-11-05T08:33:00.116 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0022765250000000002 2023-11-05T08:33:00.117 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.117 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.117 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:00.117 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.129395+0000", 2023-11-05T08:33:00.117 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5755999999999999e-05 2023-11-05T08:33:00.117 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.118 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.118 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:00.118 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.140178+0000", 2023-11-05T08:33:00.121 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010782523 2023-11-05T08:33:00.121 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.122 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.122 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:00.122 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.140267+0000", 2023-11-05T08:33:00.122 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9580999999999995e-05 2023-11-05T08:33:00.122 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.123 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:00.123 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.123 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.123 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.123 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as3 42/33 ECSubWrite(tid=39, reqid=osd.7.0:39, at_version=42'18, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:33:00.123 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:55.128574+0000", 2023-11-05T08:33:00.124 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4.8109384400000001, 2023-11-05T08:33:00.124 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.013189111, 2023-11-05T08:33:00.124 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:00.124 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:00.124 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:00.125 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.125 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:00.125 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.128574+0000", 2023-11-05T08:33:00.125 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.125 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.126 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.126 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:00.126 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.128574+0000", 2023-11-05T08:33:00.126 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.127 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.127 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.127 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:00.127 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.128575+0000", 2023-11-05T08:33:00.128 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1370000000000001e-06 2023-11-05T08:33:00.128 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.128 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.128 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:00.128 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.128665+0000", 2023-11-05T08:33:00.129 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9571e-05 2023-11-05T08:33:00.129 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.129 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.129 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:00.129 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.128693+0000", 2023-11-05T08:33:00.130 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8822e-05 2023-11-05T08:33:00.130 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.130 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.130 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:00.130 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.128737+0000", 2023-11-05T08:33:00.131 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3469e-05 2023-11-05T08:33:00.131 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.131 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.131 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:00.131 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.130720+0000", 2023-11-05T08:33:00.132 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0019833229999999999 2023-11-05T08:33:00.132 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.132 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.132 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:00.132 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.130746+0000", 2023-11-05T08:33:00.132 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5326000000000001e-05 2023-11-05T08:33:00.133 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.133 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.133 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:00.133 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.141675+0000", 2023-11-05T08:33:00.133 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010929589 2023-11-05T08:33:00.134 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.134 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.134 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:00.134 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.141763+0000", 2023-11-05T08:33:00.134 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.7874000000000006e-05 2023-11-05T08:33:00.135 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.135 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:00.135 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.135 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.135 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.136 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es1 42/30 ECSubWrite(tid=43, reqid=osd.2.0:43, at_version=42'29, trim_to=0'0, roll_forward_to=41'27))", 2023-11-05T08:33:00.136 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:55.134325+0000", 2023-11-05T08:33:00.136 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4.8051872380000002, 2023-11-05T08:33:00.136 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.008930136, 2023-11-05T08:33:00.136 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:00.137 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:00.137 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:00.137 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.137 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:00.137 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134325+0000", 2023-11-05T08:33:00.138 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.138 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.138 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.138 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:00.138 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134325+0000", 2023-11-05T08:33:00.139 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.139 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.139 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.139 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:00.139 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134327+0000", 2023-11-05T08:33:00.139 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6899999999999999e-06 2023-11-05T08:33:00.140 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.140 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.140 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:00.140 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134380+0000", 2023-11-05T08:33:00.140 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3458000000000002e-05 2023-11-05T08:33:00.141 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.141 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.141 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:00.141 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134410+0000", 2023-11-05T08:33:00.141 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9405999999999999e-05 2023-11-05T08:33:00.142 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.142 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.142 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:00.142 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134433+0000", 2023-11-05T08:33:00.142 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3745999999999999e-05 2023-11-05T08:33:00.143 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.143 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.143 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:00.143 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134900+0000", 2023-11-05T08:33:00.143 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00046645999999999998 2023-11-05T08:33:00.144 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.144 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.144 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:00.144 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134925+0000", 2023-11-05T08:33:00.145 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5196000000000001e-05 2023-11-05T08:33:00.145 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.145 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.145 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:00.146 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.143157+0000", 2023-11-05T08:33:00.146 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0082321760000000008 2023-11-05T08:33:00.146 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.146 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.146 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:00.147 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.143255+0000", 2023-11-05T08:33:00.147 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.8004000000000006e-05 2023-11-05T08:33:00.147 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.147 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:00.147 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.147 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.148 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.148 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as3 42/33 ECSubWrite(tid=40, reqid=osd.7.0:39, at_version=0'0, trim_to=0'0, roll_forward_to=42'18))", 2023-11-05T08:33:00.148 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:55.151411+0000", 2023-11-05T08:33:00.148 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4.7881017689999998, 2023-11-05T08:33:00.148 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0042408790000000003, 2023-11-05T08:33:00.148 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:00.149 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:00.149 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:00.149 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.149 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:00.149 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.151411+0000", 2023-11-05T08:33:00.149 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.150 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.150 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.150 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:00.150 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.151411+0000", 2023-11-05T08:33:00.150 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.151 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.151 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.151 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:00.151 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.151413+0000", 2023-11-05T08:33:00.151 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9199999999999998e-06 2023-11-05T08:33:00.152 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.152 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.152 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:00.152 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.151450+0000", 2023-11-05T08:33:00.152 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.7815000000000003e-05 2023-11-05T08:33:00.152 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.153 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.153 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:00.153 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.151496+0000", 2023-11-05T08:33:00.153 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.6078999999999997e-05 2023-11-05T08:33:00.153 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.154 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.154 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:00.154 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.151526+0000", 2023-11-05T08:33:00.154 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9836e-05 2023-11-05T08:33:00.154 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.155 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.155 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:00.155 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.151669+0000", 2023-11-05T08:33:00.155 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014300900000000001 2023-11-05T08:33:00.155 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.156 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.156 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:00.156 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.151697+0000", 2023-11-05T08:33:00.156 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8183000000000001e-05 2023-11-05T08:33:00.156 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.156 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.157 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:00.157 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.155569+0000", 2023-11-05T08:33:00.157 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0038712619999999999 2023-11-05T08:33:00.157 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.157 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.158 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:00.158 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.155652+0000", 2023-11-05T08:33:00.158 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.2775000000000006e-05 2023-11-05T08:33:00.158 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.158 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:00.159 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.159 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.159 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.159 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es1 42/30 ECSubWrite(tid=44, reqid=osd.2.0:43, at_version=0'0, trim_to=0'0, roll_forward_to=42'29))", 2023-11-05T08:33:00.159 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:55.158018+0000", 2023-11-05T08:33:00.160 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4.7814942340000002, 2023-11-05T08:33:00.160 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0025501270000000001, 2023-11-05T08:33:00.160 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:00.160 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:00.160 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:00.161 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.161 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:00.161 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.158018+0000", 2023-11-05T08:33:00.161 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.161 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.161 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.162 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:00.162 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.158018+0000", 2023-11-05T08:33:00.162 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.162 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.162 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.163 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:00.163 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.158019+0000", 2023-11-05T08:33:00.163 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.1200000000000002e-07 2023-11-05T08:33:00.163 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.163 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.164 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:00.164 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.158039+0000", 2023-11-05T08:33:00.164 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.0479000000000002e-05 2023-11-05T08:33:00.164 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.164 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.165 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:00.165 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.158062+0000", 2023-11-05T08:33:00.165 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.2308000000000001e-05 2023-11-05T08:33:00.165 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.165 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.166 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:00.166 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.158080+0000", 2023-11-05T08:33:00.166 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.8414e-05 2023-11-05T08:33:00.166 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.166 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.167 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:00.167 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.158341+0000", 2023-11-05T08:33:00.167 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00026081800000000001 2023-11-05T08:33:00.167 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.167 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.167 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:00.168 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.158363+0000", 2023-11-05T08:33:00.168 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1834000000000001e-05 2023-11-05T08:33:00.168 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.168 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.168 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:00.169 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.160499+0000", 2023-11-05T08:33:00.169 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002135999 2023-11-05T08:33:00.169 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.169 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.169 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:00.169 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.160568+0000", 2023-11-05T08:33:00.170 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9462999999999995e-05 2023-11-05T08:33:00.170 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.170 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:00.170 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.170 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.171 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.171 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.16s4 46/45 ECSubWrite(tid=47, reqid=osd.11.0:47, at_version=46'13, trim_to=0'0, roll_forward_to=46'11))", 2023-11-05T08:33:00.171 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:59.233671+0000", 2023-11-05T08:33:00.171 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.70584160500000004, 2023-11-05T08:33:00.171 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0022166759999999999, 2023-11-05T08:33:00.172 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:00.172 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:00.172 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:00.172 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.172 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:00.173 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:59.233671+0000", 2023-11-05T08:33:00.173 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.173 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.173 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.173 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:00.174 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:59.233671+0000", 2023-11-05T08:33:00.174 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.174 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.174 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.174 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:00.175 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:59.233672+0000", 2023-11-05T08:33:00.175 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6139999999999999e-06 2023-11-05T08:33:00.175 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.175 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.175 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:00.176 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:59.233730+0000", 2023-11-05T08:33:00.176 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7447000000000001e-05 2023-11-05T08:33:00.176 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.176 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.176 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:00.176 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:59.233769+0000", 2023-11-05T08:33:00.177 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.9613999999999997e-05 2023-11-05T08:33:00.177 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.177 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.177 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:00.177 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:59.233792+0000", 2023-11-05T08:33:00.178 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.2642000000000001e-05 2023-11-05T08:33:00.178 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.178 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.178 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:00.178 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:59.234017+0000", 2023-11-05T08:33:00.179 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00022518700000000001 2023-11-05T08:33:00.179 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.179 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.179 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:00.179 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:59.234036+0000", 2023-11-05T08:33:00.180 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.8719000000000002e-05 2023-11-05T08:33:00.180 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.180 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.180 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:00.180 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:59.235852+0000", 2023-11-05T08:33:00.180 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0018159439999999999 2023-11-05T08:33:00.181 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.181 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.181 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:00.181 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:59.235887+0000", 2023-11-05T08:33:00.181 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5509000000000002e-05 2023-11-05T08:33:00.182 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.182 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:00.182 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.182 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.182 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:00.183 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:33:00.184 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2023-11-05T08:33:00.185 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:33:00.181+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:33:00.223 INFO:teuthology.orchestra.run.smithi090.stderr:Exporting 2.2 info 2.2( empty local-lis/les=33/34 n=0 ec=25/25 lis/c=33/33 les/c/f=34/34/0 sis=33) 2023-11-05T08:33:00.228 INFO:teuthology.orchestra.run.smithi090.stderr:Export successful 2023-11-05T08:33:00.231 INFO:tasks.rados.rados.0.smithi090.stdout:119: done (1 left) 2023-11-05T08:33:00.231 INFO:tasks.rados.rados.0.smithi090.stdout:120: snap_create 2023-11-05T08:33:00.284 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:00.281+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:00.331 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:33:00.332 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2023-11-05T08:33:00.332 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 0 2023-11-05T08:33:00.332 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:33:00.345 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2023-11-05T08:33:00.386 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:33:00.383+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:33:00.440 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-05T08:33:00.487 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:33:00.487+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:33:00.514 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:33:00.515 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2023-11-05T08:33:00.515 INFO:teuthology.orchestra.run.smithi053.stdout: "complaint_time": 30, 2023-11-05T08:33:00.515 INFO:teuthology.orchestra.run.smithi053.stdout: "num_blocked_ops": 0 2023-11-05T08:33:00.515 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:33:00.530 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2023-11-05T08:33:00.578 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-11-05T08:33:00.588 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:33:00.584+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:33:00.598 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1: {} 2023-11-05T08:33:00.598 INFO:teuthology.orchestra.run.smithi002.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-11-05T08:33:00.618 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2: {} 2023-11-05T08:33:00.619 INFO:teuthology.orchestra.run.smithi002.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-11-05T08:33:00.619 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-11-05T08:33:00.636 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4: {} 2023-11-05T08:33:00.637 INFO:teuthology.orchestra.run.smithi002.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-11-05T08:33:00.656 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:33:00.656 INFO:teuthology.orchestra.run.smithi053.stdout: "size": 20, 2023-11-05T08:33:00.657 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 600, 2023-11-05T08:33:00.657 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2023-11-05T08:33:00.657 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.657 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:00.657 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:28.710073+0000", 2023-11-05T08:33:00.658 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 31.944273252999999, 2023-11-05T08:33:00.658 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.052834791999999998, 2023-11-05T08:33:00.658 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:00.658 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:00.658 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:00.658 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:00.659 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:00.659 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 1 2023-11-05T08:33:00.659 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.659 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:00.659 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.659 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:00.660 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710073+0000", 2023-11-05T08:33:00.660 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.660 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.660 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.660 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:00.661 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710073+0000", 2023-11-05T08:33:00.661 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.661 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.661 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.661 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:00.662 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710076+0000", 2023-11-05T08:33:00.662 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1729999999999999e-06 2023-11-05T08:33:00.662 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.662 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.662 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:00.663 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710108+0000", 2023-11-05T08:33:00.663 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1458000000000003e-05 2023-11-05T08:33:00.663 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.663 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.663 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:00.663 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710140+0000", 2023-11-05T08:33:00.664 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2532e-05 2023-11-05T08:33:00.664 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.664 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.664 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:00.664 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710165+0000", 2023-11-05T08:33:00.664 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5205e-05 2023-11-05T08:33:00.665 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.665 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.665 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:00.665 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710311+0000", 2023-11-05T08:33:00.665 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014532799999999999 2023-11-05T08:33:00.666 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.666 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.666 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2023-11-05T08:33:00.666 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710327+0000", 2023-11-05T08:33:00.666 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6059e-05 2023-11-05T08:33:00.667 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.667 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.667 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:00.667 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.759348+0000", 2023-11-05T08:33:00.667 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.049021481999999998 2023-11-05T08:33:00.667 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.668 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.668 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:00.668 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.759880+0000", 2023-11-05T08:33:00.668 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00053197400000000005 2023-11-05T08:33:00.668 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.669 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.669 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:00.669 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.760478+0000", 2023-11-05T08:33:00.669 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00059839300000000002 2023-11-05T08:33:00.669 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.670 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.670 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:00.670 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.761955+0000", 2023-11-05T08:33:00.670 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001476991 2023-11-05T08:33:00.671 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.671 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.671 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:00.671 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.762611+0000", 2023-11-05T08:33:00.671 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00065598700000000002 2023-11-05T08:33:00.671 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.672 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.672 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:00.672 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.762782+0000", 2023-11-05T08:33:00.672 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00017064000000000001 2023-11-05T08:33:00.672 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.673 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.673 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:00.673 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.762908+0000", 2023-11-05T08:33:00.673 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012557000000000001 2023-11-05T08:33:00.673 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.674 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:00.674 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.674 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.674 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.674 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:00.675 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.151546+0000", 2023-11-05T08:33:00.675 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 31.502799905, 2023-11-05T08:33:00.675 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.018837386000000001, 2023-11-05T08:33:00.675 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:00.675 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:00.676 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:00.676 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:00.676 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:00.676 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 113 2023-11-05T08:33:00.676 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.677 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:00.677 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.677 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:00.677 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151546+0000", 2023-11-05T08:33:00.677 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.677 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.678 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.678 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:00.678 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151546+0000", 2023-11-05T08:33:00.678 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.679 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.679 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.679 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:00.679 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151547+0000", 2023-11-05T08:33:00.679 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9699999999999995e-07 2023-11-05T08:33:00.680 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.680 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.680 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:00.680 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151553+0000", 2023-11-05T08:33:00.680 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9830000000000001e-06 2023-11-05T08:33:00.681 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.681 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.681 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:00.681 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151562+0000", 2023-11-05T08:33:00.681 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.5060000000000001e-06 2023-11-05T08:33:00.682 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.682 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.682 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:00.682 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151569+0000", 2023-11-05T08:33:00.682 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9160000000000004e-06 2023-11-05T08:33:00.683 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.683 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.683 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:00.683 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.163944+0000", 2023-11-05T08:33:00.683 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012374703000000001 2023-11-05T08:33:00.684 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.684 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.684 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:00.684 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.164091+0000", 2023-11-05T08:33:00.684 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014739199999999999 2023-11-05T08:33:00.685 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.685 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.685 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:00.685 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.164295+0000", 2023-11-05T08:33:00.685 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00020334899999999999 2023-11-05T08:33:00.686 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.686 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.686 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:00.686 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.167412+0000", 2023-11-05T08:33:00.686 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.003117529 2023-11-05T08:33:00.686 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.687 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.687 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:00.687 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.170241+0000", 2023-11-05T08:33:00.687 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0028283890000000002 2023-11-05T08:33:00.687 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.688 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.688 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:00.688 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.170326+0000", 2023-11-05T08:33:00.688 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.5581999999999995e-05 2023-11-05T08:33:00.688 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.689 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.689 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:00.689 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.170384+0000", 2023-11-05T08:33:00.689 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7340000000000003e-05 2023-11-05T08:33:00.689 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.689 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:00.690 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.690 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.690 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.690 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:00.690 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.151583+0000", 2023-11-05T08:33:00.690 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 31.502763460000001, 2023-11-05T08:33:00.691 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.020286660000000001, 2023-11-05T08:33:00.691 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:00.691 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:00.691 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:00.691 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:00.692 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:00.692 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 114 2023-11-05T08:33:00.692 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.692 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:00.692 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.692 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:00.693 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151583+0000", 2023-11-05T08:33:00.693 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.693 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.693 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.693 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:00.694 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151583+0000", 2023-11-05T08:33:00.694 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.694 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.694 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.694 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:00.695 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151583+0000", 2023-11-05T08:33:00.695 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.2600000000000002e-07 2023-11-05T08:33:00.695 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.695 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.695 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:00.695 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151589+0000", 2023-11-05T08:33:00.696 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5099999999999998e-06 2023-11-05T08:33:00.696 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.696 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.696 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:00.696 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151598+0000", 2023-11-05T08:33:00.697 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1730000000000001e-06 2023-11-05T08:33:00.697 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.697 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.697 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:00.697 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151608+0000", 2023-11-05T08:33:00.697 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0059e-05 2023-11-05T08:33:00.698 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.698 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.698 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:00.698 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.165015+0000", 2023-11-05T08:33:00.698 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.013407207000000001 2023-11-05T08:33:00.699 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.699 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.699 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:00.699 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.165268+0000", 2023-11-05T08:33:00.699 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00025284999999999999 2023-11-05T08:33:00.700 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.700 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.700 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:00.700 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.165635+0000", 2023-11-05T08:33:00.700 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00036695899999999998 2023-11-05T08:33:00.701 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.701 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.701 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:00.701 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.170105+0000", 2023-11-05T08:33:00.701 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0044696960000000004 2023-11-05T08:33:00.701 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.702 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.702 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:00.702 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.171719+0000", 2023-11-05T08:33:00.702 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001613855 2023-11-05T08:33:00.703 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.703 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.703 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:00.703 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.171810+0000", 2023-11-05T08:33:00.703 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1625000000000005e-05 2023-11-05T08:33:00.704 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.704 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.704 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:00.704 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.171869+0000", 2023-11-05T08:33:00.705 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.91e-05 2023-11-05T08:33:00.705 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.705 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:00.705 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.705 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.706 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.706 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:00.706 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.151660+0000", 2023-11-05T08:33:00.706 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 31.502685661000001, 2023-11-05T08:33:00.706 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022553719, 2023-11-05T08:33:00.707 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:00.707 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:00.707 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:00.707 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:00.707 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:00.707 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 115 2023-11-05T08:33:00.708 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.708 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:00.708 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.708 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:00.708 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151660+0000", 2023-11-05T08:33:00.709 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.709 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.709 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.709 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:00.709 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151660+0000", 2023-11-05T08:33:00.710 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.710 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.710 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.710 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:00.710 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151661+0000", 2023-11-05T08:33:00.711 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.5600000000000004e-07 2023-11-05T08:33:00.711 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.711 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.711 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:00.712 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151799+0000", 2023-11-05T08:33:00.712 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000137207 2023-11-05T08:33:00.712 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.712 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.712 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:00.713 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151814+0000", 2023-11-05T08:33:00.713 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4885000000000001e-05 2023-11-05T08:33:00.713 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.713 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.713 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:00.714 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151827+0000", 2023-11-05T08:33:00.714 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3413e-05 2023-11-05T08:33:00.714 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.714 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.714 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:00.715 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.167513+0000", 2023-11-05T08:33:00.715 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.015686091999999999 2023-11-05T08:33:00.715 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.715 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.715 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:00.716 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.167652+0000", 2023-11-05T08:33:00.716 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013890500000000001 2023-11-05T08:33:00.716 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.716 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.716 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:00.716 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.167860+0000", 2023-11-05T08:33:00.717 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00020838599999999999 2023-11-05T08:33:00.717 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.717 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.717 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:00.717 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.171602+0000", 2023-11-05T08:33:00.718 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0037417449999999999 2023-11-05T08:33:00.718 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.718 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.718 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:00.718 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174068+0000", 2023-11-05T08:33:00.719 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0024658010000000001 2023-11-05T08:33:00.719 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.719 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.719 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:00.719 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174152+0000", 2023-11-05T08:33:00.719 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.3752999999999996e-05 2023-11-05T08:33:00.720 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.720 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.720 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:00.720 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174214+0000", 2023-11-05T08:33:00.720 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.2576000000000005e-05 2023-11-05T08:33:00.721 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.721 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:00.721 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.721 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.721 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.721 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:00.722 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.151849+0000", 2023-11-05T08:33:00.722 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 31.502497304999999, 2023-11-05T08:33:00.722 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022629176000000001, 2023-11-05T08:33:00.722 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:00.722 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:00.723 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:00.723 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:00.723 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:00.723 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 116 2023-11-05T08:33:00.723 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.724 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:00.724 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.724 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:00.724 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151849+0000", 2023-11-05T08:33:00.724 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.725 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.725 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.725 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:00.725 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151849+0000", 2023-11-05T08:33:00.725 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.726 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.726 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.726 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:00.726 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151850+0000", 2023-11-05T08:33:00.726 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.037e-06 2023-11-05T08:33:00.726 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.727 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.727 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:00.727 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151858+0000", 2023-11-05T08:33:00.727 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.3389999999999993e-06 2023-11-05T08:33:00.727 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.727 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.728 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:00.728 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151868+0000", 2023-11-05T08:33:00.728 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6670000000000008e-06 2023-11-05T08:33:00.728 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.728 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.728 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:00.729 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151875+0000", 2023-11-05T08:33:00.729 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9229999999999996e-06 2023-11-05T08:33:00.729 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.729 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.730 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:00.730 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.168777+0000", 2023-11-05T08:33:00.730 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.016901884999999998 2023-11-05T08:33:00.730 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.730 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.730 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:00.731 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.169009+0000", 2023-11-05T08:33:00.731 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00023236400000000001 2023-11-05T08:33:00.731 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.731 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.731 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:00.732 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.169365+0000", 2023-11-05T08:33:00.732 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00035625300000000001 2023-11-05T08:33:00.732 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.732 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.732 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:00.732 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.171624+0000", 2023-11-05T08:33:00.733 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0022587169999999999 2023-11-05T08:33:00.733 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.733 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.733 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:00.734 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174314+0000", 2023-11-05T08:33:00.734 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002690263 2023-11-05T08:33:00.734 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.734 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.734 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:00.734 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174405+0000", 2023-11-05T08:33:00.735 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1064999999999996e-05 2023-11-05T08:33:00.735 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.735 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.735 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:00.736 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174478+0000", 2023-11-05T08:33:00.736 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.2663000000000005e-05 2023-11-05T08:33:00.736 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.736 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:00.736 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.736 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.737 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.737 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:00.737 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.151886+0000", 2023-11-05T08:33:00.737 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 31.502459890000001, 2023-11-05T08:33:00.737 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.023174162000000002, 2023-11-05T08:33:00.737 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:00.738 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:00.738 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:00.738 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:00.738 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:00.738 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 117 2023-11-05T08:33:00.739 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.739 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:00.739 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.739 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:00.739 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151886+0000", 2023-11-05T08:33:00.739 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.740 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.740 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.740 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:00.740 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151886+0000", 2023-11-05T08:33:00.740 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.740 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.741 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.741 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:00.741 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151887+0000", 2023-11-05T08:33:00.741 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.2300000000000001e-07 2023-11-05T08:33:00.741 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.741 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.741 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:00.742 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151893+0000", 2023-11-05T08:33:00.742 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.19e-06 2023-11-05T08:33:00.742 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.742 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.742 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:00.742 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151905+0000", 2023-11-05T08:33:00.743 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1883e-05 2023-11-05T08:33:00.743 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.743 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.743 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:00.743 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151912+0000", 2023-11-05T08:33:00.744 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.0690000000000004e-06 2023-11-05T08:33:00.744 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.744 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.744 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:00.744 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.170509+0000", 2023-11-05T08:33:00.745 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.018596686000000001 2023-11-05T08:33:00.745 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.745 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.745 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:00.745 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.170791+0000", 2023-11-05T08:33:00.745 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00028221799999999999 2023-11-05T08:33:00.746 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.746 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.746 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:00.746 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.171137+0000", 2023-11-05T08:33:00.746 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000346454 2023-11-05T08:33:00.747 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.747 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.747 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:00.747 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.172413+0000", 2023-11-05T08:33:00.747 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0012752340000000001 2023-11-05T08:33:00.747 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.748 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.748 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:00.748 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174926+0000", 2023-11-05T08:33:00.748 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0025137050000000002 2023-11-05T08:33:00.748 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.749 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.749 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:00.749 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.175007+0000", 2023-11-05T08:33:00.749 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.0413000000000004e-05 2023-11-05T08:33:00.749 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.750 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.750 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:00.750 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.175060+0000", 2023-11-05T08:33:00.750 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3687000000000003e-05 2023-11-05T08:33:00.750 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.751 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:00.751 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.751 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.751 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.751 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:00.751 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.352803+0000", 2023-11-05T08:33:00.752 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 31.301542957999999, 2023-11-05T08:33:00.752 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017043243, 2023-11-05T08:33:00.752 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:00.752 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:00.752 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:00.753 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:00.753 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:00.753 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 140 2023-11-05T08:33:00.753 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.753 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:00.754 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.754 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:00.754 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.352803+0000", 2023-11-05T08:33:00.754 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.754 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.755 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.755 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:00.755 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.352803+0000", 2023-11-05T08:33:00.755 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.755 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.756 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.756 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:00.756 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.352804+0000", 2023-11-05T08:33:00.756 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1600000000000001e-07 2023-11-05T08:33:00.756 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.757 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.757 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:00.757 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.352932+0000", 2023-11-05T08:33:00.757 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012813399999999999 2023-11-05T08:33:00.757 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.757 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.758 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:00.758 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.352943+0000", 2023-11-05T08:33:00.758 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0601000000000001e-05 2023-11-05T08:33:00.758 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.758 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.759 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:00.759 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.352948+0000", 2023-11-05T08:33:00.759 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3029999999999999e-06 2023-11-05T08:33:00.759 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.759 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.759 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:00.760 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.362447+0000", 2023-11-05T08:33:00.760 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0094990089999999992 2023-11-05T08:33:00.760 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.760 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.760 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:00.760 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.362610+0000", 2023-11-05T08:33:00.761 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00016328600000000001 2023-11-05T08:33:00.761 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.761 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.761 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:00.761 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.362809+0000", 2023-11-05T08:33:00.761 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00019889199999999999 2023-11-05T08:33:00.762 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.762 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.762 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:00.762 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.364900+0000", 2023-11-05T08:33:00.762 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002091369 2023-11-05T08:33:00.763 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.763 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.763 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:00.763 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.369777+0000", 2023-11-05T08:33:00.763 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0048767860000000001 2023-11-05T08:33:00.764 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.764 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.764 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:00.764 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.369819+0000", 2023-11-05T08:33:00.764 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1653999999999998e-05 2023-11-05T08:33:00.764 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.765 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.765 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:00.765 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.369846+0000", 2023-11-05T08:33:00.765 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7593000000000002e-05 2023-11-05T08:33:00.765 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.765 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:00.766 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.766 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.766 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.766 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:00.766 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.352968+0000", 2023-11-05T08:33:00.767 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 31.301377935000001, 2023-11-05T08:33:00.767 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017162192, 2023-11-05T08:33:00.767 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:00.767 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:00.767 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:00.767 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:00.768 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:00.768 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 141 2023-11-05T08:33:00.768 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.768 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:00.768 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.768 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:00.769 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.352968+0000", 2023-11-05T08:33:00.769 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.769 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.769 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.769 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:00.770 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.352968+0000", 2023-11-05T08:33:00.770 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.770 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.770 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.770 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:00.771 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.352969+0000", 2023-11-05T08:33:00.771 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5200000000000002e-07 2023-11-05T08:33:00.771 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.771 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.771 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:00.771 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.352972+0000", 2023-11-05T08:33:00.772 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.7249999999999999e-06 2023-11-05T08:33:00.772 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.772 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.772 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:00.772 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.352981+0000", 2023-11-05T08:33:00.773 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9749999999999996e-06 2023-11-05T08:33:00.773 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.773 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.773 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:00.773 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.352985+0000", 2023-11-05T08:33:00.774 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.0400000000000003e-06 2023-11-05T08:33:00.774 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.774 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.774 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:00.774 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.364169+0000", 2023-11-05T08:33:00.775 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011183451000000001 2023-11-05T08:33:00.775 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.775 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.775 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:00.775 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.364285+0000", 2023-11-05T08:33:00.776 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00011637900000000001 2023-11-05T08:33:00.776 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.776 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.776 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:00.776 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.364464+0000", 2023-11-05T08:33:00.776 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00017891900000000001 2023-11-05T08:33:00.777 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.777 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.777 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:00.777 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.365555+0000", 2023-11-05T08:33:00.777 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0010906539999999999 2023-11-05T08:33:00.778 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.778 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.778 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:00.778 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.369947+0000", 2023-11-05T08:33:00.778 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.004391697 2023-11-05T08:33:00.779 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.779 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.779 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:00.779 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.370078+0000", 2023-11-05T08:33:00.779 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013115699999999999 2023-11-05T08:33:00.779 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.780 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.780 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:00.780 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.370130+0000", 2023-11-05T08:33:00.780 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2743e-05 2023-11-05T08:33:00.780 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.780 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:00.781 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.781 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.781 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.781 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:162 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:00.781 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.670856+0000", 2023-11-05T08:33:00.782 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 30.983490498999998, 2023-11-05T08:33:00.782 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.016361348000000001, 2023-11-05T08:33:00.782 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:00.782 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:00.782 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:00.783 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:00.783 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:00.783 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 162 2023-11-05T08:33:00.783 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.783 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:00.784 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.784 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:00.784 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670856+0000", 2023-11-05T08:33:00.784 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.784 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.785 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.785 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:00.785 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670856+0000", 2023-11-05T08:33:00.785 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.785 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.785 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.786 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:00.786 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670857+0000", 2023-11-05T08:33:00.786 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.8999999999999995e-07 2023-11-05T08:33:00.786 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.786 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.787 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:00.787 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670862+0000", 2023-11-05T08:33:00.787 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9030000000000001e-06 2023-11-05T08:33:00.787 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.787 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.787 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:00.788 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670872+0000", 2023-11-05T08:33:00.788 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.5659999999999996e-06 2023-11-05T08:33:00.788 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.788 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.788 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:00.789 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670879+0000", 2023-11-05T08:33:00.789 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.08e-06 2023-11-05T08:33:00.789 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.789 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.789 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:00.789 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.683898+0000", 2023-11-05T08:33:00.790 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.013018897 2023-11-05T08:33:00.790 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.790 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.790 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:00.790 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.684137+0000", 2023-11-05T08:33:00.791 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00023945699999999999 2023-11-05T08:33:00.791 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.791 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.791 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:00.791 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.684487+0000", 2023-11-05T08:33:00.791 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00034995099999999998 2023-11-05T08:33:00.792 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.792 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.792 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:00.792 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.686952+0000", 2023-11-05T08:33:00.792 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002464548 2023-11-05T08:33:00.793 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.793 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.793 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:00.793 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.687067+0000", 2023-11-05T08:33:00.793 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00011500799999999999 2023-11-05T08:33:00.794 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.794 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.794 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:00.794 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.687157+0000", 2023-11-05T08:33:00.794 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.0271000000000004e-05 2023-11-05T08:33:00.795 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.898 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.898 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:00.898 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.687217+0000", 2023-11-05T08:33:00.898 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9777e-05 2023-11-05T08:33:00.899 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.899 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:00.899 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.899 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.899 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.900 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:163 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:00.900 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.670890+0000", 2023-11-05T08:33:00.900 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 30.983455661000001, 2023-11-05T08:33:00.900 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.02132307, 2023-11-05T08:33:00.900 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:00.901 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:00.901 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:00.901 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:00.901 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:00.901 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 163 2023-11-05T08:33:00.902 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.902 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:00.902 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.902 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:00.902 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670890+0000", 2023-11-05T08:33:00.903 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.903 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.903 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.903 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:00.903 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670890+0000", 2023-11-05T08:33:00.904 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.904 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.904 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.904 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:00.904 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670891+0000", 2023-11-05T08:33:00.905 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3300000000000002e-07 2023-11-05T08:33:00.905 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.905 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.905 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:00.905 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670897+0000", 2023-11-05T08:33:00.906 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.4600000000000002e-06 2023-11-05T08:33:00.906 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.906 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.906 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:00.906 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670906+0000", 2023-11-05T08:33:00.907 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.3130000000000004e-06 2023-11-05T08:33:00.907 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.907 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.907 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:00.907 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670913+0000", 2023-11-05T08:33:00.908 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9859999999999997e-06 2023-11-05T08:33:00.908 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.908 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.908 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:00.908 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.687360+0000", 2023-11-05T08:33:00.908 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.016447244 2023-11-05T08:33:00.909 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.909 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.909 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:00.909 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.687588+0000", 2023-11-05T08:33:00.909 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00022757699999999999 2023-11-05T08:33:00.909 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.910 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.910 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:00.910 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.687946+0000", 2023-11-05T08:33:00.910 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00035789399999999999 2023-11-05T08:33:00.910 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.910 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.911 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:00.911 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.691929+0000", 2023-11-05T08:33:00.911 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0039835690000000002 2023-11-05T08:33:00.911 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.911 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.911 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:00.912 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.691970+0000", 2023-11-05T08:33:00.912 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.0351e-05 2023-11-05T08:33:00.912 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.912 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.912 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:00.913 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692111+0000", 2023-11-05T08:33:00.913 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014114499999999999 2023-11-05T08:33:00.913 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.913 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.913 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:00.914 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692214+0000", 2023-11-05T08:33:00.914 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000102898 2023-11-05T08:33:00.914 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.914 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:00.914 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.915 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.915 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.915 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:00.915 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.670924+0000", 2023-11-05T08:33:00.915 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 30.983422253000001, 2023-11-05T08:33:00.915 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.021621676999999999, 2023-11-05T08:33:00.916 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:00.916 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:00.916 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:00.916 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:00.916 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:00.917 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 164 2023-11-05T08:33:00.917 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.917 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:00.917 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.917 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:00.918 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670924+0000", 2023-11-05T08:33:00.918 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.918 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.918 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.918 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:00.919 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670924+0000", 2023-11-05T08:33:00.919 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.919 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.919 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.919 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:00.919 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670925+0000", 2023-11-05T08:33:00.920 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.8299999999999996e-07 2023-11-05T08:33:00.920 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.920 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.920 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:00.920 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671022+0000", 2023-11-05T08:33:00.920 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.7445e-05 2023-11-05T08:33:00.921 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.921 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.921 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:00.921 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671033+0000", 2023-11-05T08:33:00.921 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1206000000000001e-05 2023-11-05T08:33:00.921 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.922 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.922 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:00.922 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671045+0000", 2023-11-05T08:33:00.922 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1797000000000001e-05 2023-11-05T08:33:00.922 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.922 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.923 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:00.923 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.688874+0000", 2023-11-05T08:33:00.923 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017829424 2023-11-05T08:33:00.923 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.924 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.924 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:00.924 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.689097+0000", 2023-11-05T08:33:00.924 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000222828 2023-11-05T08:33:00.924 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.924 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.925 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:00.925 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.689444+0000", 2023-11-05T08:33:00.925 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00034672099999999998 2023-11-05T08:33:00.925 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.925 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.926 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:00.926 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692240+0000", 2023-11-05T08:33:00.926 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0027959030000000002 2023-11-05T08:33:00.926 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.926 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.927 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:00.927 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692339+0000", 2023-11-05T08:33:00.927 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.8801000000000005e-05 2023-11-05T08:33:00.927 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.927 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.928 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:00.928 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692435+0000", 2023-11-05T08:33:00.928 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6311999999999995e-05 2023-11-05T08:33:00.928 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.928 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.928 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:00.929 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692546+0000", 2023-11-05T08:33:00.929 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00011055700000000001 2023-11-05T08:33:00.929 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.929 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:00.929 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.929 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.930 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.930 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:00.930 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.671060+0000", 2023-11-05T08:33:00.930 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 30.983286586999998, 2023-11-05T08:33:00.930 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022080292000000001, 2023-11-05T08:33:00.931 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:00.931 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:00.931 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:00.931 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:00.931 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:00.932 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 165 2023-11-05T08:33:00.932 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.932 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:00.932 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.932 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:00.932 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671060+0000", 2023-11-05T08:33:00.933 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.933 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.933 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.933 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:00.933 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671060+0000", 2023-11-05T08:33:00.934 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.934 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.934 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.934 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:00.934 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671060+0000", 2023-11-05T08:33:00.935 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.1699999999999997e-07 2023-11-05T08:33:00.935 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.935 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.935 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:00.935 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671067+0000", 2023-11-05T08:33:00.935 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5690000000000001e-06 2023-11-05T08:33:00.937 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.937 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.938 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:00.938 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671076+0000", 2023-11-05T08:33:00.938 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.5270000000000004e-06 2023-11-05T08:33:00.938 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.938 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.939 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:00.939 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671084+0000", 2023-11-05T08:33:00.939 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.3459999999999996e-06 2023-11-05T08:33:00.939 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.939 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.940 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:00.940 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.690832+0000", 2023-11-05T08:33:00.940 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.019747793 2023-11-05T08:33:00.940 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.940 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.941 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:00.941 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.690971+0000", 2023-11-05T08:33:00.941 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000139245 2023-11-05T08:33:00.941 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.941 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.941 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:00.942 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.691178+0000", 2023-11-05T08:33:00.942 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000206892 2023-11-05T08:33:00.942 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.942 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.942 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:00.943 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692704+0000", 2023-11-05T08:33:00.943 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00152655 2023-11-05T08:33:00.943 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.943 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.943 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:00.944 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692904+0000", 2023-11-05T08:33:00.944 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000199376 2023-11-05T08:33:00.944 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.944 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.944 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:00.944 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.693044+0000", 2023-11-05T08:33:00.945 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013989199999999999 2023-11-05T08:33:00.945 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.945 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.945 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:00.945 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.693140+0000", 2023-11-05T08:33:00.946 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6285000000000004e-05 2023-11-05T08:33:00.946 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.946 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:00.946 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.946 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.947 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.947 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:184 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:00.947 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.866152+0000", 2023-11-05T08:33:00.947 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 30.788194312000002, 2023-11-05T08:33:00.947 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.019116587000000001, 2023-11-05T08:33:00.948 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:00.948 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:00.948 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:00.948 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:00.948 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:00.948 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 184 2023-11-05T08:33:00.949 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.949 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:00.949 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.949 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:00.949 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866152+0000", 2023-11-05T08:33:00.949 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.950 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.950 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.950 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:00.950 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866152+0000", 2023-11-05T08:33:00.951 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.951 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.951 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.951 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:00.951 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866152+0000", 2023-11-05T08:33:00.951 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4600000000000004e-07 2023-11-05T08:33:00.952 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.952 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.952 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:00.952 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866158+0000", 2023-11-05T08:33:00.952 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5629999999999999e-06 2023-11-05T08:33:00.953 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.953 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.953 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:00.953 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866167+0000", 2023-11-05T08:33:00.953 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.2229999999999996e-06 2023-11-05T08:33:00.954 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.954 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.954 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:00.954 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866175+0000", 2023-11-05T08:33:00.954 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.3699999999999997e-06 2023-11-05T08:33:00.955 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.955 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.955 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:00.955 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.878285+0000", 2023-11-05T08:33:00.955 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012110164 2023-11-05T08:33:00.956 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.956 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.956 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:00.956 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.878584+0000", 2023-11-05T08:33:00.956 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00029961300000000001 2023-11-05T08:33:00.957 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.957 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.957 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:00.957 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.878942+0000", 2023-11-05T08:33:00.957 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00035795100000000002 2023-11-05T08:33:00.958 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.958 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.958 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:00.958 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.883845+0000", 2023-11-05T08:33:00.958 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0049021450000000001 2023-11-05T08:33:00.959 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.959 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.959 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:00.959 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885018+0000", 2023-11-05T08:33:00.959 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0011738160000000001 2023-11-05T08:33:00.960 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.960 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.960 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:00.960 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885167+0000", 2023-11-05T08:33:00.960 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014895800000000001 2023-11-05T08:33:00.960 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.961 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.961 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:00.961 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885268+0000", 2023-11-05T08:33:00.961 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010113800000000001 2023-11-05T08:33:00.961 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.962 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:00.962 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.962 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.962 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.962 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:185 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:00.963 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.866212+0000", 2023-11-05T08:33:00.963 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 30.788134591999999, 2023-11-05T08:33:00.963 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.019448929, 2023-11-05T08:33:00.963 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:00.963 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:00.964 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:00.964 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:00.964 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:00.964 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 185 2023-11-05T08:33:00.964 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.964 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:00.965 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.965 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:00.965 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866212+0000", 2023-11-05T08:33:00.965 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.965 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.966 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.966 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:00.966 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866212+0000", 2023-11-05T08:33:00.966 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.966 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.967 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.967 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:00.967 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866213+0000", 2023-11-05T08:33:00.967 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0559999999999999e-06 2023-11-05T08:33:00.967 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.968 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.968 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:00.968 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866308+0000", 2023-11-05T08:33:00.968 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.4984999999999999e-05 2023-11-05T08:33:00.968 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.968 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.969 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:00.969 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866320+0000", 2023-11-05T08:33:00.969 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2639e-05 2023-11-05T08:33:00.969 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.969 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.969 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:00.970 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866328+0000", 2023-11-05T08:33:00.970 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.1240000000000005e-06 2023-11-05T08:33:00.970 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.970 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.970 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:00.971 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.879899+0000", 2023-11-05T08:33:00.971 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.013571059999999999 2023-11-05T08:33:00.971 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.971 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.971 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:00.971 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.880034+0000", 2023-11-05T08:33:00.972 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000134676 2023-11-05T08:33:00.972 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.972 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.972 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:00.972 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.880258+0000", 2023-11-05T08:33:00.973 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00022358100000000001 2023-11-05T08:33:00.973 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.973 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.973 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:00.973 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885299+0000", 2023-11-05T08:33:00.973 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.005041678 2023-11-05T08:33:00.974 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.974 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.974 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:00.974 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885436+0000", 2023-11-05T08:33:00.975 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000136472 2023-11-05T08:33:00.975 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.975 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.975 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:00.975 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885557+0000", 2023-11-05T08:33:00.975 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000121644 2023-11-05T08:33:00.976 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.976 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.976 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:00.976 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885660+0000", 2023-11-05T08:33:00.976 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000103014 2023-11-05T08:33:00.977 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.977 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:00.977 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.977 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.977 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.978 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:186 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:00.978 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.866343+0000", 2023-11-05T08:33:00.978 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 30.788002648999999, 2023-11-05T08:33:00.978 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.021691589000000001, 2023-11-05T08:33:00.978 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:00.979 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:00.979 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:00.979 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:00.979 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:00.979 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 186 2023-11-05T08:33:00.980 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.980 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:00.980 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.980 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:00.980 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866343+0000", 2023-11-05T08:33:00.980 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.981 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.981 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.981 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:00.981 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866343+0000", 2023-11-05T08:33:00.981 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.982 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.982 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.982 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:00.982 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866344+0000", 2023-11-05T08:33:00.982 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.6000000000000003e-07 2023-11-05T08:33:00.983 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.983 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.983 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:00.983 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866351+0000", 2023-11-05T08:33:00.983 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3090000000000001e-06 2023-11-05T08:33:00.984 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.984 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.984 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:00.984 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866360+0000", 2023-11-05T08:33:00.984 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.2599999999999994e-06 2023-11-05T08:33:00.985 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.985 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.985 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:00.985 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866367+0000", 2023-11-05T08:33:00.985 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.7599999999999997e-06 2023-11-05T08:33:00.985 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.986 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.986 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:00.986 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.882269+0000", 2023-11-05T08:33:00.986 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.015902218999999999 2023-11-05T08:33:00.986 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.987 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.987 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:00.987 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.882545+0000", 2023-11-05T08:33:00.987 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00027617199999999998 2023-11-05T08:33:00.987 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.988 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.988 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:00.988 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.882925+0000", 2023-11-05T08:33:00.988 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00037995900000000003 2023-11-05T08:33:00.988 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.989 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.989 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:00.989 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.887718+0000", 2023-11-05T08:33:00.989 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0047935069999999998 2023-11-05T08:33:00.989 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.990 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.990 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:00.990 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.887836+0000", 2023-11-05T08:33:00.990 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000117401 2023-11-05T08:33:00.990 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.991 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.991 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:00.991 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.887931+0000", 2023-11-05T08:33:00.991 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.4802000000000005e-05 2023-11-05T08:33:00.991 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.991 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.992 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:00.992 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888035+0000", 2023-11-05T08:33:00.992 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010444 2023-11-05T08:33:00.992 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.992 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:00.992 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:00.993 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.993 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.993 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:187 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:00.993 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.866377+0000", 2023-11-05T08:33:00.993 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 30.787968687999999, 2023-11-05T08:33:00.994 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022216751, 2023-11-05T08:33:00.994 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:00.994 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:00.994 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:00.994 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:00.994 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:00.995 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 187 2023-11-05T08:33:00.995 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.995 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:00.995 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.995 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:00.996 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866377+0000", 2023-11-05T08:33:00.996 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.996 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.996 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.996 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:00.996 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866377+0000", 2023-11-05T08:33:00.997 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:00.997 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.997 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.997 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:00.997 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866378+0000", 2023-11-05T08:33:00.998 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9999999999999997e-07 2023-11-05T08:33:00.998 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.998 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.998 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:00.998 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866384+0000", 2023-11-05T08:33:00.999 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.4099999999999999e-06 2023-11-05T08:33:00.999 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:00.999 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:00.999 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:00.999 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866393+0000", 2023-11-05T08:33:00.999 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9719999999999998e-06 2023-11-05T08:33:01.000 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.000 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.000 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:01.000 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866400+0000", 2023-11-05T08:33:01.000 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.2740000000000002e-06 2023-11-05T08:33:01.001 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.001 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.001 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:01.001 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.883936+0000", 2023-11-05T08:33:01.001 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017536513 2023-11-05T08:33:01.002 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.002 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.002 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:01.002 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.884061+0000", 2023-11-05T08:33:01.002 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012435300000000001 2023-11-05T08:33:01.003 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.003 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.003 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:01.003 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.884277+0000", 2023-11-05T08:33:01.003 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00021665500000000001 2023-11-05T08:33:01.003 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.004 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.004 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:01.004 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888193+0000", 2023-11-05T08:33:01.004 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0039160319999999998 2023-11-05T08:33:01.004 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.005 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.005 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:01.005 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888235+0000", 2023-11-05T08:33:01.005 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1267999999999999e-05 2023-11-05T08:33:01.005 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.005 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.006 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:01.006 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888373+0000", 2023-11-05T08:33:01.006 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000138789 2023-11-05T08:33:01.006 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.006 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.007 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:01.007 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888594+0000", 2023-11-05T08:33:01.007 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00022078500000000001 2023-11-05T08:33:01.007 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:01.007 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:01.007 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:01.008 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.008 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.008 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:188 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:01.008 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.866411+0000", 2023-11-05T08:33:01.008 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 30.787935427000001, 2023-11-05T08:33:01.009 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.023678187999999999, 2023-11-05T08:33:01.009 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:01.009 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:01.009 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:01.009 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:01.010 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:01.010 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 188 2023-11-05T08:33:01.010 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.010 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:01.010 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.011 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:01.011 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866411+0000", 2023-11-05T08:33:01.011 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:01.011 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.011 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.012 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:01.012 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866411+0000", 2023-11-05T08:33:01.012 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:01.012 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.012 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.012 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:01.013 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866411+0000", 2023-11-05T08:33:01.013 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.44e-07 2023-11-05T08:33:01.013 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.013 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.013 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:01.014 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866531+0000", 2023-11-05T08:33:01.014 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000119664 2023-11-05T08:33:01.014 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.014 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.014 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:01.015 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866545+0000", 2023-11-05T08:33:01.015 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3875e-05 2023-11-05T08:33:01.015 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.015 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.015 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:01.016 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866554+0000", 2023-11-05T08:33:01.016 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9099999999999994e-06 2023-11-05T08:33:01.016 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.016 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.016 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:01.017 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885784+0000", 2023-11-05T08:33:01.017 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.019230651000000001 2023-11-05T08:33:01.017 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.017 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.017 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:01.018 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.886020+0000", 2023-11-05T08:33:01.018 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00023538300000000001 2023-11-05T08:33:01.018 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.018 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.018 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:01.018 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.886386+0000", 2023-11-05T08:33:01.019 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00036659399999999998 2023-11-05T08:33:01.019 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.019 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.019 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:01.019 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888662+0000", 2023-11-05T08:33:01.020 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0022756019999999998 2023-11-05T08:33:01.020 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.020 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.020 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:01.020 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.889922+0000", 2023-11-05T08:33:01.021 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0012603149999999999 2023-11-05T08:33:01.021 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.021 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.021 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:01.021 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.890034+0000", 2023-11-05T08:33:01.022 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000111283 2023-11-05T08:33:01.022 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.022 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.022 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:01.022 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.890089+0000", 2023-11-05T08:33:01.023 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5266999999999998e-05 2023-11-05T08:33:01.023 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:01.023 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:01.023 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:01.023 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.024 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.024 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:189 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:01.024 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.866572+0000", 2023-11-05T08:33:01.024 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 30.787774468999999, 2023-11-05T08:33:01.024 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.024895937999999999, 2023-11-05T08:33:01.025 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:01.025 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:01.025 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:01.025 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:01.025 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:01.025 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 189 2023-11-05T08:33:01.026 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.026 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:01.026 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.026 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:01.026 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866572+0000", 2023-11-05T08:33:01.027 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:01.027 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.027 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.027 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:01.027 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866572+0000", 2023-11-05T08:33:01.027 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:01.028 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.028 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.028 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:01.028 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866572+0000", 2023-11-05T08:33:01.028 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.8000000000000005e-07 2023-11-05T08:33:01.029 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.029 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.029 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:01.029 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866581+0000", 2023-11-05T08:33:01.029 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9800000000000004e-06 2023-11-05T08:33:01.030 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.030 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.030 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:01.030 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866591+0000", 2023-11-05T08:33:01.030 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6059999999999992e-06 2023-11-05T08:33:01.031 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.031 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.031 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:01.031 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866598+0000", 2023-11-05T08:33:01.031 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.1029999999999996e-06 2023-11-05T08:33:01.032 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.032 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.032 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:01.032 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888850+0000", 2023-11-05T08:33:01.032 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022251615999999998 2023-11-05T08:33:01.032 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.033 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.033 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:01.033 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888988+0000", 2023-11-05T08:33:01.033 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000138436 2023-11-05T08:33:01.033 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.034 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.034 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:01.034 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.889206+0000", 2023-11-05T08:33:01.034 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00021818099999999999 2023-11-05T08:33:01.034 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.035 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.035 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:01.035 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.890621+0000", 2023-11-05T08:33:01.035 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001414219 2023-11-05T08:33:01.035 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.036 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.036 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:01.036 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.891316+0000", 2023-11-05T08:33:01.036 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00069570500000000002 2023-11-05T08:33:01.036 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.036 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.037 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:01.037 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.891406+0000", 2023-11-05T08:33:01.037 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9901999999999994e-05 2023-11-05T08:33:01.037 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.037 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.038 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:01.038 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.891468+0000", 2023-11-05T08:33:01.038 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1409999999999996e-05 2023-11-05T08:33:01.038 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:01.038 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:01.038 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:01.039 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.039 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.039 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:240 3.2s0 3:41c0ae03:::smithi09039252-50:head [delete] snapc 5=[5,3] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e37)", 2023-11-05T08:33:01.039 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:50.075207+0000", 2023-11-05T08:33:01.039 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 10.579139355000001, 2023-11-05T08:33:01.040 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017250943000000001, 2023-11-05T08:33:01.040 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:01.040 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:01.040 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:01.040 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:01.040 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:01.041 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 240 2023-11-05T08:33:01.041 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.041 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:01.041 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.041 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:01.041 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:50.075207+0000", 2023-11-05T08:33:01.042 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:01.042 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.042 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.042 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:01.042 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:50.075207+0000", 2023-11-05T08:33:01.043 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:01.043 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.043 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.043 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:01.043 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:50.075208+0000", 2023-11-05T08:33:01.044 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.689e-06 2023-11-05T08:33:01.044 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.044 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.044 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:01.044 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:50.075217+0000", 2023-11-05T08:33:01.044 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.3040000000000005e-06 2023-11-05T08:33:01.045 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.045 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.045 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:01.045 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:50.075232+0000", 2023-11-05T08:33:01.045 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5423000000000001e-05 2023-11-05T08:33:01.046 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.046 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.046 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:01.046 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:50.075245+0000", 2023-11-05T08:33:01.046 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2493999999999999e-05 2023-11-05T08:33:01.047 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.047 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.047 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:01.047 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:50.075326+0000", 2023-11-05T08:33:01.047 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.0885999999999998e-05 2023-11-05T08:33:01.048 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.048 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.048 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op must wait for map", 2023-11-05T08:33:01.048 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:50.075355+0000", 2023-11-05T08:33:01.048 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8918999999999999e-05 2023-11-05T08:33:01.049 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.049 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.049 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:01.049 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:50.086759+0000", 2023-11-05T08:33:01.049 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011404334 2023-11-05T08:33:01.049 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.050 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.050 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:01.050 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:50.086897+0000", 2023-11-05T08:33:01.050 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013802700000000001 2023-11-05T08:33:01.050 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.051 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.051 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:01.051 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:50.087725+0000", 2023-11-05T08:33:01.051 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00082781100000000002 2023-11-05T08:33:01.051 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.051 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.052 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:01.052 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:50.091222+0000", 2023-11-05T08:33:01.052 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0034972950000000001 2023-11-05T08:33:01.052 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.052 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.052 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:01.053 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:50.092380+0000", 2023-11-05T08:33:01.053 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001158299 2023-11-05T08:33:01.053 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.053 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.053 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:01.054 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:50.092458+0000", 2023-11-05T08:33:01.054 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.7461999999999998e-05 2023-11-05T08:33:01.054 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:01.054 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:01.054 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:01.054 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.055 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.055 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:292 3.2s0 3:41aa5db9:::smithi09039252-7:head [delete] snapc 8=[8,7,6,5] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e41)", 2023-11-05T08:33:01.055 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.121023+0000", 2023-11-05T08:33:01.055 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 6.5333230540000002, 2023-11-05T08:33:01.055 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.061679772000000001, 2023-11-05T08:33:01.056 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:01.056 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:01.056 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:01.056 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:01.056 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:01.057 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 292 2023-11-05T08:33:01.057 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.057 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:01.057 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.057 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:01.057 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.121023+0000", 2023-11-05T08:33:01.058 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:01.058 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.058 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.058 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:01.058 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.121023+0000", 2023-11-05T08:33:01.059 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:01.059 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.059 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.059 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:01.059 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.121026+0000", 2023-11-05T08:33:01.059 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0369999999999999e-06 2023-11-05T08:33:01.060 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.060 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.060 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:01.060 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.121039+0000", 2023-11-05T08:33:01.060 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3103e-05 2023-11-05T08:33:01.061 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.061 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.061 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:01.061 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.121086+0000", 2023-11-05T08:33:01.061 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7095000000000002e-05 2023-11-05T08:33:01.062 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.062 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.062 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:01.062 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.121131+0000", 2023-11-05T08:33:01.062 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4455e-05 2023-11-05T08:33:01.063 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.063 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.063 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:01.063 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.129136+0000", 2023-11-05T08:33:01.063 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0080047439999999994 2023-11-05T08:33:01.064 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.064 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.064 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2023-11-05T08:33:01.064 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.129142+0000", 2023-11-05T08:33:01.064 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3759999999999996e-06 2023-11-05T08:33:01.065 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.065 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.065 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:01.065 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.158095+0000", 2023-11-05T08:33:01.065 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.028953402 2023-11-05T08:33:01.065 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.066 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.066 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:01.066 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.158534+0000", 2023-11-05T08:33:01.066 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00043911800000000001 2023-11-05T08:33:01.066 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.067 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.067 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:01.067 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.160380+0000", 2023-11-05T08:33:01.067 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001845923 2023-11-05T08:33:01.067 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.068 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.068 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:01.068 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.177604+0000", 2023-11-05T08:33:01.068 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017223405000000001 2023-11-05T08:33:01.068 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.068 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.069 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:01.069 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.182556+0000", 2023-11-05T08:33:01.069 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0049522740000000004 2023-11-05T08:33:01.069 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:01.069 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:01.069 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:01.070 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.182703+0000", 2023-11-05T08:33:01.070 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014684 2023-11-05T08:33:01.070 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:01.070 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:01.070 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:01.071 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:01.071 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:01.071 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:33:01.072 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5: {} 2023-11-05T08:33:01.073 INFO:teuthology.orchestra.run.smithi002.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-11-05T08:33:01.074 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6: {} 2023-11-05T08:33:01.074 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --force --op remove --pgid 2.2 2023-11-05T08:33:01.075 INFO:teuthology.orchestra.run.smithi002.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-11-05T08:33:01.075 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-11-05T08:33:01.076 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:01.069+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:01.076 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7: {} 2023-11-05T08:33:01.077 INFO:teuthology.orchestra.run.smithi002.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-11-05T08:33:01.077 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8: {} 2023-11-05T08:33:01.078 INFO:teuthology.orchestra.run.smithi002.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-11-05T08:33:01.078 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9: {} 2023-11-05T08:33:01.079 INFO:teuthology.orchestra.run.smithi002.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-11-05T08:33:01.079 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10: {} 2023-11-05T08:33:01.080 INFO:teuthology.orchestra.run.smithi002.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-11-05T08:33:01.080 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11: {} 2023-11-05T08:33:01.080 INFO:teuthology.orchestra.run.smithi002.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-11-05T08:33:01.081 INFO:teuthology.orchestra.run.smithi002.stdout:osd.12: {} 2023-11-05T08:33:01.081 INFO:teuthology.orchestra.run.smithi002.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-11-05T08:33:01.082 INFO:teuthology.orchestra.run.smithi002.stdout:osd.13: {} 2023-11-05T08:33:01.082 INFO:teuthology.orchestra.run.smithi002.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-11-05T08:33:01.082 INFO:teuthology.orchestra.run.smithi002.stdout:osd.14: {} 2023-11-05T08:33:01.083 INFO:teuthology.orchestra.run.smithi002.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-11-05T08:33:01.083 INFO:teuthology.orchestra.run.smithi002.stdout:osd.15: {} 2023-11-05T08:33:01.084 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-05T08:33:01.085 INFO:teuthology.orchestra.run.smithi002.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-11-05T08:33:01.085 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-05T08:33:01.172 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-05T08:33:01.200 INFO:tasks.rados.rados.0.smithi090.stdout:120: done (1 left) 2023-11-05T08:33:01.200 INFO:tasks.rados.rados.0.smithi090.stdout:121: setattr oid 39 current snap is 8 2023-11-05T08:33:01.202 INFO:tasks.rados.rados.0.smithi090.stdout:122: read oid 34 snap 4 2023-11-05T08:33:01.202 INFO:tasks.rados.rados.0.smithi090.stdout:122: expect (ObjNum 33 snap 0 seq_num 33) 2023-11-05T08:33:01.202 INFO:tasks.rados.rados.0.smithi090.stdout:123: delete oid 31 current snap is 8 2023-11-05T08:33:01.214 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:01.214 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:01.218 INFO:tasks.rados.rados.0.smithi090.stdout:122: expect (ObjNum 33 snap 0 seq_num 33) 2023-11-05T08:33:01.231 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-11-05T08:33:01.236 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 39 v 27 (ObjNum 38 snap 0 seq_num 38) dirty exists 2023-11-05T08:33:01.236 INFO:tasks.rados.rados.0.smithi090.stdout:121: done (3 left) 2023-11-05T08:33:01.237 INFO:tasks.rados.rados.0.smithi090.stdout:122: done (2 left) 2023-11-05T08:33:01.237 INFO:tasks.rados.rados.0.smithi090.stdout:123: done (1 left) 2023-11-05T08:33:01.237 INFO:tasks.rados.rados.0.smithi090.stdout:124: copy_from oid 46 from oid 19 current snap is 8 2023-11-05T08:33:01.237 INFO:tasks.rados.rados.0.smithi090.stdout:125: snap_remove snap 2 2023-11-05T08:33:01.275 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:33:01.270+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:33:01.348 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:01.348 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:01.362 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-11-05T08:33:01.375 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:33:01.374+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:33:01.436 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-11-05T08:33:01.474 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-05T08:33:01.485 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:01.485 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:01.499 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-11-05T08:33:01.577 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:33:01.573+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:33:01.615 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:01.616 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:01.629 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-11-05T08:33:01.678 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:01.673+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:01.748 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:01.748 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:01.762 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-11-05T08:33:01.779 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:01.777+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:01.880 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:33:01.879+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:33:01.883 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:01.883 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:01.897 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2023-11-05T08:33:01.981 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:33:01.976+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:33:02.067 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:02.067 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:02.081 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2023-11-05T08:33:02.082 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:33:02.076+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:33:02.182 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:33:02.178+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:33:02.200 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:02.200 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:02.202 INFO:teuthology.orchestra.run.smithi002.stderr:nodeep-scrub is unset 2023-11-05T08:33:02.211 INFO:tasks.rados.rados.0.smithi090.stdout:125: done (2 left) 2023-11-05T08:33:02.211 INFO:tasks.rados.rados.0.smithi090.stdout:126: snap_create 2023-11-05T08:33:02.211 INFO:tasks.rados.rados.0.smithi090.stdout:124: finishing copy_from racing read to smithi09039252-46 2023-11-05T08:33:02.211 INFO:tasks.rados.rados.0.smithi090.stdout:124: finishing copy_from to smithi09039252-46 2023-11-05T08:33:02.211 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 46 v 2 (ObjNum 18 snap 0 seq_num 18) dirty exists 2023-11-05T08:33:02.218 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2023-11-05T08:33:02.283 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:33:02.283+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:33:02.284 INFO:teuthology.orchestra.run.smithi090.stdout: marking collection for removal 2023-11-05T08:33:02.284 INFO:teuthology.orchestra.run.smithi090.stdout:setting '_remove' omap key 2023-11-05T08:33:02.285 INFO:teuthology.orchestra.run.smithi090.stdout:finish_remove_pgs 2.2_head removing 2.2 2023-11-05T08:33:02.289 INFO:teuthology.orchestra.run.smithi090.stdout:Remove successful 2023-11-05T08:33:02.345 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:02.345 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:02.359 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2023-11-05T08:33:02.403 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:33:02.403+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:33:02.485 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:33:02.482+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:33:02.523 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:02.523 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:02.537 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2023-11-05T08:33:02.586 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:33:02.583+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:33:02.660 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:02.661 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:02.674 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2023-11-05T08:33:02.687 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:33:02.685+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:33:02.788 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:33:02.786+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:33:02.827 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.2.2.3 2023-11-05T08:33:02.843 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:02.843 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:02.857 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-11-05T08:33:02.888 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:33:02.887+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:33:02.989 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:02.984+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:02.998 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:02.998 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:03.012 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-11-05T08:33:03.055 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~8s 2023-11-05T08:33:03.086 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-05T08:33:03.090 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:33:03.087+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:33:03.173 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:03.174 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:03.187 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-11-05T08:33:03.191 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:33:03.189+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:33:03.210 INFO:tasks.rados.rados.0.smithi090.stdout:124: done (2 left) 2023-11-05T08:33:03.211 INFO:tasks.rados.rados.0.smithi090.stdout:126: done (1 left) 2023-11-05T08:33:03.211 INFO:tasks.rados.rados.0.smithi090.stdout:127: read oid 9 snap -1 2023-11-05T08:33:03.211 INFO:tasks.rados.rados.0.smithi090.stdout:127: expect (ObjNum 6 snap 0 seq_num 6) 2023-11-05T08:33:03.211 INFO:tasks.rados.rados.0.smithi090.stdout:128: snap_remove snap 7 2023-11-05T08:33:03.228 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-11-05T08:33:03.249 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1: {} 2023-11-05T08:33:03.250 INFO:teuthology.orchestra.run.smithi002.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-11-05T08:33:03.271 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2: {} 2023-11-05T08:33:03.271 INFO:teuthology.orchestra.run.smithi002.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-11-05T08:33:03.271 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-11-05T08:33:03.288 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4: {} 2023-11-05T08:33:03.288 INFO:teuthology.orchestra.run.smithi002.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-11-05T08:33:03.290 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-05T08:33:03.309 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5: {} 2023-11-05T08:33:03.309 INFO:teuthology.orchestra.run.smithi002.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-11-05T08:33:03.317 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:03.318 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:03.331 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6: {} 2023-11-05T08:33:03.331 INFO:teuthology.orchestra.run.smithi002.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-11-05T08:33:03.337 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2023-11-05T08:33:03.349 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7: {} 2023-11-05T08:33:03.349 INFO:teuthology.orchestra.run.smithi002.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-11-05T08:33:03.366 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8: {} 2023-11-05T08:33:03.367 INFO:teuthology.orchestra.run.smithi002.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-11-05T08:33:03.387 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9: {} 2023-11-05T08:33:03.387 INFO:teuthology.orchestra.run.smithi002.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-11-05T08:33:03.393 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:33:03.389+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:33:03.409 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10: {} 2023-11-05T08:33:03.409 INFO:teuthology.orchestra.run.smithi002.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-11-05T08:33:03.427 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11: {} 2023-11-05T08:33:03.428 INFO:teuthology.orchestra.run.smithi002.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-11-05T08:33:03.447 INFO:teuthology.orchestra.run.smithi002.stdout:osd.12: {} 2023-11-05T08:33:03.447 INFO:teuthology.orchestra.run.smithi002.stderr:osd.12: osd_enable_op_tracker = 'true' 2023-11-05T08:33:03.468 INFO:teuthology.orchestra.run.smithi002.stdout:osd.13: {} 2023-11-05T08:33:03.468 INFO:teuthology.orchestra.run.smithi002.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-11-05T08:33:03.489 INFO:teuthology.orchestra.run.smithi002.stdout:osd.14: {} 2023-11-05T08:33:03.490 INFO:teuthology.orchestra.run.smithi002.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-11-05T08:33:03.493 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:03.493+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:03.500 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:33:03.501 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-11-05T08:33:03.501 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-11-05T08:33:03.501 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:33:03.509 INFO:teuthology.orchestra.run.smithi002.stdout:osd.15: {} 2023-11-05T08:33:03.509 INFO:teuthology.orchestra.run.smithi002.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-11-05T08:33:03.513 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-05T08:33:03.514 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-05T08:33:03.515 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2023-11-05T08:33:03.594 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:33:03.588+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:33:03.633 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:33:03.633 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-11-05T08:33:03.633 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-11-05T08:33:03.633 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-11-05T08:33:03.633 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:33:03.647 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2023-11-05T08:33:03.694 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:03.693+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:03.795 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:33:03.793+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:33:03.829 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:33:03.829 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-11-05T08:33:03.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-11-05T08:33:03.830 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-11-05T08:33:03.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.830 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.es3 36/30 ECSubWrite(tid=33, reqid=osd.2.0:33, at_version=36'21, trim_to=0'0, roll_forward_to=34'19))", 2023-11-05T08:33:03.830 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.081758+0000", 2023-11-05T08:33:03.831 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 14.744970502999999, 2023-11-05T08:33:03.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053246250000000004, 2023-11-05T08:33:03.831 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:03.831 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:03.831 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:03.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:03.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.081758+0000", 2023-11-05T08:33:03.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:03.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:03.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.081758+0000", 2023-11-05T08:33:03.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:03.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:03.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.081758+0000", 2023-11-05T08:33:03.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.99e-07 2023-11-05T08:33:03.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:03.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.081810+0000", 2023-11-05T08:33:03.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1440999999999998e-05 2023-11-05T08:33:03.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:03.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.081834+0000", 2023-11-05T08:33:03.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4085999999999999e-05 2023-11-05T08:33:03.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:03.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.081860+0000", 2023-11-05T08:33:03.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6258000000000002e-05 2023-11-05T08:33:03.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:03.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.082041+0000", 2023-11-05T08:33:03.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000181065 2023-11-05T08:33:03.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:03.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.082056+0000", 2023-11-05T08:33:03.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5099000000000001e-05 2023-11-05T08:33:03.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:03.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.086982+0000", 2023-11-05T08:33:03.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049261189999999996 2023-11-05T08:33:03.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:03.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.087082+0000", 2023-11-05T08:33:03.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9858000000000005e-05 2023-11-05T08:33:03.841 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:03.841 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:03.841 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:03.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.841 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.2s6 37/30 ECSubWrite(tid=185, reqid=client.4691.0:240, at_version=37'7, trim_to=0'0, roll_forward_to=33'5))", 2023-11-05T08:33:03.842 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:50.089141+0000", 2023-11-05T08:33:03.842 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.737587326, 2023-11-05T08:33:03.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029497260000000002, 2023-11-05T08:33:03.842 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:03.842 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.089141+0000", 2023-11-05T08:33:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.089141+0000", 2023-11-05T08:33:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.089142+0000", 2023-11-05T08:33:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1060000000000001e-06 2023-11-05T08:33:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.089181+0000", 2023-11-05T08:33:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9223e-05 2023-11-05T08:33:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.089209+0000", 2023-11-05T08:33:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7729000000000002e-05 2023-11-05T08:33:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.089223+0000", 2023-11-05T08:33:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4039000000000001e-05 2023-11-05T08:33:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.089383+0000", 2023-11-05T08:33:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000160154 2023-11-05T08:33:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.089400+0000", 2023-11-05T08:33:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6662000000000001e-05 2023-11-05T08:33:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.092040+0000", 2023-11-05T08:33:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026403429999999999 2023-11-05T08:33:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.092091+0000", 2023-11-05T08:33:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0470000000000003e-05 2023-11-05T08:33:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.as6 40/33 ECSubWrite(tid=30, reqid=client.4691.0:283, at_version=40'8, trim_to=0'0, roll_forward_to=39'6))", 2023-11-05T08:33:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:53.168255+0000", 2023-11-05T08:33:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 10.65847321, 2023-11-05T08:33:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053454050000000001, 2023-11-05T08:33:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.168255+0000", 2023-11-05T08:33:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.168255+0000", 2023-11-05T08:33:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.168256+0000", 2023-11-05T08:33:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1699999999999997e-07 2023-11-05T08:33:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:03.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.168538+0000", 2023-11-05T08:33:03.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028188099999999999 2023-11-05T08:33:03.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:03.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.168574+0000", 2023-11-05T08:33:03.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6307999999999999e-05 2023-11-05T08:33:03.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:03.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.168600+0000", 2023-11-05T08:33:03.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6145000000000001e-05 2023-11-05T08:33:03.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:03.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.168767+0000", 2023-11-05T08:33:03.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016706699999999999 2023-11-05T08:33:03.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:03.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.168783+0000", 2023-11-05T08:33:03.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6205000000000002e-05 2023-11-05T08:33:03.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:03.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.173507+0000", 2023-11-05T08:33:03.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0047232660000000003 2023-11-05T08:33:03.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:03.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.173600+0000", 2023-11-05T08:33:03.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3715999999999995e-05 2023-11-05T08:33:03.864 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:03.864 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:03.864 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:03.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.865 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.as6 40/33 ECSubWrite(tid=31, reqid=client.4691.0:284, at_version=40'9, trim_to=0'0, roll_forward_to=39'6))", 2023-11-05T08:33:03.865 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:53.174164+0000", 2023-11-05T08:33:03.865 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 10.652564182999999, 2023-11-05T08:33:03.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003777267, 2023-11-05T08:33:03.866 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:03.866 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:03.866 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:03.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:03.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.174164+0000", 2023-11-05T08:33:03.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:03.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:03.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.174164+0000", 2023-11-05T08:33:03.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:03.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:03.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.174167+0000", 2023-11-05T08:33:03.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6960000000000001e-06 2023-11-05T08:33:03.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:03.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.174316+0000", 2023-11-05T08:33:03.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014939 2023-11-05T08:33:03.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:03.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.174354+0000", 2023-11-05T08:33:03.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8005e-05 2023-11-05T08:33:03.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:03.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.174370+0000", 2023-11-05T08:33:03.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6099999999999998e-05 2023-11-05T08:33:03.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:03.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.174529+0000", 2023-11-05T08:33:03.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015849899999999999 2023-11-05T08:33:03.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:03.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.174545+0000", 2023-11-05T08:33:03.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6426000000000002e-05 2023-11-05T08:33:03.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:03.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.177879+0000", 2023-11-05T08:33:03.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033334580000000001 2023-11-05T08:33:03.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:03.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.177941+0000", 2023-11-05T08:33:03.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2693000000000004e-05 2023-11-05T08:33:03.875 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:03.875 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:03.875 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:03.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.876 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.es3 41/30 ECSubWrite(tid=40, reqid=osd.2.0:40, at_version=41'27, trim_to=0'0, roll_forward_to=40'25))", 2023-11-05T08:33:03.876 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:54.120247+0000", 2023-11-05T08:33:03.876 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 9.7064815440000007, 2023-11-05T08:33:03.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0072038550000000003, 2023-11-05T08:33:03.877 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:03.877 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:03.877 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:03.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:03.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.120247+0000", 2023-11-05T08:33:03.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:03.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.120247+0000", 2023-11-05T08:33:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.120247+0000", 2023-11-05T08:33:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7000000000000005e-07 2023-11-05T08:33:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.120307+0000", 2023-11-05T08:33:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9858000000000002e-05 2023-11-05T08:33:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.120340+0000", 2023-11-05T08:33:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3000999999999998e-05 2023-11-05T08:33:03.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:03.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.120365+0000", 2023-11-05T08:33:03.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5232000000000002e-05 2023-11-05T08:33:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.120516+0000", 2023-11-05T08:33:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015087400000000001 2023-11-05T08:33:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.120542+0000", 2023-11-05T08:33:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5562e-05 2023-11-05T08:33:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.127420+0000", 2023-11-05T08:33:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.006878009 2023-11-05T08:33:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.127450+0000", 2023-11-05T08:33:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0749e-05 2023-11-05T08:33:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.as6 41/33 ECSubWrite(tid=35, reqid=osd.7.0:35, at_version=41'12, trim_to=0'0, roll_forward_to=40'10))", 2023-11-05T08:33:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:54.126816+0000", 2023-11-05T08:33:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 9.6999118640000006, 2023-11-05T08:33:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0080159169999999991, 2023-11-05T08:33:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.126816+0000", 2023-11-05T08:33:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.126816+0000", 2023-11-05T08:33:03.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:03.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:03.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.126817+0000", 2023-11-05T08:33:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0999999999999998e-07 2023-11-05T08:33:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.126872+0000", 2023-11-05T08:33:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4970999999999999e-05 2023-11-05T08:33:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.126902+0000", 2023-11-05T08:33:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0045000000000001e-05 2023-11-05T08:33:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.126927+0000", 2023-11-05T08:33:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4586000000000001e-05 2023-11-05T08:33:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.127271+0000", 2023-11-05T08:33:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034440499999999999 2023-11-05T08:33:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.127298+0000", 2023-11-05T08:33:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6724999999999999e-05 2023-11-05T08:33:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.134738+0000", 2023-11-05T08:33:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0074405959999999998 2023-11-05T08:33:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:03.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.134832+0000", 2023-11-05T08:33:03.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3979000000000003e-05 2023-11-05T08:33:03.897 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:03.897 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:03.897 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.as6 41/33 ECSubWrite(tid=36, reqid=osd.7.0:36, at_version=41'14, trim_to=0'0, roll_forward_to=40'10))", 2023-11-05T08:33:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:54.130628+0000", 2023-11-05T08:33:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 9.6961002969999992, 2023-11-05T08:33:03.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0042689700000000004, 2023-11-05T08:33:03.899 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:03.899 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:03.899 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:03.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:03.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.130628+0000", 2023-11-05T08:33:03.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:03.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.130628+0000", 2023-11-05T08:33:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.130629+0000", 2023-11-05T08:33:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1999999999999998e-07 2023-11-05T08:33:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.130664+0000", 2023-11-05T08:33:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5108999999999999e-05 2023-11-05T08:33:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.130677+0000", 2023-11-05T08:33:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3495999999999999e-05 2023-11-05T08:33:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.130687+0000", 2023-11-05T08:33:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1630000000000002e-06 2023-11-05T08:33:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.130780+0000", 2023-11-05T08:33:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3057e-05 2023-11-05T08:33:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.130790+0000", 2023-11-05T08:33:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0456e-05 2023-11-05T08:33:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.134841+0000", 2023-11-05T08:33:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040509710000000004 2023-11-05T08:33:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.134897+0000", 2023-11-05T08:33:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5797999999999997e-05 2023-11-05T08:33:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.2s6 41/40 ECSubWrite(tid=187, reqid=client.4691.0:292, at_version=41'9, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:54.161570+0000", 2023-11-05T08:33:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 9.6651579079999994, 2023-11-05T08:33:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026231179999999998, 2023-11-05T08:33:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.161570+0000", 2023-11-05T08:33:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.161570+0000", 2023-11-05T08:33:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.161571+0000", 2023-11-05T08:33:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1850000000000001e-06 2023-11-05T08:33:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.161613+0000", 2023-11-05T08:33:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1516e-05 2023-11-05T08:33:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.161640+0000", 2023-11-05T08:33:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7259000000000001e-05 2023-11-05T08:33:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.161661+0000", 2023-11-05T08:33:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0319e-05 2023-11-05T08:33:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.161778+0000", 2023-11-05T08:33:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000117027 2023-11-05T08:33:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.161800+0000", 2023-11-05T08:33:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2934000000000001e-05 2023-11-05T08:33:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.164107+0000", 2023-11-05T08:33:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023064750000000001 2023-11-05T08:33:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.164193+0000", 2023-11-05T08:33:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6403000000000006e-05 2023-11-05T08:33:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.as6 41/33 ECSubWrite(tid=37, reqid=osd.7.0:36, at_version=0'0, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:33:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:54.168372+0000", 2023-11-05T08:33:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 9.6583559520000009, 2023-11-05T08:33:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004595643, 2023-11-05T08:33:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.168372+0000", 2023-11-05T08:33:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.168372+0000", 2023-11-05T08:33:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.168374+0000", 2023-11-05T08:33:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8589999999999999e-06 2023-11-05T08:33:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.168433+0000", 2023-11-05T08:33:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9342999999999997e-05 2023-11-05T08:33:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.168499+0000", 2023-11-05T08:33:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5097000000000004e-05 2023-11-05T08:33:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:03.927 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.168523+0000", 2023-11-05T08:33:03.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4022000000000001e-05 2023-11-05T08:33:03.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.927 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:03.927 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.168608+0000", 2023-11-05T08:33:03.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5537000000000005e-05 2023-11-05T08:33:03.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:03.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.168618+0000", 2023-11-05T08:33:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.027e-05 2023-11-05T08:33:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.172915+0000", 2023-11-05T08:33:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0042961689999999999 2023-11-05T08:33:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.172968+0000", 2023-11-05T08:33:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3346000000000001e-05 2023-11-05T08:33:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.2s6 41/40 ECSubWrite(tid=188, reqid=osd.13.0:188, at_version=41'11, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:54.169058+0000", 2023-11-05T08:33:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 9.6576700659999997, 2023-11-05T08:33:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039449539999999996, 2023-11-05T08:33:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.169058+0000", 2023-11-05T08:33:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:03.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.169058+0000", 2023-11-05T08:33:03.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:03.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:03.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.169059+0000", 2023-11-05T08:33:03.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8599999999999997e-07 2023-11-05T08:33:03.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:03.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.169084+0000", 2023-11-05T08:33:03.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4700999999999999e-05 2023-11-05T08:33:03.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:03.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.169098+0000", 2023-11-05T08:33:03.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4763000000000001e-05 2023-11-05T08:33:03.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:03.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.169107+0000", 2023-11-05T08:33:03.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5359999999999992e-06 2023-11-05T08:33:03.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:03.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.169192+0000", 2023-11-05T08:33:03.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4766999999999997e-05 2023-11-05T08:33:03.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:03.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.169201+0000", 2023-11-05T08:33:03.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9060000000000008e-06 2023-11-05T08:33:03.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:03.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.172972+0000", 2023-11-05T08:33:03.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037716550000000001 2023-11-05T08:33:03.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:03.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.173003+0000", 2023-11-05T08:33:03.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0840000000000003e-05 2023-11-05T08:33:03.941 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:03.942 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:03.942 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:03.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.942 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.11s1 41/40 ECSubWrite(tid=66, reqid=client.4691.0:288, at_version=41'14, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:03.942 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:54.177478+0000", 2023-11-05T08:33:03.943 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 9.6492502059999996, 2023-11-05T08:33:03.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039913029999999999, 2023-11-05T08:33:03.943 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:03.943 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:03.943 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:03.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:03.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.177478+0000", 2023-11-05T08:33:03.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:03.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:03.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.177478+0000", 2023-11-05T08:33:03.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:03.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:03.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.177479+0000", 2023-11-05T08:33:03.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8299999999999996e-07 2023-11-05T08:33:03.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:03.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.177711+0000", 2023-11-05T08:33:03.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023244699999999999 2023-11-05T08:33:03.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:03.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.177725+0000", 2023-11-05T08:33:03.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.396e-05 2023-11-05T08:33:03.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:03.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.177736+0000", 2023-11-05T08:33:03.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0913e-05 2023-11-05T08:33:03.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:03.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.177789+0000", 2023-11-05T08:33:03.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3156999999999999e-05 2023-11-05T08:33:03.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:03.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.177795+0000", 2023-11-05T08:33:03.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6289999999999998e-06 2023-11-05T08:33:03.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:03.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.181431+0000", 2023-11-05T08:33:03.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036362640000000002 2023-11-05T08:33:03.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:03.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.181469+0000", 2023-11-05T08:33:03.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8250000000000001e-05 2023-11-05T08:33:03.952 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:03.952 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:03.953 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:03.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.953 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.as6 42/33 ECSubWrite(tid=38, reqid=osd.7.0:38, at_version=42'16, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:33:03.953 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:55.122849+0000", 2023-11-05T08:33:03.953 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.7038793079999994, 2023-11-05T08:33:03.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015960385000000001, 2023-11-05T08:33:03.954 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:03.954 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:03.954 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:03.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:03.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.122849+0000", 2023-11-05T08:33:03.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:03.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:03.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.122849+0000", 2023-11-05T08:33:03.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:03.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:03.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.122850+0000", 2023-11-05T08:33:03.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1699999999999997e-07 2023-11-05T08:33:03.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:03.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.122895+0000", 2023-11-05T08:33:03.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5717999999999998e-05 2023-11-05T08:33:03.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:03.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.122919+0000", 2023-11-05T08:33:03.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3932000000000001e-05 2023-11-05T08:33:03.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:03.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.122941+0000", 2023-11-05T08:33:03.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1708999999999998e-05 2023-11-05T08:33:03.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:03.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.125137+0000", 2023-11-05T08:33:03.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021962219999999998 2023-11-05T08:33:03.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:03.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.125161+0000", 2023-11-05T08:33:03.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3436000000000001e-05 2023-11-05T08:33:03.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:03.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.138719+0000", 2023-11-05T08:33:03.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013558115000000001 2023-11-05T08:33:03.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:03.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.138809+0000", 2023-11-05T08:33:03.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0536000000000002e-05 2023-11-05T08:33:03.964 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:03.964 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:03.964 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:03.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.965 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.as6 42/33 ECSubWrite(tid=39, reqid=osd.7.0:39, at_version=42'18, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:33:03.965 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:55.125666+0000", 2023-11-05T08:33:03.965 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.7010621490000002, 2023-11-05T08:33:03.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013940509, 2023-11-05T08:33:03.965 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:03.965 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:03.966 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:03.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:03.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.125666+0000", 2023-11-05T08:33:03.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:03.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:03.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.125666+0000", 2023-11-05T08:33:03.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:03.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:03.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.125667+0000", 2023-11-05T08:33:03.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.47e-07 2023-11-05T08:33:03.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:03.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.125713+0000", 2023-11-05T08:33:03.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5748000000000003e-05 2023-11-05T08:33:03.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:03.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.125737+0000", 2023-11-05T08:33:03.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4369e-05 2023-11-05T08:33:03.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:03.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.125758+0000", 2023-11-05T08:33:03.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0608999999999999e-05 2023-11-05T08:33:03.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:03.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.126227+0000", 2023-11-05T08:33:03.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000468944 2023-11-05T08:33:03.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:03.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.126251+0000", 2023-11-05T08:33:03.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4788999999999999e-05 2023-11-05T08:33:03.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:03.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.139523+0000", 2023-11-05T08:33:03.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013271577 2023-11-05T08:33:03.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:03.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.139607+0000", 2023-11-05T08:33:03.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3526e-05 2023-11-05T08:33:03.975 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:03.975 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:03.975 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:03.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.975 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.es3 42/30 ECSubWrite(tid=43, reqid=osd.2.0:43, at_version=42'29, trim_to=0'0, roll_forward_to=41'27))", 2023-11-05T08:33:03.976 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:55.134690+0000", 2023-11-05T08:33:03.976 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.6920379449999992, 2023-11-05T08:33:03.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.007527003, 2023-11-05T08:33:03.976 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:03.976 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:03.976 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:03.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:03.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.134690+0000", 2023-11-05T08:33:03.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:03.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:03.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.134690+0000", 2023-11-05T08:33:03.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:03.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:03.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.134692+0000", 2023-11-05T08:33:03.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4699999999999999e-06 2023-11-05T08:33:03.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:03.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.134749+0000", 2023-11-05T08:33:03.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6941000000000003e-05 2023-11-05T08:33:03.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:03.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.134788+0000", 2023-11-05T08:33:03.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8952000000000002e-05 2023-11-05T08:33:03.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:03.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.134810+0000", 2023-11-05T08:33:03.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2785e-05 2023-11-05T08:33:03.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:03.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.135041+0000", 2023-11-05T08:33:03.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000230604 2023-11-05T08:33:03.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:03.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.135067+0000", 2023-11-05T08:33:03.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6012000000000001e-05 2023-11-05T08:33:03.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:03.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.142127+0000", 2023-11-05T08:33:03.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0070602629999999998 2023-11-05T08:33:03.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:03.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.142217+0000", 2023-11-05T08:33:03.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9975999999999994e-05 2023-11-05T08:33:03.986 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:03.986 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:03.986 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:03.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.987 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/40 ECSubWrite(tid=192, reqid=osd.13.0:192, at_version=42'13, trim_to=0'0, roll_forward_to=41'11))", 2023-11-05T08:33:03.987 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:55.149522+0000", 2023-11-05T08:33:03.987 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.6772059699999993, 2023-11-05T08:33:03.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039559950000000003, 2023-11-05T08:33:03.988 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:03.988 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:03.988 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:03.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:03.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.149522+0000", 2023-11-05T08:33:03.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:03.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:03.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.149522+0000", 2023-11-05T08:33:03.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:03.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:03.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.149524+0000", 2023-11-05T08:33:03.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8339999999999999e-06 2023-11-05T08:33:03.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:03.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.149574+0000", 2023-11-05T08:33:03.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0028000000000003e-05 2023-11-05T08:33:03.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:03.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.149619+0000", 2023-11-05T08:33:03.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5433999999999999e-05 2023-11-05T08:33:03.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:03.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.149643+0000", 2023-11-05T08:33:03.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3493e-05 2023-11-05T08:33:03.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:03.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.149805+0000", 2023-11-05T08:33:03.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016218600000000001 2023-11-05T08:33:03.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:03.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.149837+0000", 2023-11-05T08:33:03.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1619000000000003e-05 2023-11-05T08:33:03.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:03.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.153327+0000", 2023-11-05T08:33:03.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034906609999999999 2023-11-05T08:33:03.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:03.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.153478+0000", 2023-11-05T08:33:03.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015074000000000001 2023-11-05T08:33:03.997 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:03.997 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:03.998 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:03.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:03.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:03.998 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.as6 42/33 ECSubWrite(tid=40, reqid=osd.7.0:39, at_version=0'0, trim_to=0'0, roll_forward_to=42'18))", 2023-11-05T08:33:03.998 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:55.151821+0000", 2023-11-05T08:33:03.999 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.6749067619999991, 2023-11-05T08:33:03.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028565370000000001, 2023-11-05T08:33:03.999 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:03.999 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:03.999 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:03.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:04.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.151821+0000", 2023-11-05T08:33:04.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:04.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.151821+0000", 2023-11-05T08:33:04.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:04.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.151823+0000", 2023-11-05T08:33:04.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9460000000000001e-06 2023-11-05T08:33:04.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:04.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.151851+0000", 2023-11-05T08:33:04.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7333000000000001e-05 2023-11-05T08:33:04.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:04.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.151891+0000", 2023-11-05T08:33:04.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0784000000000001e-05 2023-11-05T08:33:04.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:04.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.151916+0000", 2023-11-05T08:33:04.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4436000000000001e-05 2023-11-05T08:33:04.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:04.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.152117+0000", 2023-11-05T08:33:04.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020099899999999999 2023-11-05T08:33:04.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:04.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.152142+0000", 2023-11-05T08:33:04.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5269000000000002e-05 2023-11-05T08:33:04.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:04.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.154567+0000", 2023-11-05T08:33:04.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024247589999999999 2023-11-05T08:33:04.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:04.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.154678+0000", 2023-11-05T08:33:04.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111011 2023-11-05T08:33:04.009 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.009 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:04.009 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.009 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.11s1 46/45 ECSubWrite(tid=0, reqid=osd.10.0:0, at_version=46'17, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:04.010 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:59.221890+0000", 2023-11-05T08:33:04.010 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.604838655, 2023-11-05T08:33:04.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0086406989999999999, 2023-11-05T08:33:04.010 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:04.010 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:04.010 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:04.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:04.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.221890+0000", 2023-11-05T08:33:04.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:04.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.221890+0000", 2023-11-05T08:33:04.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:04.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.221891+0000", 2023-11-05T08:33:04.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.536e-06 2023-11-05T08:33:04.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:04.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.221950+0000", 2023-11-05T08:33:04.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8615e-05 2023-11-05T08:33:04.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:04.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.221983+0000", 2023-11-05T08:33:04.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3225000000000001e-05 2023-11-05T08:33:04.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:04.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222006+0000", 2023-11-05T08:33:04.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3445999999999999e-05 2023-11-05T08:33:04.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:04.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222169+0000", 2023-11-05T08:33:04.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016258700000000001 2023-11-05T08:33:04.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:04.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222195+0000", 2023-11-05T08:33:04.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6041999999999999e-05 2023-11-05T08:33:04.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:04.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.230485+0000", 2023-11-05T08:33:04.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0082901520000000003 2023-11-05T08:33:04.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:04.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.230530+0000", 2023-11-05T08:33:04.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5096000000000003e-05 2023-11-05T08:33:04.020 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.020 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:04.020 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.020 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.11s1 46/45 ECSubWrite(tid=1, reqid=osd.10.0:0, at_version=0'0, trim_to=0'0, roll_forward_to=46'17))", 2023-11-05T08:33:04.021 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:59.232116+0000", 2023-11-05T08:33:04.021 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.5946117859999998, 2023-11-05T08:33:04.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0035202570000000002, 2023-11-05T08:33:04.021 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:04.021 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:04.021 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:04.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:04.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232116+0000", 2023-11-05T08:33:04.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:04.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232116+0000", 2023-11-05T08:33:04.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:04.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232118+0000", 2023-11-05T08:33:04.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.46e-06 2023-11-05T08:33:04.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:04.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232142+0000", 2023-11-05T08:33:04.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4221000000000001e-05 2023-11-05T08:33:04.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:04.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232166+0000", 2023-11-05T08:33:04.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3808e-05 2023-11-05T08:33:04.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:04.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232518+0000", 2023-11-05T08:33:04.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035172900000000001 2023-11-05T08:33:04.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:04.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232685+0000", 2023-11-05T08:33:04.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016766200000000001 2023-11-05T08:33:04.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:04.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232706+0000", 2023-11-05T08:33:04.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1087999999999999e-05 2023-11-05T08:33:04.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:04.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.235582+0000", 2023-11-05T08:33:04.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002875393 2023-11-05T08:33:04.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:04.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.235637+0000", 2023-11-05T08:33:04.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4895999999999998e-05 2023-11-05T08:33:04.031 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.031 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:04.031 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.031 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWriteReply(3.16s0 46/45 ECSubWriteReply(tid=48, last_complete=46'13, committed=1, applied=1))", 2023-11-05T08:33:04.032 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:59.241406+0000", 2023-11-05T08:33:04.032 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.5853223180000002, 2023-11-05T08:33:04.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030312730000000001, 2023-11-05T08:33:04.032 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:04.032 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:04.033 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:04.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:04.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.241406+0000", 2023-11-05T08:33:04.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:04.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.241406+0000", 2023-11-05T08:33:04.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:04.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.241407+0000", 2023-11-05T08:33:04.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2580000000000001e-06 2023-11-05T08:33:04.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:04.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.241416+0000", 2023-11-05T08:33:04.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.49e-06 2023-11-05T08:33:04.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:04.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.241436+0000", 2023-11-05T08:33:04.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0268999999999999e-05 2023-11-05T08:33:04.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:04.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.241450+0000", 2023-11-05T08:33:04.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4168999999999999e-05 2023-11-05T08:33:04.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:04.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.244260+0000", 2023-11-05T08:33:04.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028095780000000001 2023-11-05T08:33:04.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:04.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.244437+0000", 2023-11-05T08:33:04.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017750900000000001 2023-11-05T08:33:04.040 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.040 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:04.040 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.041 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWriteReply(3.16s0 46/45 ECSubWriteReply(tid=48, last_complete=46'13, committed=1, applied=1))", 2023-11-05T08:33:04.041 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:59.242037+0000", 2023-11-05T08:33:04.041 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.5846907009999995, 2023-11-05T08:33:04.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025841039999999998, 2023-11-05T08:33:04.042 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:04.042 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:04.042 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:04.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:04.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.242037+0000", 2023-11-05T08:33:04.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:04.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.242037+0000", 2023-11-05T08:33:04.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:04.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.242038+0000", 2023-11-05T08:33:04.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7100000000000011e-07 2023-11-05T08:33:04.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:04.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.242050+0000", 2023-11-05T08:33:04.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2054e-05 2023-11-05T08:33:04.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:04.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.242087+0000", 2023-11-05T08:33:04.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6776999999999997e-05 2023-11-05T08:33:04.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:04.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.242102+0000", 2023-11-05T08:33:04.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4844000000000001e-05 2023-11-05T08:33:04.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:04.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.244582+0000", 2023-11-05T08:33:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024800550000000001 2023-11-05T08:33:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.244622+0000", 2023-11-05T08:33:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9403000000000002e-05 2023-11-05T08:33:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:04.051 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:33:04.052 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2023-11-05T08:33:04.053 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:04.049+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:04.069 INFO:teuthology.orchestra.run.smithi090.stdout:get_pg_num_history pg_num_history pg_num_history(e43 pg_nums {1={22=1},2={25=8},3={30=16,40=26}} deleted_pools ) 2023-11-05T08:33:04.069 INFO:teuthology.orchestra.run.smithi090.stdout:Importing pgid 2.2 2023-11-05T08:33:04.069 INFO:teuthology.orchestra.run.smithi090.stdout:write_pg epoch 34 info 2.2( empty local-lis/les=33/34 n=0 ec=25/25 lis/c=33/33 les/c/f=34/34/0 sis=33) 2023-11-05T08:33:04.070 INFO:teuthology.orchestra.run.smithi090.stdout:Import successful 2023-11-05T08:33:04.153 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:33:04.148+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:33:04.175 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:33:04.175 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-11-05T08:33:04.176 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-11-05T08:33:04.176 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:33:04.189 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2023-11-05T08:33:04.224 INFO:tasks.rados.rados.0.smithi090.stdout:128: done (2 left) 2023-11-05T08:33:04.225 INFO:tasks.rados.rados.0.smithi090.stdout:129: read oid 6 snap -1 2023-11-05T08:33:04.225 INFO:tasks.rados.rados.0.smithi090.stdout:129: expect (ObjNum 5 snap 0 seq_num 5) 2023-11-05T08:33:04.225 INFO:tasks.rados.rados.0.smithi090.stdout:130: delete oid 11 current snap is 9 2023-11-05T08:33:04.225 INFO:tasks.rados.rados.0.smithi090.stdout:127: expect (ObjNum 6 snap 0 seq_num 6) 2023-11-05T08:33:04.235 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-11-05T08:33:04.242 INFO:tasks.rados.rados.0.smithi090.stdout:129: expect (ObjNum 5 snap 0 seq_num 5) 2023-11-05T08:33:04.253 INFO:tasks.rados.rados.0.smithi090.stdout:127: done (3 left) 2023-11-05T08:33:04.254 INFO:tasks.rados.rados.0.smithi090.stdout:129: done (2 left) 2023-11-05T08:33:04.254 INFO:tasks.rados.rados.0.smithi090.stdout:130: done (1 left) 2023-11-05T08:33:04.254 INFO:tasks.rados.rados.0.smithi090.stdout:131: snap_remove snap 8 2023-11-05T08:33:04.254 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:04.248+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:04.352 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:33:04.352 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-11-05T08:33:04.353 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-11-05T08:33:04.353 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-11-05T08:33:04.353 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:33:04.353 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-05T08:33:04.366 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2023-11-05T08:33:04.455 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:33:04.454+0000 7fdc5cd58640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:33:04.531 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:33:04.531 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-11-05T08:33:04.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-11-05T08:33:04.532 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-11-05T08:33:04.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.532 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.3s3 36/30 ECSubWrite(tid=183, reqid=osd.13.0:183, at_version=36'12, trim_to=0'0, roll_forward_to=35'10))", 2023-11-05T08:33:04.532 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.072455+0000", 2023-11-05T08:33:04.533 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.455476601999999, 2023-11-05T08:33:04.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0063400499999999999, 2023-11-05T08:33:04.533 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:04.533 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:04.533 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:04.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:04.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072455+0000", 2023-11-05T08:33:04.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:04.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072455+0000", 2023-11-05T08:33:04.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:04.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072456+0000", 2023-11-05T08:33:04.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9999999999999998e-07 2023-11-05T08:33:04.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:04.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072512+0000", 2023-11-05T08:33:04.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5667000000000001e-05 2023-11-05T08:33:04.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:04.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072537+0000", 2023-11-05T08:33:04.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5833000000000002e-05 2023-11-05T08:33:04.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:04.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072581+0000", 2023-11-05T08:33:04.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3470999999999998e-05 2023-11-05T08:33:04.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:04.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072752+0000", 2023-11-05T08:33:04.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017130100000000001 2023-11-05T08:33:04.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:04.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072775+0000", 2023-11-05T08:33:04.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3379000000000001e-05 2023-11-05T08:33:04.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:04.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.078646+0000", 2023-11-05T08:33:04.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0058709419999999997 2023-11-05T08:33:04.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:04.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.078795+0000", 2023-11-05T08:33:04.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014895699999999999 2023-11-05T08:33:04.543 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.543 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:04.543 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.544 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.8s5 36/30 ECSubWrite(tid=23, reqid=osd.5.0:23, at_version=36'15, trim_to=0'0, roll_forward_to=34'13))", 2023-11-05T08:33:04.544 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.079609+0000", 2023-11-05T08:33:04.544 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.448322900000001, 2023-11-05T08:33:04.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049579109999999997, 2023-11-05T08:33:04.544 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:04.545 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:04.545 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:04.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:04.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079609+0000", 2023-11-05T08:33:04.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:04.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079609+0000", 2023-11-05T08:33:04.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:04.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079609+0000", 2023-11-05T08:33:04.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.27e-07 2023-11-05T08:33:04.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:04.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079645+0000", 2023-11-05T08:33:04.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5522000000000003e-05 2023-11-05T08:33:04.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:04.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079672+0000", 2023-11-05T08:33:04.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6648e-05 2023-11-05T08:33:04.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:04.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079693+0000", 2023-11-05T08:33:04.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1206e-05 2023-11-05T08:33:04.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:04.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079849+0000", 2023-11-05T08:33:04.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156112 2023-11-05T08:33:04.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:04.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079873+0000", 2023-11-05T08:33:04.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3589000000000001e-05 2023-11-05T08:33:04.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:04.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.084477+0000", 2023-11-05T08:33:04.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0046040969999999997 2023-11-05T08:33:04.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:04.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.084567+0000", 2023-11-05T08:33:04.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0309999999999994e-05 2023-11-05T08:33:04.554 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.554 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:04.555 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.555 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.3s3 36/30 ECSubWrite(tid=184, reqid=osd.13.0:183, at_version=0'0, trim_to=0'0, roll_forward_to=36'12))", 2023-11-05T08:33:04.555 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.080288+0000", 2023-11-05T08:33:04.555 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.447644049999999, 2023-11-05T08:33:04.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002084163, 2023-11-05T08:33:04.556 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:04.556 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:04.556 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:04.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:04.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.080288+0000", 2023-11-05T08:33:04.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:04.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.080288+0000", 2023-11-05T08:33:04.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:04.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.080290+0000", 2023-11-05T08:33:04.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.663e-06 2023-11-05T08:33:04.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:04.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.080318+0000", 2023-11-05T08:33:04.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8195e-05 2023-11-05T08:33:04.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:04.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.080346+0000", 2023-11-05T08:33:04.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8419e-05 2023-11-05T08:33:04.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:04.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.080367+0000", 2023-11-05T08:33:04.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0559000000000001e-05 2023-11-05T08:33:04.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:04.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.080530+0000", 2023-11-05T08:33:04.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000163165 2023-11-05T08:33:04.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:04.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.080554+0000", 2023-11-05T08:33:04.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3935999999999999e-05 2023-11-05T08:33:04.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:04.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.082291+0000", 2023-11-05T08:33:04.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017370370000000001 2023-11-05T08:33:04.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:04.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.082372+0000", 2023-11-05T08:33:04.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1188999999999998e-05 2023-11-05T08:33:04.565 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.566 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:04.566 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.566 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.8s5 36/30 ECSubWrite(tid=24, reqid=osd.5.0:23, at_version=0'0, trim_to=0'0, roll_forward_to=36'15))", 2023-11-05T08:33:04.567 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.088806+0000", 2023-11-05T08:33:04.567 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.439125629999999, 2023-11-05T08:33:04.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055977739999999998, 2023-11-05T08:33:04.567 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:04.567 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:04.567 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:04.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:04.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.088806+0000", 2023-11-05T08:33:04.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:04.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.088806+0000", 2023-11-05T08:33:04.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:04.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.092263+0000", 2023-11-05T08:33:04.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034565210000000002 2023-11-05T08:33:04.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:04.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.092287+0000", 2023-11-05T08:33:04.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4505999999999999e-05 2023-11-05T08:33:04.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:04.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.092314+0000", 2023-11-05T08:33:04.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6335000000000001e-05 2023-11-05T08:33:04.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:04.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.092329+0000", 2023-11-05T08:33:04.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5486e-05 2023-11-05T08:33:04.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:04.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.092519+0000", 2023-11-05T08:33:04.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019026700000000001 2023-11-05T08:33:04.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:04.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.092545+0000", 2023-11-05T08:33:04.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5644999999999999e-05 2023-11-05T08:33:04.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:04.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.094327+0000", 2023-11-05T08:33:04.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001781543 2023-11-05T08:33:04.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:04.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.094404+0000", 2023-11-05T08:33:04.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7471000000000004e-05 2023-11-05T08:33:04.577 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.577 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:04.577 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.578 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.2s4 37/30 ECSubWrite(tid=185, reqid=client.4691.0:240, at_version=37'7, trim_to=0'0, roll_forward_to=33'5))", 2023-11-05T08:33:04.578 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:50.088409+0000", 2023-11-05T08:33:04.578 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 14.439523427999999, 2023-11-05T08:33:04.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036490530000000002, 2023-11-05T08:33:04.578 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:04.579 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:04.579 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:04.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:04.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.088409+0000", 2023-11-05T08:33:04.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:04.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.088409+0000", 2023-11-05T08:33:04.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:04.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.088409+0000", 2023-11-05T08:33:04.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.569999999999999e-07 2023-11-05T08:33:04.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:04.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.088461+0000", 2023-11-05T08:33:04.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1665000000000001e-05 2023-11-05T08:33:04.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:04.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.088486+0000", 2023-11-05T08:33:04.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4757999999999999e-05 2023-11-05T08:33:04.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:04.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.088505+0000", 2023-11-05T08:33:04.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8885999999999998e-05 2023-11-05T08:33:04.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:04.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.088651+0000", 2023-11-05T08:33:04.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014625900000000001 2023-11-05T08:33:04.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:04.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.088678+0000", 2023-11-05T08:33:04.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7229e-05 2023-11-05T08:33:04.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:04.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.091969+0000", 2023-11-05T08:33:04.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032904900000000001 2023-11-05T08:33:04.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:04.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.092058+0000", 2023-11-05T08:33:04.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8808999999999996e-05 2023-11-05T08:33:04.588 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.589 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:04.589 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.589 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.2s4 37/30 ECSubWrite(tid=186, reqid=client.4691.0:240, at_version=0'0, trim_to=0'0, roll_forward_to=37'7))", 2023-11-05T08:33:04.590 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:50.093109+0000", 2023-11-05T08:33:04.590 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 14.434822596, 2023-11-05T08:33:04.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002263953, 2023-11-05T08:33:04.590 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:04.590 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:04.591 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:04.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:04.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.093109+0000", 2023-11-05T08:33:04.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:04.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.093109+0000", 2023-11-05T08:33:04.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:04.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.093111+0000", 2023-11-05T08:33:04.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.443e-06 2023-11-05T08:33:04.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:04.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.093174+0000", 2023-11-05T08:33:04.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2730000000000004e-05 2023-11-05T08:33:04.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:04.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.093224+0000", 2023-11-05T08:33:04.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0518e-05 2023-11-05T08:33:04.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:04.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.093253+0000", 2023-11-05T08:33:04.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9258e-05 2023-11-05T08:33:04.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:04.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.093422+0000", 2023-11-05T08:33:04.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000168379 2023-11-05T08:33:04.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:04.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.093447+0000", 2023-11-05T08:33:04.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4998999999999999e-05 2023-11-05T08:33:04.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:04.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.095306+0000", 2023-11-05T08:33:04.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0018594659999999999 2023-11-05T08:33:04.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:04.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.095373+0000", 2023-11-05T08:33:04.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7160000000000001e-05 2023-11-05T08:33:04.600 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.600 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:04.600 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.601 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.2s4 41/40 ECSubWrite(tid=187, reqid=client.4691.0:292, at_version=41'9, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:04.601 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:54.162954+0000", 2023-11-05T08:33:04.601 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 10.364977506000001, 2023-11-05T08:33:04.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0045082990000000003, 2023-11-05T08:33:04.601 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:04.601 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:04.602 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:04.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:04.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.162954+0000", 2023-11-05T08:33:04.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:04.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.162954+0000", 2023-11-05T08:33:04.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:04.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.162956+0000", 2023-11-05T08:33:04.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.86e-06 2023-11-05T08:33:04.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:04.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.163012+0000", 2023-11-05T08:33:04.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5744e-05 2023-11-05T08:33:04.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:04.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.163048+0000", 2023-11-05T08:33:04.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6346000000000001e-05 2023-11-05T08:33:04.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:04.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.163075+0000", 2023-11-05T08:33:04.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6622e-05 2023-11-05T08:33:04.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:04.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.163236+0000", 2023-11-05T08:33:04.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016064599999999999 2023-11-05T08:33:04.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:04.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.163262+0000", 2023-11-05T08:33:04.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6356e-05 2023-11-05T08:33:04.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:04.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.167302+0000", 2023-11-05T08:33:04.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040403000000000001 2023-11-05T08:33:04.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:04.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.167463+0000", 2023-11-05T08:33:04.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016042500000000001 2023-11-05T08:33:04.611 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.611 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:04.611 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.612 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpRead(3.18s5 41/40 ECSubRead(tid=25, to_read={3:1aa38538:::smithi09039252-11:head=0,274432,0}, subchunks={3:1aa38538:::smithi09039252-11:head=[0,1]}, attrs_to_read=))", 2023-11-05T08:33:04.612 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:54.170684+0000", 2023-11-05T08:33:04.612 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 10.357247967999999, 2023-11-05T08:33:04.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025613950000000002, 2023-11-05T08:33:04.613 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:04.613 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:04.613 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:04.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:04.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.170684+0000", 2023-11-05T08:33:04.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:04.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.170684+0000", 2023-11-05T08:33:04.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:04.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.170685+0000", 2023-11-05T08:33:04.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5899999999999995e-07 2023-11-05T08:33:04.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:04.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.170692+0000", 2023-11-05T08:33:04.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5039999999999996e-06 2023-11-05T08:33:04.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:04.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.170703+0000", 2023-11-05T08:33:04.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.099e-05 2023-11-05T08:33:04.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:04.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.170710+0000", 2023-11-05T08:33:04.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9779999999999999e-06 2023-11-05T08:33:04.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:04.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.170768+0000", 2023-11-05T08:33:04.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7862000000000003e-05 2023-11-05T08:33:04.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:04.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.173245+0000", 2023-11-05T08:33:04.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002477202 2023-11-05T08:33:04.622 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.622 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:04.622 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.623 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.2s4 41/40 ECSubWrite(tid=188, reqid=osd.13.0:188, at_version=41'11, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:04.623 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:54.171988+0000", 2023-11-05T08:33:04.623 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 10.355943976000001, 2023-11-05T08:33:04.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037888499999999999, 2023-11-05T08:33:04.623 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:04.624 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:04.624 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:04.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:04.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.171988+0000", 2023-11-05T08:33:04.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:04.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.171988+0000", 2023-11-05T08:33:04.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:04.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.171988+0000", 2023-11-05T08:33:04.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1400000000000001e-07 2023-11-05T08:33:04.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:04.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.172004+0000", 2023-11-05T08:33:04.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6308e-05 2023-11-05T08:33:04.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:04.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.172011+0000", 2023-11-05T08:33:04.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9430000000000003e-06 2023-11-05T08:33:04.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:04.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.172018+0000", 2023-11-05T08:33:04.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6250000000000001e-06 2023-11-05T08:33:04.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:04.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.172082+0000", 2023-11-05T08:33:04.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4311000000000001e-05 2023-11-05T08:33:04.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:04.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.172093+0000", 2023-11-05T08:33:04.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1045e-05 2023-11-05T08:33:04.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:04.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.175705+0000", 2023-11-05T08:33:04.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036118819999999999 2023-11-05T08:33:04.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:04.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.175777+0000", 2023-11-05T08:33:04.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1521999999999997e-05 2023-11-05T08:33:04.633 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.633 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:04.634 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.634 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.13s3 41/40 ECSubWrite(tid=190, reqid=osd.13.0:190, at_version=41'14, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:04.634 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:54.189971+0000", 2023-11-05T08:33:04.635 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 10.337960773000001, 2023-11-05T08:33:04.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003736746, 2023-11-05T08:33:04.635 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:04.635 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:04.635 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:04.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:04.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.189971+0000", 2023-11-05T08:33:04.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:04.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.189971+0000", 2023-11-05T08:33:04.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:04.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.189972+0000", 2023-11-05T08:33:04.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4900000000000005e-07 2023-11-05T08:33:04.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:04.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.190023+0000", 2023-11-05T08:33:04.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1162999999999998e-05 2023-11-05T08:33:04.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:04.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.190044+0000", 2023-11-05T08:33:04.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0761999999999999e-05 2023-11-05T08:33:04.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:04.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.190062+0000", 2023-11-05T08:33:04.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7977000000000001e-05 2023-11-05T08:33:04.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:04.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.190138+0000", 2023-11-05T08:33:04.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5907000000000004e-05 2023-11-05T08:33:04.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:04.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.190148+0000", 2023-11-05T08:33:04.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0344e-05 2023-11-05T08:33:04.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:04.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.193643+0000", 2023-11-05T08:33:04.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003494718 2023-11-05T08:33:04.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:04.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.193708+0000", 2023-11-05T08:33:04.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5025999999999997e-05 2023-11-05T08:33:04.645 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.646 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:04.646 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.646 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.2s4 42/40 ECSubWrite(tid=192, reqid=osd.13.0:192, at_version=42'13, trim_to=0'0, roll_forward_to=41'11))", 2023-11-05T08:33:04.647 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:55.150758+0000", 2023-11-05T08:33:04.647 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 9.3771744169999991, 2023-11-05T08:33:04.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032901889999999998, 2023-11-05T08:33:04.647 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:04.647 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:04.648 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:04.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:04.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.150758+0000", 2023-11-05T08:33:04.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:04.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.150758+0000", 2023-11-05T08:33:04.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:04.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.150758+0000", 2023-11-05T08:33:04.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9999999999999998e-07 2023-11-05T08:33:04.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:04.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.150792+0000", 2023-11-05T08:33:04.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4369999999999998e-05 2023-11-05T08:33:04.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:04.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.150820+0000", 2023-11-05T08:33:04.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7285999999999999e-05 2023-11-05T08:33:04.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:04.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.150833+0000", 2023-11-05T08:33:04.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3603e-05 2023-11-05T08:33:04.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:04.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.150981+0000", 2023-11-05T08:33:04.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000148028 2023-11-05T08:33:04.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:04.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.151008+0000", 2023-11-05T08:33:04.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7169e-05 2023-11-05T08:33:04.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:04.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.153963+0000", 2023-11-05T08:33:04.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029548700000000001 2023-11-05T08:33:04.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:04.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.154048+0000", 2023-11-05T08:33:04.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4363000000000005e-05 2023-11-05T08:33:04.657 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.657 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:04.657 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.658 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.16s1 46/45 ECSubWrite(tid=45, reqid=osd.11.0:45, at_version=46'11, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:04.658 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:59.219281+0000", 2023-11-05T08:33:04.658 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.3086504909999999, 2023-11-05T08:33:04.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.005540456, 2023-11-05T08:33:04.659 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:04.659 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:04.659 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:04.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:04.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219281+0000", 2023-11-05T08:33:04.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:04.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219281+0000", 2023-11-05T08:33:04.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:04.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219283+0000", 2023-11-05T08:33:04.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.637e-06 2023-11-05T08:33:04.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:04.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219344+0000", 2023-11-05T08:33:04.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0831000000000001e-05 2023-11-05T08:33:04.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:04.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219376+0000", 2023-11-05T08:33:04.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1945e-05 2023-11-05T08:33:04.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:04.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219417+0000", 2023-11-05T08:33:04.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1335000000000003e-05 2023-11-05T08:33:04.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:04.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219601+0000", 2023-11-05T08:33:04.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018345699999999999 2023-11-05T08:33:04.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:04.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219627+0000", 2023-11-05T08:33:04.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6732000000000001e-05 2023-11-05T08:33:04.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:04.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.224749+0000", 2023-11-05T08:33:04.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0051213129999999997 2023-11-05T08:33:04.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:04.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.224822+0000", 2023-11-05T08:33:04.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3206000000000003e-05 2023-11-05T08:33:04.668 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.668 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:04.669 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.669 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.11s2 46/45 ECSubWrite(tid=0, reqid=osd.10.0:0, at_version=46'17, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:04.669 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:59.222006+0000", 2023-11-05T08:33:04.670 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.3059257989999997, 2023-11-05T08:33:04.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053541609999999996, 2023-11-05T08:33:04.670 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:04.670 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:04.670 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:04.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:04.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222006+0000", 2023-11-05T08:33:04.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:04.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222006+0000", 2023-11-05T08:33:04.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:04.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222008+0000", 2023-11-05T08:33:04.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7430000000000001e-06 2023-11-05T08:33:04.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:04.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222062+0000", 2023-11-05T08:33:04.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3891000000000003e-05 2023-11-05T08:33:04.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:04.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222095+0000", 2023-11-05T08:33:04.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3148999999999997e-05 2023-11-05T08:33:04.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:04.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222118+0000", 2023-11-05T08:33:04.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3258999999999999e-05 2023-11-05T08:33:04.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:04.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222279+0000", 2023-11-05T08:33:04.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016113 2023-11-05T08:33:04.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:04.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222303+0000", 2023-11-05T08:33:04.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4088999999999999e-05 2023-11-05T08:33:04.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:04.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.227275+0000", 2023-11-05T08:33:04.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049715599999999999 2023-11-05T08:33:04.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:04.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.227360+0000", 2023-11-05T08:33:04.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5340000000000006e-05 2023-11-05T08:33:04.680 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.680 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:04.680 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.681 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.16s1 46/45 ECSubWrite(tid=46, reqid=osd.11.0:45, at_version=0'0, trim_to=0'0, roll_forward_to=46'11))", 2023-11-05T08:33:04.681 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:59.228866+0000", 2023-11-05T08:33:04.681 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.2990663720000004, 2023-11-05T08:33:04.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024106370000000002, 2023-11-05T08:33:04.681 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.228866+0000", 2023-11-05T08:33:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.228866+0000", 2023-11-05T08:33:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.228867+0000", 2023-11-05T08:33:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.395e-06 2023-11-05T08:33:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.228892+0000", 2023-11-05T08:33:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4723999999999999e-05 2023-11-05T08:33:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.228916+0000", 2023-11-05T08:33:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4159e-05 2023-11-05T08:33:04.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:04.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.228933+0000", 2023-11-05T08:33:04.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7037e-05 2023-11-05T08:33:04.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:04.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.229066+0000", 2023-11-05T08:33:04.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013269599999999999 2023-11-05T08:33:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.229086+0000", 2023-11-05T08:33:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0353999999999999e-05 2023-11-05T08:33:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:04.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.231210+0000", 2023-11-05T08:33:04.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021238300000000002 2023-11-05T08:33:04.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:04.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.231276+0000", 2023-11-05T08:33:04.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6441999999999998e-05 2023-11-05T08:33:04.691 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.692 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:04.692 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.692 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.11s2 46/45 ECSubWrite(tid=1, reqid=osd.10.0:0, at_version=0'0, trim_to=0'0, roll_forward_to=46'17))", 2023-11-05T08:33:04.692 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:59.232093+0000", 2023-11-05T08:33:04.693 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.295839323, 2023-11-05T08:33:04.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037473570000000002, 2023-11-05T08:33:04.693 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:04.693 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:04.693 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:04.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:04.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232093+0000", 2023-11-05T08:33:04.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:04.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232093+0000", 2023-11-05T08:33:04.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:04.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232094+0000", 2023-11-05T08:33:04.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0380000000000001e-06 2023-11-05T08:33:04.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:04.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232122+0000", 2023-11-05T08:33:04.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8585000000000002e-05 2023-11-05T08:33:04.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:04.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232142+0000", 2023-11-05T08:33:04.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9357000000000001e-05 2023-11-05T08:33:04.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:04.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232157+0000", 2023-11-05T08:33:04.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5141e-05 2023-11-05T08:33:04.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:04.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232272+0000", 2023-11-05T08:33:04.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114945 2023-11-05T08:33:04.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:04.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232285+0000", 2023-11-05T08:33:04.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2917e-05 2023-11-05T08:33:04.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:04.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.235777+0000", 2023-11-05T08:33:04.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003492069 2023-11-05T08:33:04.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:04.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.235840+0000", 2023-11-05T08:33:04.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3305000000000005e-05 2023-11-05T08:33:04.703 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.703 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:04.703 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.704 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.16s1 46/45 ECSubWrite(tid=47, reqid=osd.11.0:47, at_version=46'13, trim_to=0'0, roll_forward_to=46'11))", 2023-11-05T08:33:04.704 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:59.233948+0000", 2023-11-05T08:33:04.704 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.29398357, 2023-11-05T08:33:04.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048084130000000001, 2023-11-05T08:33:04.705 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:04.705 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:04.705 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:04.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:04.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.233948+0000", 2023-11-05T08:33:04.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:04.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.233948+0000", 2023-11-05T08:33:04.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:04.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.233950+0000", 2023-11-05T08:33:04.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5969999999999999e-06 2023-11-05T08:33:04.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:04.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.234005+0000", 2023-11-05T08:33:04.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5414999999999997e-05 2023-11-05T08:33:04.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:04.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.234040+0000", 2023-11-05T08:33:04.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.455e-05 2023-11-05T08:33:04.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:04.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.234061+0000", 2023-11-05T08:33:04.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1036e-05 2023-11-05T08:33:04.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:04.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.234178+0000", 2023-11-05T08:33:04.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000116822 2023-11-05T08:33:04.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:04.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.234195+0000", 2023-11-05T08:33:04.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6801999999999999e-05 2023-11-05T08:33:04.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:04.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.238676+0000", 2023-11-05T08:33:04.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004481044 2023-11-05T08:33:04.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:04.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.238757+0000", 2023-11-05T08:33:04.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1147000000000001e-05 2023-11-05T08:33:04.714 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.715 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:04.715 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.715 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.16s1 46/45 ECSubWrite(tid=48, reqid=osd.11.0:47, at_version=0'0, trim_to=0'0, roll_forward_to=46'13))", 2023-11-05T08:33:04.716 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:59.240171+0000", 2023-11-05T08:33:04.716 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.2877609630000002, 2023-11-05T08:33:04.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0045632529999999998, 2023-11-05T08:33:04.716 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:04.716 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:04.716 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:04.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:04.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240171+0000", 2023-11-05T08:33:04.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:04.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240171+0000", 2023-11-05T08:33:04.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:04.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240172+0000", 2023-11-05T08:33:04.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.282e-06 2023-11-05T08:33:04.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:04.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240193+0000", 2023-11-05T08:33:04.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0601000000000001e-05 2023-11-05T08:33:04.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:04.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240214+0000", 2023-11-05T08:33:04.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1574999999999999e-05 2023-11-05T08:33:04.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:04.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240230+0000", 2023-11-05T08:33:04.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5656999999999999e-05 2023-11-05T08:33:04.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:04.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240361+0000", 2023-11-05T08:33:04.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000130488 2023-11-05T08:33:04.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:04.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240372+0000", 2023-11-05T08:33:04.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1158999999999999e-05 2023-11-05T08:33:04.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:04.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.244663+0000", 2023-11-05T08:33:04.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004291563 2023-11-05T08:33:04.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:04.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.244734+0000", 2023-11-05T08:33:04.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0927999999999996e-05 2023-11-05T08:33:04.726 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.726 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:04.726 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.727 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.14s2 47/46 [PushOp(3:292b6c2e:::smithi09039252-40:8, version: 41'13, data_included: [524288~163840], data_size: 163840, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:292b6c2e:::smithi09039252-40:8@41'13, size: 2752512, copy_subset: [], clone_subset: {}, snapset: 8=[]:{7: [7, 6, 5, 3], 8: [8]}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2752512, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2023-11-05T08:33:04.727 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:00.278142+0000", 2023-11-05T08:33:04.727 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.2497902569999999, 2023-11-05T08:33:04.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020573710000000001, 2023-11-05T08:33:04.728 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:04.728 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:04.728 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:04.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:04.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:00.278142+0000", 2023-11-05T08:33:04.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:04.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:00.278142+0000", 2023-11-05T08:33:04.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:04.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:00.278142+0000", 2023-11-05T08:33:04.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1200000000000002e-07 2023-11-05T08:33:04.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:04.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:00.278332+0000", 2023-11-05T08:33:04.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018975900000000001 2023-11-05T08:33:04.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:04.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:00.278351+0000", 2023-11-05T08:33:04.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.878e-05 2023-11-05T08:33:04.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:04.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:00.278368+0000", 2023-11-05T08:33:04.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7467000000000001e-05 2023-11-05T08:33:04.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:04.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:00.278550+0000", 2023-11-05T08:33:04.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018185400000000001 2023-11-05T08:33:04.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:04.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:00.280199+0000", 2023-11-05T08:33:04.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016487990000000001 2023-11-05T08:33:04.736 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.736 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:04.736 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.737 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDScrubReserve(3.0s2 RELEASE e50)", 2023-11-05T08:33:04.737 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:03.810656+0000", 2023-11-05T08:33:04.737 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.71727622099999999, 2023-11-05T08:33:04.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0027462369999999999, 2023-11-05T08:33:04.737 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:04.737 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:04.738 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:04.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:04.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:03.810656+0000", 2023-11-05T08:33:04.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:04.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:03.810656+0000", 2023-11-05T08:33:04.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:04.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:03.810656+0000", 2023-11-05T08:33:04.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4300000000000003e-07 2023-11-05T08:33:04.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:04.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:03.810660+0000", 2023-11-05T08:33:04.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9500000000000003e-06 2023-11-05T08:33:04.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:04.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:03.810682+0000", 2023-11-05T08:33:04.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2086000000000002e-05 2023-11-05T08:33:04.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:04.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:03.810695+0000", 2023-11-05T08:33:04.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2809000000000001e-05 2023-11-05T08:33:04.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:04.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:03.810761+0000", 2023-11-05T08:33:04.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5746000000000004e-05 2023-11-05T08:33:04.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-11-05T08:33:04.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:03.810769+0000", 2023-11-05T08:33:04.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5029999999999997e-06 2023-11-05T08:33:04.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:04.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:03.813324+0000", 2023-11-05T08:33:04.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025551229999999999 2023-11-05T08:33:04.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:04.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:03.813365+0000", 2023-11-05T08:33:04.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0179000000000003e-05 2023-11-05T08:33:04.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:04.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:03.813402+0000", 2023-11-05T08:33:04.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7298000000000001e-05 2023-11-05T08:33:04.748 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.748 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:04.748 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.749 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.16s1 51/45 ECSubWrite(tid=58, reqid=osd.11.0:58, at_version=51'19, trim_to=0'0, roll_forward_to=49'17))", 2023-11-05T08:33:04.749 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:04.226221+0000", 2023-11-05T08:33:04.749 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.30171119099999999, 2023-11-05T08:33:04.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.005713414, 2023-11-05T08:33:04.750 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:04.750 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:04.750 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:04.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:04.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226221+0000", 2023-11-05T08:33:04.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:04.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226221+0000", 2023-11-05T08:33:04.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:04.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:04.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226221+0000", 2023-11-05T08:33:04.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.53e-07 2023-11-05T08:33:04.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:04.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226253+0000", 2023-11-05T08:33:04.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1671999999999998e-05 2023-11-05T08:33:04.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:04.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226269+0000", 2023-11-05T08:33:04.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6150999999999998e-05 2023-11-05T08:33:04.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:04.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226282+0000", 2023-11-05T08:33:04.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3465000000000001e-05 2023-11-05T08:33:04.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:04.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226474+0000", 2023-11-05T08:33:04.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019140499999999999 2023-11-05T08:33:04.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:04.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226501+0000", 2023-11-05T08:33:04.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7523000000000001e-05 2023-11-05T08:33:04.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.758 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:04.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.231902+0000", 2023-11-05T08:33:04.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0054011019999999996 2023-11-05T08:33:04.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:04.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:04.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:04.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.231934+0000", 2023-11-05T08:33:04.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1742999999999997e-05 2023-11-05T08:33:04.759 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.760 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:04.760 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.760 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:04.760 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:04.760 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:33:04.762 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-11-05T08:33:04.763 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f /var/log/ceph/exp.2.2.3 2023-11-05T08:33:04.764 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:33:04.759+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:33:04.863 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:33:04.863+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:33:04.883 INFO:teuthology.orchestra.run.smithi002.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-05T08:33:04.885 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-05T08:33:04.886 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-11-05T08:33:04.964 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:33:04.963+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:33:04.995 INFO:teuthology.orchestra.run.smithi002.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-05T08:33:04.997 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-05T08:33:04.998 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-11-05T08:33:05.065 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:33:05.062+0000 7fdc5cd58640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:33:05.102 INFO:teuthology.orchestra.run.smithi002.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-05T08:33:05.105 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-05T08:33:05.106 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-11-05T08:33:05.166 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:33:05.164+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:33:05.229 INFO:teuthology.orchestra.run.smithi002.stderr:noscrub is set 2023-11-05T08:33:05.238 INFO:tasks.rados.rados.0.smithi090.stdout:131: done (1 left) 2023-11-05T08:33:05.238 INFO:tasks.rados.rados.0.smithi090.stdout:132: delete oid 23 current snap is 9 2023-11-05T08:33:05.249 INFO:tasks.rados.rados.0.smithi090.stdout:132: done (1 left) 2023-11-05T08:33:05.250 INFO:tasks.rados.rados.0.smithi090.stdout:133: rmattr oid 30 current snap is 9 2023-11-05T08:33:05.250 INFO:tasks.rados.rados.0.smithi090.stdout:133: done (1 left) 2023-11-05T08:33:05.250 INFO:tasks.rados.rados.0.smithi090.stdout:134: read oid 17 snap -1 2023-11-05T08:33:05.250 INFO:tasks.rados.rados.0.smithi090.stdout:134: expect deleted 2023-11-05T08:33:05.251 INFO:tasks.rados.rados.0.smithi090.stdout:135: setattr oid 13 current snap is 9 2023-11-05T08:33:05.254 INFO:tasks.rados.rados.0.smithi090.stdout:136: copy_from oid 35 from oid 5 current snap is 9 2023-11-05T08:33:05.254 INFO:tasks.rados.rados.0.smithi090.stdout:137: copy_from oid 3 from oid 33 current snap is 9 2023-11-05T08:33:05.255 INFO:tasks.rados.rados.0.smithi090.stdout:138: read oid 39 snap -1 2023-11-05T08:33:05.255 INFO:tasks.rados.rados.0.smithi090.stdout:138: expect (ObjNum 38 snap 0 seq_num 38) 2023-11-05T08:33:05.255 INFO:tasks.rados.rados.0.smithi090.stdout:139: copy_from oid 15 from oid 18 current snap is 9 2023-11-05T08:33:05.255 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 11 current snap is 9 2023-11-05T08:33:05.255 INFO:tasks.rados.rados.0.smithi090.stdout:140: seq_num 75 ranges {0=491520,491520=770048} 2023-11-05T08:33:05.263 INFO:tasks.rados.rados.0.smithi090.stdout:140: writing smithi09039252-11 from 0 to 491520 tid 1 2023-11-05T08:33:05.271 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:33:05.265+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:33:05.278 INFO:tasks.rados.rados.0.smithi090.stdout:140: writing smithi09039252-11 from 491520 to 1261568 tid 2 2023-11-05T08:33:05.279 INFO:tasks.rados.rados.0.smithi090.stdout:141: read oid 43 snap -1 2023-11-05T08:33:05.279 INFO:tasks.rados.rados.0.smithi090.stdout:141: expect (ObjNum 67 snap 6 seq_num 67) 2023-11-05T08:33:05.279 INFO:tasks.rados.rados.0.smithi090.stdout:139: finishing copy_from racing read to smithi09039252-15 2023-11-05T08:33:05.280 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 13 v 24 (ObjNum 50 snap 0 seq_num 50) dirty exists 2023-11-05T08:33:05.280 INFO:tasks.rados.rados.0.smithi090.stdout:139: finishing copy_from to smithi09039252-15 2023-11-05T08:33:05.280 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 15 v 2 (ObjNum 17 snap 0 seq_num 17) dirty exists 2023-11-05T08:33:05.280 INFO:tasks.rados.rados.0.smithi090.stdout:138: expect (ObjNum 38 snap 0 seq_num 38) 2023-11-05T08:33:05.285 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:33:05.285 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [], 2023-11-05T08:33:05.285 INFO:teuthology.orchestra.run.smithi002.stdout: "num_ops": 0 2023-11-05T08:33:05.286 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:33:05.301 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-11-05T08:33:05.331 INFO:tasks.rados.rados.0.smithi090.stdout:137: finishing copy_from racing read to smithi09039252-3 2023-11-05T08:33:05.331 INFO:tasks.rados.rados.0.smithi090.stdout:137: finishing copy_from to smithi09039252-3 2023-11-05T08:33:05.331 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 3 v 4 (ObjNum 32 snap 0 seq_num 32) dirty exists 2023-11-05T08:33:05.331 INFO:tasks.rados.rados.0.smithi090.stdout:140: finishing write tid 1 to smithi09039252-11 2023-11-05T08:33:05.332 INFO:tasks.rados.rados.0.smithi090.stdout:140: finishing write tid 2 to smithi09039252-11 2023-11-05T08:33:05.332 INFO:tasks.rados.rados.0.smithi090.stdout:134: done (8 left) 2023-11-05T08:33:05.332 INFO:tasks.rados.rados.0.smithi090.stdout:135: done (7 left) 2023-11-05T08:33:05.332 INFO:tasks.rados.rados.0.smithi090.stdout:137: done (6 left) 2023-11-05T08:33:05.332 INFO:tasks.rados.rados.0.smithi090.stdout:138: done (5 left) 2023-11-05T08:33:05.333 INFO:tasks.rados.rados.0.smithi090.stdout:139: done (4 left) 2023-11-05T08:33:05.333 INFO:tasks.rados.rados.0.smithi090.stdout:142: snap_create 2023-11-05T08:33:05.333 INFO:tasks.rados.rados.0.smithi090.stdout:140: finishing write tid 3 to smithi09039252-11 2023-11-05T08:33:05.333 INFO:tasks.rados.rados.0.smithi090.stdout:136: finishing copy_from racing read to smithi09039252-35 2023-11-05T08:33:05.333 INFO:tasks.rados.rados.0.smithi090.stdout:141: expect (ObjNum 67 snap 6 seq_num 67) 2023-11-05T08:33:05.368 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:33:05.367+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:33:05.415 INFO:tasks.rados.rados.0.smithi090.stdout:136: finishing copy_from to smithi09039252-35 2023-11-05T08:33:05.415 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 35 v 10 (ObjNum 40 snap 0 seq_num 40) dirty exists 2023-11-05T08:33:05.415 INFO:tasks.rados.rados.0.smithi090.stdout:140: finishing write tid 4 to smithi09039252-11 2023-11-05T08:33:05.416 INFO:tasks.rados.rados.0.smithi090.stdout:140: oid 11 updating version 0 to 24 2023-11-05T08:33:05.416 INFO:tasks.rados.rados.0.smithi090.stdout:140: oid 11 updating version 24 to 26 2023-11-05T08:33:05.416 INFO:tasks.rados.rados.0.smithi090.stdout:140: oid 11 version 26 is already newer than 25 2023-11-05T08:33:05.416 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 11 v 26 (ObjNum 75 snap 9 seq_num 75) dirty exists 2023-11-05T08:33:05.416 INFO:tasks.rados.rados.0.smithi090.stdout:140: left oid 11 (ObjNum 75 snap 9 seq_num 75) 2023-11-05T08:33:05.459 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:33:05.460 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [], 2023-11-05T08:33:05.460 INFO:teuthology.orchestra.run.smithi002.stdout: "complaint_time": 30, 2023-11-05T08:33:05.460 INFO:teuthology.orchestra.run.smithi002.stdout: "num_blocked_ops": 0 2023-11-05T08:33:05.460 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:33:05.469 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:33:05.466+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:33:05.474 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-11-05T08:33:05.515 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-05T08:33:05.570 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:05.565+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:05.642 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:33:05.643 INFO:teuthology.orchestra.run.smithi002.stdout: "size": 20, 2023-11-05T08:33:05.643 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 600, 2023-11-05T08:33:05.643 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [ 2023-11-05T08:33:05.643 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.644 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.5s2 35/30 ECSubWrite(tid=20, reqid=client.4691.0:221, at_version=35'11, trim_to=0'0, roll_forward_to=33'9))", 2023-11-05T08:33:05.644 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:48.086738+0000", 2023-11-05T08:33:05.644 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 17.555237449, 2023-11-05T08:33:05.644 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0065328219999999998, 2023-11-05T08:33:05.644 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:05.645 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:05.645 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:05.645 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.645 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:05.645 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.086738+0000", 2023-11-05T08:33:05.646 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.646 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.646 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.646 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:05.646 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.086738+0000", 2023-11-05T08:33:05.647 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.647 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.647 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.647 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:05.648 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.086739+0000", 2023-11-05T08:33:05.648 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.22e-07 2023-11-05T08:33:05.648 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.648 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.648 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:05.649 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.087297+0000", 2023-11-05T08:33:05.649 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00055798899999999995 2023-11-05T08:33:05.649 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.649 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.649 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:05.650 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.087338+0000", 2023-11-05T08:33:05.650 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.1171000000000003e-05 2023-11-05T08:33:05.650 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.650 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.650 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:05.650 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.087364+0000", 2023-11-05T08:33:05.651 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.5826999999999999e-05 2023-11-05T08:33:05.651 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.651 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.651 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:05.651 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.087572+0000", 2023-11-05T08:33:05.652 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000208574 2023-11-05T08:33:05.652 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.652 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.652 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:05.652 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.087598+0000", 2023-11-05T08:33:05.652 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.5901000000000002e-05 2023-11-05T08:33:05.653 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.653 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.653 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:05.653 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.093224+0000", 2023-11-05T08:33:05.653 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0056260210000000001 2023-11-05T08:33:05.654 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.654 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.654 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:05.654 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.093271+0000", 2023-11-05T08:33:05.654 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.7117000000000003e-05 2023-11-05T08:33:05.655 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.655 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:05.655 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.655 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.655 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.656 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.5s2 35/30 ECSubWrite(tid=21, reqid=client.4691.0:222, at_version=35'12, trim_to=0'0, roll_forward_to=33'9))", 2023-11-05T08:33:05.656 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:48.091862+0000", 2023-11-05T08:33:05.656 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 17.550113975999999, 2023-11-05T08:33:05.656 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.003284333, 2023-11-05T08:33:05.656 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:05.656 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:05.657 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:05.657 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.657 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:05.658 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.091862+0000", 2023-11-05T08:33:05.658 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.658 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.658 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.659 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:05.659 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.091862+0000", 2023-11-05T08:33:05.659 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.659 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.659 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.659 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:05.660 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.091863+0000", 2023-11-05T08:33:05.660 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.237e-06 2023-11-05T08:33:05.660 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.660 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.660 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:05.661 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.092010+0000", 2023-11-05T08:33:05.661 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00014685699999999999 2023-11-05T08:33:05.661 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.661 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.662 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:05.662 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.092034+0000", 2023-11-05T08:33:05.662 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.4596e-05 2023-11-05T08:33:05.662 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.662 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.662 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:05.663 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.092047+0000", 2023-11-05T08:33:05.663 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.2841e-05 2023-11-05T08:33:05.663 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.663 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.663 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:05.663 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.092166+0000", 2023-11-05T08:33:05.664 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000118317 2023-11-05T08:33:05.664 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.664 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.664 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:05.664 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.092182+0000", 2023-11-05T08:33:05.665 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.5852999999999999e-05 2023-11-05T08:33:05.665 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.665 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.665 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:05.665 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.095105+0000", 2023-11-05T08:33:05.665 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00292312 2023-11-05T08:33:05.666 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.666 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.666 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:05.666 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.095146+0000", 2023-11-05T08:33:05.666 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.1511999999999998e-05 2023-11-05T08:33:05.667 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.667 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:05.667 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.667 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.667 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.668 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.ds6 36/30 ECSubWrite(tid=20, reqid=client.4691.0:233, at_version=36'10, trim_to=0'0, roll_forward_to=34'8))", 2023-11-05T08:33:05.668 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:49.067390+0000", 2023-11-05T08:33:05.668 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 16.574586068999999, 2023-11-05T08:33:05.668 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0062948609999999997, 2023-11-05T08:33:05.668 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:05.668 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:05.669 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:05.669 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.669 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:05.669 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.067390+0000", 2023-11-05T08:33:05.669 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.670 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.670 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.670 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:05.670 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.067390+0000", 2023-11-05T08:33:05.670 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.671 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.671 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.671 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:05.671 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.067390+0000", 2023-11-05T08:33:05.671 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.9700000000000003e-07 2023-11-05T08:33:05.671 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.672 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.672 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:05.672 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.067421+0000", 2023-11-05T08:33:05.672 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.1350000000000003e-05 2023-11-05T08:33:05.672 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.672 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.673 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:05.673 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.067439+0000", 2023-11-05T08:33:05.673 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.7833e-05 2023-11-05T08:33:05.673 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.673 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.674 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:05.674 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.067452+0000", 2023-11-05T08:33:05.674 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.2318e-05 2023-11-05T08:33:05.674 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.674 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.674 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:05.675 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.067542+0000", 2023-11-05T08:33:05.675 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.0628e-05 2023-11-05T08:33:05.675 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.675 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.675 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:05.675 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.067552+0000", 2023-11-05T08:33:05.676 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0275000000000001e-05 2023-11-05T08:33:05.676 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.676 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.676 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:05.676 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.073642+0000", 2023-11-05T08:33:05.676 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.006089779 2023-11-05T08:33:05.677 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.677 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.677 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:05.677 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.073685+0000", 2023-11-05T08:33:05.677 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.2381e-05 2023-11-05T08:33:05.677 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.678 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:05.678 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.678 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.678 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.678 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.3s2 36/30 ECSubWrite(tid=183, reqid=osd.13.0:183, at_version=36'12, trim_to=0'0, roll_forward_to=35'10))", 2023-11-05T08:33:05.679 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:49.071402+0000", 2023-11-05T08:33:05.679 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 16.570573628999998, 2023-11-05T08:33:05.679 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0050718050000000004, 2023-11-05T08:33:05.679 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:05.679 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:05.680 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:05.680 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.680 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:05.680 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.071402+0000", 2023-11-05T08:33:05.680 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.681 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.681 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.681 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:05.681 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.071402+0000", 2023-11-05T08:33:05.681 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.681 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.682 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.682 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:05.682 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.071403+0000", 2023-11-05T08:33:05.682 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.7599999999999998e-07 2023-11-05T08:33:05.682 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.683 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.683 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:05.683 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.071436+0000", 2023-11-05T08:33:05.683 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.3207999999999998e-05 2023-11-05T08:33:05.684 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.684 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.684 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:05.684 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.071453+0000", 2023-11-05T08:33:05.684 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.7042e-05 2023-11-05T08:33:05.684 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.685 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.685 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:05.685 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.071467+0000", 2023-11-05T08:33:05.685 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4384e-05 2023-11-05T08:33:05.686 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.686 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.686 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:05.686 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.071663+0000", 2023-11-05T08:33:05.686 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00019627300000000001 2023-11-05T08:33:05.686 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.687 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.687 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:05.687 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.071693+0000", 2023-11-05T08:33:05.687 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.9663e-05 2023-11-05T08:33:05.688 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.688 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.688 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:05.688 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.076421+0000", 2023-11-05T08:33:05.688 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.004727467 2023-11-05T08:33:05.688 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.689 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.689 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:05.689 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.076474+0000", 2023-11-05T08:33:05.689 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.3392e-05 2023-11-05T08:33:05.689 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.690 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:05.690 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.690 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.690 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.690 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.ds6 36/30 ECSubWrite(tid=21, reqid=client.4691.0:234, at_version=36'11, trim_to=0'0, roll_forward_to=34'8))", 2023-11-05T08:33:05.691 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:49.071464+0000", 2023-11-05T08:33:05.691 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 16.570511671999999, 2023-11-05T08:33:05.691 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0050888599999999997, 2023-11-05T08:33:05.691 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:05.691 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:05.692 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:05.692 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.692 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:05.692 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.071464+0000", 2023-11-05T08:33:05.692 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.693 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.693 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.693 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:05.693 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.071464+0000", 2023-11-05T08:33:05.693 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.693 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.694 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.694 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:05.694 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.071465+0000", 2023-11-05T08:33:05.694 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.2600000000000001e-07 2023-11-05T08:33:05.695 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.695 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.695 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:05.695 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.071490+0000", 2023-11-05T08:33:05.695 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.4879999999999999e-05 2023-11-05T08:33:05.695 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.696 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.696 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:05.696 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.071504+0000", 2023-11-05T08:33:05.696 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4484000000000001e-05 2023-11-05T08:33:05.696 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.696 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.697 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:05.697 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.071519+0000", 2023-11-05T08:33:05.697 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4878999999999999e-05 2023-11-05T08:33:05.697 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.697 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.697 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:05.698 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.072586+0000", 2023-11-05T08:33:05.698 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0010669290000000001 2023-11-05T08:33:05.698 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.698 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.698 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:05.699 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.072878+0000", 2023-11-05T08:33:05.699 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00029156000000000001 2023-11-05T08:33:05.699 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.699 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.699 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:05.699 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.076495+0000", 2023-11-05T08:33:05.700 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0036176649999999999 2023-11-05T08:33:05.700 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.700 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.700 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:05.700 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.076553+0000", 2023-11-05T08:33:05.700 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.7537000000000002e-05 2023-11-05T08:33:05.701 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.701 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:05.701 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.701 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.701 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.701 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.5s2 36/30 ECSubWrite(tid=25, reqid=osd.15.0:25, at_version=36'15, trim_to=0'0, roll_forward_to=35'13))", 2023-11-05T08:33:05.702 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:49.074899+0000", 2023-11-05T08:33:05.702 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 16.567076461999999, 2023-11-05T08:33:05.702 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0031894169999999999, 2023-11-05T08:33:05.702 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:05.702 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:05.703 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:05.703 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.703 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:05.704 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.074899+0000", 2023-11-05T08:33:05.704 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.704 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.704 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.705 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:05.705 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.074899+0000", 2023-11-05T08:33:05.705 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.705 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.705 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.706 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:05.706 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.074900+0000", 2023-11-05T08:33:05.706 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.6899999999999999e-07 2023-11-05T08:33:05.707 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.707 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.707 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:05.708 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.074926+0000", 2023-11-05T08:33:05.708 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.6211e-05 2023-11-05T08:33:05.709 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.709 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.709 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:05.709 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.074939+0000", 2023-11-05T08:33:05.710 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.2814e-05 2023-11-05T08:33:05.710 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.710 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.710 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:05.711 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.074949+0000", 2023-11-05T08:33:05.711 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0763e-05 2023-11-05T08:33:05.711 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.711 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.712 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:05.712 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.075024+0000", 2023-11-05T08:33:05.712 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.4202999999999994e-05 2023-11-05T08:33:05.712 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.712 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.712 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:05.713 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.075033+0000", 2023-11-05T08:33:05.713 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.0149999999999992e-06 2023-11-05T08:33:05.713 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.713 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.713 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:05.714 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.078044+0000", 2023-11-05T08:33:05.714 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0030110900000000001 2023-11-05T08:33:05.714 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.714 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.714 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:05.715 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.078089+0000", 2023-11-05T08:33:05.715 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.5052000000000002e-05 2023-11-05T08:33:05.715 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.716 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:05.716 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.716 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.716 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.717 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:287 3.11s0 3:88a306be:::smithi09039252-16:head [append 0~770048] snapc 8=[8,7,6,5,3] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:33:05.717 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:53.182407+0000", 2023-11-05T08:33:05.717 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 12.459569042, 2023-11-05T08:33:05.717 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.011495974, 2023-11-05T08:33:05.717 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:05.718 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:05.718 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:05.718 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:05.718 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:05.718 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 287 2023-11-05T08:33:05.719 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.719 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:05.719 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.719 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:05.719 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.182407+0000", 2023-11-05T08:33:05.719 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.720 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.720 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.720 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:05.720 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.182407+0000", 2023-11-05T08:33:05.720 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.721 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.721 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.721 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:05.721 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.182409+0000", 2023-11-05T08:33:05.721 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.3590000000000002e-06 2023-11-05T08:33:05.721 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.722 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.722 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:05.722 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.183780+0000", 2023-11-05T08:33:05.722 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00137108 2023-11-05T08:33:05.722 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.723 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.723 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:05.723 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.183800+0000", 2023-11-05T08:33:05.723 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.9819000000000001e-05 2023-11-05T08:33:05.723 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.724 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.724 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:05.724 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.183810+0000", 2023-11-05T08:33:05.724 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.6940000000000007e-06 2023-11-05T08:33:05.724 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.724 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.725 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:05.725 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.183873+0000", 2023-11-05T08:33:05.725 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.3536999999999998e-05 2023-11-05T08:33:05.725 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.725 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.726 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for peered", 2023-11-05T08:33:05.726 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.183877+0000", 2023-11-05T08:33:05.726 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.1779999999999996e-06 2023-11-05T08:33:05.726 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.727 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.727 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:05.727 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.171366+0000", 2023-11-05T08:33:05.727 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.98748831999999997 2023-11-05T08:33:05.728 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.728 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.728 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:05.728 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.171750+0000", 2023-11-05T08:33:05.728 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00038434800000000001 2023-11-05T08:33:05.728 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.729 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.729 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:05.729 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.174288+0000", 2023-11-05T08:33:05.729 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.002537957 2023-11-05T08:33:05.730 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.730 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.730 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:05.730 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.181565+0000", 2023-11-05T08:33:05.730 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0072769530000000001 2023-11-05T08:33:05.730 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.731 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.731 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:33:05.731 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.193786+0000", 2023-11-05T08:33:05.732 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.012221037000000001 2023-11-05T08:33:05.732 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.732 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.732 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:05.733 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.193903+0000", 2023-11-05T08:33:05.733 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000116692 2023-11-05T08:33:05.733 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.733 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:05.733 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.734 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.734 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.734 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:288 3.11s0 3:88a306be:::smithi09039252-16:head [append 0~376832] snapc 8=[8,7,6,5,3] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:33:05.734 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:53.186355+0000", 2023-11-05T08:33:05.734 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 12.455621262999999, 2023-11-05T08:33:05.734 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0078397800000001, 2023-11-05T08:33:05.735 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:05.735 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:05.735 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:05.735 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:05.735 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:05.736 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 288 2023-11-05T08:33:05.736 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.736 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:05.737 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.737 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:05.737 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186355+0000", 2023-11-05T08:33:05.737 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.738 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.738 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.738 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:05.738 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186355+0000", 2023-11-05T08:33:05.738 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.739 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.739 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.740 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:05.740 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186355+0000", 2023-11-05T08:33:05.740 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.9400000000000004e-07 2023-11-05T08:33:05.741 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.741 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.741 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:05.742 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186649+0000", 2023-11-05T08:33:05.742 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00029353199999999999 2023-11-05T08:33:05.742 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.743 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.743 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:05.743 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186662+0000", 2023-11-05T08:33:05.744 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.2728000000000001e-05 2023-11-05T08:33:05.744 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.744 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.744 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:05.744 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186667+0000", 2023-11-05T08:33:05.745 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.8270000000000003e-06 2023-11-05T08:33:05.745 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.745 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.745 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:05.745 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186742+0000", 2023-11-05T08:33:05.746 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.4328999999999999e-05 2023-11-05T08:33:05.746 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.746 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.746 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for peered", 2023-11-05T08:33:05.746 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186744+0000", 2023-11-05T08:33:05.746 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.266e-06 2023-11-05T08:33:05.747 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.747 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.747 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:05.747 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.176861+0000", 2023-11-05T08:33:05.748 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.990116577 2023-11-05T08:33:05.748 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.748 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.748 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:05.748 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.177014+0000", 2023-11-05T08:33:05.748 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000153661 2023-11-05T08:33:05.749 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.749 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.749 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:05.749 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.178253+0000", 2023-11-05T08:33:05.749 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0012384399999999999 2023-11-05T08:33:05.750 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.750 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.750 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:05.750 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.181931+0000", 2023-11-05T08:33:05.750 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0036777730000000001 2023-11-05T08:33:05.751 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.751 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.751 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:33:05.751 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.194093+0000", 2023-11-05T08:33:05.751 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.012162885999999999 2023-11-05T08:33:05.752 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.752 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.752 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:05.752 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.194194+0000", 2023-11-05T08:33:05.752 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00010086699999999999 2023-11-05T08:33:05.752 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.753 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:05.753 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.753 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.753 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.753 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:289 3.11s0 3:88a306be:::smithi09039252-16:head [setxattr _header (56) in=7b] snapc 8=[8,7,6,5,3] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:33:05.754 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:53.186682+0000", 2023-11-05T08:33:05.754 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 12.455293916, 2023-11-05T08:33:05.754 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.008829143, 2023-11-05T08:33:05.754 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:05.754 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:05.755 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:05.755 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:05.755 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:05.755 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 289 2023-11-05T08:33:05.755 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.756 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:05.756 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.756 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:05.756 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186682+0000", 2023-11-05T08:33:05.756 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.757 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.757 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.757 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:05.757 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186682+0000", 2023-11-05T08:33:05.757 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.758 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.758 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.758 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:05.759 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186683+0000", 2023-11-05T08:33:05.759 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.8999999999999996e-07 2023-11-05T08:33:05.759 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.759 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.759 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:05.760 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186687+0000", 2023-11-05T08:33:05.760 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.933e-06 2023-11-05T08:33:05.760 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.760 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.760 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:05.761 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186697+0000", 2023-11-05T08:33:05.761 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.6120000000000005e-06 2023-11-05T08:33:05.761 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.761 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.761 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:05.761 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186704+0000", 2023-11-05T08:33:05.762 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.8929999999999999e-06 2023-11-05T08:33:05.762 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.762 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.762 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:05.762 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186817+0000", 2023-11-05T08:33:05.763 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000113229 2023-11-05T08:33:05.763 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.763 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.763 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for peered", 2023-11-05T08:33:05.763 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186819+0000", 2023-11-05T08:33:05.764 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.57e-06 2023-11-05T08:33:05.764 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.764 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.764 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:05.764 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.179928+0000", 2023-11-05T08:33:05.765 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.993109099 2023-11-05T08:33:05.765 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.765 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.765 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:05.765 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.180084+0000", 2023-11-05T08:33:05.765 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00015647 2023-11-05T08:33:05.766 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.766 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.766 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:05.766 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.181012+0000", 2023-11-05T08:33:05.766 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00092730299999999998 2023-11-05T08:33:05.767 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.767 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.767 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:05.767 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.183961+0000", 2023-11-05T08:33:05.767 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0029494510000000001 2023-11-05T08:33:05.768 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.768 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.768 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:33:05.768 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.195373+0000", 2023-11-05T08:33:05.768 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.011411607000000001 2023-11-05T08:33:05.768 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.769 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.769 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:05.769 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.195511+0000", 2023-11-05T08:33:05.769 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000138286 2023-11-05T08:33:05.769 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.770 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:05.770 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.770 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.770 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.770 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:290 3.11s0 3:88a306be:::smithi09039252-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:33:05.771 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:53.186775+0000", 2023-11-05T08:33:05.771 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 12.455200431, 2023-11-05T08:33:05.771 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0117209439999999, 2023-11-05T08:33:05.771 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:05.771 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "started", 2023-11-05T08:33:05.771 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:05.772 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:05.772 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:05.772 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 290 2023-11-05T08:33:05.772 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.772 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:05.773 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.773 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:05.773 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186775+0000", 2023-11-05T08:33:05.773 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.773 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.773 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.774 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:05.774 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186775+0000", 2023-11-05T08:33:05.774 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.774 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.774 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.775 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:05.775 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186776+0000", 2023-11-05T08:33:05.775 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.7999999999999993e-07 2023-11-05T08:33:05.775 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.775 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.775 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:05.776 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186780+0000", 2023-11-05T08:33:05.776 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.9249999999999997e-06 2023-11-05T08:33:05.776 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.776 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.776 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:05.776 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186792+0000", 2023-11-05T08:33:05.777 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.1841999999999999e-05 2023-11-05T08:33:05.777 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.777 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.777 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:05.777 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186800+0000", 2023-11-05T08:33:05.777 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.1559999999999995e-06 2023-11-05T08:33:05.778 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.778 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.778 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:05.778 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186855+0000", 2023-11-05T08:33:05.778 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.4892999999999998e-05 2023-11-05T08:33:05.779 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.779 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.779 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for peered", 2023-11-05T08:33:05.779 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186856+0000", 2023-11-05T08:33:05.779 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.1999999999999999e-07 2023-11-05T08:33:05.780 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.780 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.780 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:05.780 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.181764+0000", 2023-11-05T08:33:05.780 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.99490820300000005 2023-11-05T08:33:05.780 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.781 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.781 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for rw locks", 2023-11-05T08:33:05.781 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.181917+0000", 2023-11-05T08:33:05.781 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00015282300000000001 2023-11-05T08:33:05.781 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.782 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.782 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:05.782 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.196083+0000", 2023-11-05T08:33:05.782 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.014166247999999999 2023-11-05T08:33:05.782 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.783 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.783 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:05.783 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.196295+0000", 2023-11-05T08:33:05.783 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00021158599999999999 2023-11-05T08:33:05.783 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.784 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.784 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:05.784 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.198496+0000", 2023-11-05T08:33:05.784 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0022016679999999999 2023-11-05T08:33:05.784 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.785 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:05.785 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.785 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.785 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.785 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 41/40 ECSubWriteReply(tid=65, last_complete=41'13, committed=1, applied=1))", 2023-11-05T08:33:05.785 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:54.177163+0000", 2023-11-05T08:33:05.786 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 11.464812799000001, 2023-11-05T08:33:05.786 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0044112250000000004, 2023-11-05T08:33:05.786 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:05.786 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:05.787 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:05.787 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.787 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:05.787 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.177163+0000", 2023-11-05T08:33:05.787 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.788 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.788 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.788 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:05.788 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.177163+0000", 2023-11-05T08:33:05.788 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.789 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.789 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.789 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:05.789 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.177165+0000", 2023-11-05T08:33:05.789 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.716e-06 2023-11-05T08:33:05.790 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.790 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.790 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:05.790 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.177174+0000", 2023-11-05T08:33:05.790 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.9639999999999992e-06 2023-11-05T08:33:05.790 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.791 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.791 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:05.791 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.177198+0000", 2023-11-05T08:33:05.791 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.3849000000000002e-05 2023-11-05T08:33:05.791 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.792 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.792 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:05.792 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.177229+0000", 2023-11-05T08:33:05.792 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.1677999999999997e-05 2023-11-05T08:33:05.793 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.793 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.793 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:05.793 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.179757+0000", 2023-11-05T08:33:05.793 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0025279130000000001 2023-11-05T08:33:05.794 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.794 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.794 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:05.794 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.181574+0000", 2023-11-05T08:33:05.794 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0018171050000000001 2023-11-05T08:33:05.795 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.795 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:05.795 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.795 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.795 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.795 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.14s2 41/40 ECSubWrite(tid=17, reqid=client.4691.0:278, at_version=41'14, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:05.796 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:54.185928+0000", 2023-11-05T08:33:05.796 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 11.456047460000001, 2023-11-05T08:33:05.796 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0044635960000000002, 2023-11-05T08:33:05.796 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:05.796 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:05.797 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:05.797 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.797 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:05.797 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.185928+0000", 2023-11-05T08:33:05.797 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.797 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.798 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.798 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:05.798 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.185928+0000", 2023-11-05T08:33:05.798 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.798 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.799 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.799 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:05.799 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.185930+0000", 2023-11-05T08:33:05.799 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.6330000000000001e-06 2023-11-05T08:33:05.799 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.799 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.800 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:05.800 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.186149+0000", 2023-11-05T08:33:05.800 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000218777 2023-11-05T08:33:05.800 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.800 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.800 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:05.801 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.186179+0000", 2023-11-05T08:33:05.801 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.0338999999999999e-05 2023-11-05T08:33:05.801 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.801 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.801 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:05.801 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.186194+0000", 2023-11-05T08:33:05.802 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4749000000000001e-05 2023-11-05T08:33:05.802 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.802 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.802 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:05.802 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.186347+0000", 2023-11-05T08:33:05.803 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00015294700000000001 2023-11-05T08:33:05.803 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.803 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.803 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:05.803 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.186365+0000", 2023-11-05T08:33:05.804 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.8048999999999999e-05 2023-11-05T08:33:05.804 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.804 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.804 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:05.804 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.190305+0000", 2023-11-05T08:33:05.805 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0039397820000000002 2023-11-05T08:33:05.805 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.805 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.805 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:05.806 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.190392+0000", 2023-11-05T08:33:05.806 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.7319999999999997e-05 2023-11-05T08:33:05.806 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.806 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:05.807 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.807 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.807 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.807 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.14s2 41/40 ECSubWrite(tid=18, reqid=client.4691.0:278, at_version=0'0, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:33:05.808 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:54.196240+0000", 2023-11-05T08:33:05.808 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 11.445736124, 2023-11-05T08:33:05.808 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0031170630000000002, 2023-11-05T08:33:05.809 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:05.809 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:05.809 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:05.810 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.810 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:05.810 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.196240+0000", 2023-11-05T08:33:05.811 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.811 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.811 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.811 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:05.811 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.196240+0000", 2023-11-05T08:33:05.812 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.812 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.812 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.813 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:05.813 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.196241+0000", 2023-11-05T08:33:05.813 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.7e-06 2023-11-05T08:33:05.813 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.813 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.814 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:05.814 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.196276+0000", 2023-11-05T08:33:05.814 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.4715000000000002e-05 2023-11-05T08:33:05.814 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.814 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.815 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:05.815 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.196310+0000", 2023-11-05T08:33:05.815 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.3705999999999999e-05 2023-11-05T08:33:05.815 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.815 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.815 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:05.816 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.196339+0000", 2023-11-05T08:33:05.816 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.9255e-05 2023-11-05T08:33:05.816 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.816 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.816 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:05.817 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.196532+0000", 2023-11-05T08:33:05.817 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00019293700000000001 2023-11-05T08:33:05.817 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.817 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.817 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:05.818 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.196553+0000", 2023-11-05T08:33:05.818 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.0856000000000002e-05 2023-11-05T08:33:05.818 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.818 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.818 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:05.818 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.199243+0000", 2023-11-05T08:33:05.819 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.002690399 2023-11-05T08:33:05.819 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.819 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.819 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:05.819 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.199357+0000", 2023-11-05T08:33:05.819 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000113495 2023-11-05T08:33:05.820 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.820 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:05.820 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.820 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.821 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.821 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDPGPush(3.14s6 47/46 [PushOp(3:292b6c2e:::smithi09039252-40:7, version: 39'9, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:292b6c2e:::smithi09039252-40:7@39'9, size: 1671168, copy_subset: [], clone_subset: {}, snapset: 8=[]:{7: [7, 6, 5, 3], 8: [8]}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2023-11-05T08:33:05.821 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:00.235899+0000", 2023-11-05T08:33:05.821 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 5.4060763820000002, 2023-11-05T08:33:05.821 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.003707288, 2023-11-05T08:33:05.822 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:05.822 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:05.822 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:05.822 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.822 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:05.822 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.235899+0000", 2023-11-05T08:33:05.823 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.823 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.823 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.823 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:05.823 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.235899+0000", 2023-11-05T08:33:05.824 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.824 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.824 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.824 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:05.824 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.235901+0000", 2023-11-05T08:33:05.824 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.6199999999999999e-06 2023-11-05T08:33:05.825 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.825 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.825 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:05.825 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.236430+0000", 2023-11-05T08:33:05.825 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00052928100000000004 2023-11-05T08:33:05.825 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.826 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.826 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:05.826 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.236500+0000", 2023-11-05T08:33:05.826 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.9247999999999998e-05 2023-11-05T08:33:05.826 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.827 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.827 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:05.827 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.236639+0000", 2023-11-05T08:33:05.827 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00013896500000000001 2023-11-05T08:33:05.827 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.827 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.828 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:05.828 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.236929+0000", 2023-11-05T08:33:05.828 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000290069 2023-11-05T08:33:05.828 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.828 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.829 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:05.829 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.239607+0000", 2023-11-05T08:33:05.829 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0026781050000000001 2023-11-05T08:33:05.829 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.829 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:05.829 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.830 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.830 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.830 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDPGPush(3.14s6 47/46 [PushOp(3:292b6c2e:::smithi09039252-40:7, version: 39'9, data_included: [262144~155648], data_size: 155648, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:292b6c2e:::smithi09039252-40:7@39'9, size: 1671168, copy_subset: [], clone_subset: {}, snapset: 8=[]:{7: [7, 6, 5, 3], 8: [8]}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1671168, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2023-11-05T08:33:05.830 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:00.244683+0000", 2023-11-05T08:33:05.830 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 5.3972924290000002, 2023-11-05T08:33:05.830 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0026555820000000001, 2023-11-05T08:33:05.831 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:05.831 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:05.831 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:05.831 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.831 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:05.831 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.244683+0000", 2023-11-05T08:33:05.832 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.832 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.832 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.832 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:05.832 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.244683+0000", 2023-11-05T08:33:05.833 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.833 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.833 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.834 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:05.834 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.244685+0000", 2023-11-05T08:33:05.834 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.37e-06 2023-11-05T08:33:05.834 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.834 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.835 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:05.835 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.245108+0000", 2023-11-05T08:33:05.835 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00042373200000000001 2023-11-05T08:33:05.835 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.835 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.836 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:05.836 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.245153+0000", 2023-11-05T08:33:05.836 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.4678000000000002e-05 2023-11-05T08:33:05.836 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.836 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.836 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:05.837 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.245234+0000", 2023-11-05T08:33:05.837 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.1271000000000002e-05 2023-11-05T08:33:05.837 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.837 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.837 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:05.838 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.245475+0000", 2023-11-05T08:33:05.838 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00024041899999999999 2023-11-05T08:33:05.838 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.838 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.838 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:05.839 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.247339+0000", 2023-11-05T08:33:05.839 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001864112 2023-11-05T08:33:05.841 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.841 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:05.841 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.841 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.842 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.842 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDPGPush(3.14s6 47/46 [PushOp(3:292b6c2e:::smithi09039252-40:8, version: 41'13, data_included: [262144~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:292b6c2e:::smithi09039252-40:8@41'13, size: 2752512, copy_subset: [], clone_subset: {}, snapset: 8=[]:{7: [7, 6, 5, 3], 8: [8]}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2023-11-05T08:33:05.842 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:00.271253+0000", 2023-11-05T08:33:05.842 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 5.3707232679999999, 2023-11-05T08:33:05.842 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0027597580000000002, 2023-11-05T08:33:05.842 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:05.843 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:05.843 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:05.843 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.843 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:05.843 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.271253+0000", 2023-11-05T08:33:05.844 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.844 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.844 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.844 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:05.844 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.271253+0000", 2023-11-05T08:33:05.845 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.845 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.845 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.845 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:05.845 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.271255+0000", 2023-11-05T08:33:05.846 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.2230000000000001e-06 2023-11-05T08:33:05.846 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.846 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.846 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:05.846 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.271816+0000", 2023-11-05T08:33:05.847 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00056167999999999995 2023-11-05T08:33:05.847 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.847 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.847 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:05.847 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.271878+0000", 2023-11-05T08:33:05.847 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.1933999999999994e-05 2023-11-05T08:33:05.848 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.848 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.848 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:05.848 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.271919+0000", 2023-11-05T08:33:05.849 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.0908999999999997e-05 2023-11-05T08:33:05.849 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.849 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.849 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:05.849 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.272109+0000", 2023-11-05T08:33:05.849 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00018932000000000001 2023-11-05T08:33:05.850 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.850 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.850 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:05.850 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.274012+0000", 2023-11-05T08:33:05.850 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001903692 2023-11-05T08:33:05.851 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.851 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:05.851 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.851 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.851 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.851 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:332 3.1s0 3:863368a9:::smithi09039252-6:head [read 0~98304,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2023-11-05T08:33:05.852 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:04.226233+0000", 2023-11-05T08:33:05.852 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 1.41574325, 2023-11-05T08:33:05.852 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0054114870000000004, 2023-11-05T08:33:05.852 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:05.852 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "started", 2023-11-05T08:33:05.853 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:05.853 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:05.853 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:05.853 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 332 2023-11-05T08:33:05.853 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.853 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:05.854 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.854 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:05.854 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226233+0000", 2023-11-05T08:33:05.854 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.854 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.855 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.855 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:05.855 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226233+0000", 2023-11-05T08:33:05.855 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.855 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.856 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.856 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:05.856 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226235+0000", 2023-11-05T08:33:05.856 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.0140000000000001e-06 2023-11-05T08:33:05.856 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.857 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.857 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:05.857 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226246+0000", 2023-11-05T08:33:05.857 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0992000000000001e-05 2023-11-05T08:33:05.857 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.858 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.858 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:05.858 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226271+0000", 2023-11-05T08:33:05.858 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.5732999999999999e-05 2023-11-05T08:33:05.858 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.858 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.859 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:05.859 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226290+0000", 2023-11-05T08:33:05.859 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.8793000000000001e-05 2023-11-05T08:33:05.859 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.859 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.860 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:05.860 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226800+0000", 2023-11-05T08:33:05.860 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00050957599999999997 2023-11-05T08:33:05.860 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.860 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.860 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:05.861 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.227301+0000", 2023-11-05T08:33:05.861 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00050143699999999998 2023-11-05T08:33:05.861 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.861 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.861 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:05.861 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.231644+0000", 2023-11-05T08:33:05.862 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0043429419999999998 2023-11-05T08:33:05.862 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.862 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:05.862 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.862 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.862 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.863 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:333 3.1s0 3:863368a9:::smithi09039252-6:head [read 0~98304,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2023-11-05T08:33:05.863 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:04.226324+0000", 2023-11-05T08:33:05.863 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 1.415651349, 2023-11-05T08:33:05.863 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0064759040000000002, 2023-11-05T08:33:05.863 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:05.863 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "started", 2023-11-05T08:33:05.864 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:05.864 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:05.864 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:05.864 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 333 2023-11-05T08:33:05.864 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.864 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:05.865 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.865 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:05.865 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226324+0000", 2023-11-05T08:33:05.865 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.865 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.865 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.866 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:05.866 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226324+0000", 2023-11-05T08:33:05.866 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.866 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.866 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.867 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:05.867 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226326+0000", 2023-11-05T08:33:05.867 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.243e-06 2023-11-05T08:33:05.867 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.867 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.868 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:05.868 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226335+0000", 2023-11-05T08:33:05.868 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.2560000000000008e-06 2023-11-05T08:33:05.868 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.868 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.868 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:05.869 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226356+0000", 2023-11-05T08:33:05.869 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.1406000000000001e-05 2023-11-05T08:33:05.869 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.869 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.869 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:05.869 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226372+0000", 2023-11-05T08:33:05.870 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.5923e-05 2023-11-05T08:33:05.870 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.870 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.870 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:05.870 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.228444+0000", 2023-11-05T08:33:05.871 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0020712930000000001 2023-11-05T08:33:05.871 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.871 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.871 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:05.871 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.228711+0000", 2023-11-05T08:33:05.871 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00026721699999999999 2023-11-05T08:33:05.872 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.872 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.872 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:05.872 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.232800+0000", 2023-11-05T08:33:05.872 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0040895660000000002 2023-11-05T08:33:05.873 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.873 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:05.873 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.873 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.873 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.874 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:334 3.1s0 3:863368a9:::smithi09039252-6:head [read 0~98304,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2023-11-05T08:33:05.874 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:04.226390+0000", 2023-11-05T08:33:05.874 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 1.415586148, 2023-11-05T08:33:05.874 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.017587591999999999, 2023-11-05T08:33:05.874 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:05.875 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "started", 2023-11-05T08:33:05.875 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:05.875 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:05.875 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:05.875 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 334 2023-11-05T08:33:05.875 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.876 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:05.876 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.876 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:05.876 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226390+0000", 2023-11-05T08:33:05.876 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.877 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.877 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.877 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:05.877 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226390+0000", 2023-11-05T08:33:05.877 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.877 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.878 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.878 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:05.878 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226390+0000", 2023-11-05T08:33:05.878 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.1999999999999998e-07 2023-11-05T08:33:05.878 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.879 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.879 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:05.879 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226396+0000", 2023-11-05T08:33:05.879 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.3229999999999997e-06 2023-11-05T08:33:05.879 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.880 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.880 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:05.880 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226409+0000", 2023-11-05T08:33:05.880 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.3096e-05 2023-11-05T08:33:05.880 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.880 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.881 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:05.881 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226417+0000", 2023-11-05T08:33:05.881 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.1300000000000001e-06 2023-11-05T08:33:05.881 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.881 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.882 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:05.882 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.233238+0000", 2023-11-05T08:33:05.882 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0068212999999999998 2023-11-05T08:33:05.882 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.882 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.882 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:05.883 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.233474+0000", 2023-11-05T08:33:05.883 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000235286 2023-11-05T08:33:05.883 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.883 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.884 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:05.884 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.243977+0000", 2023-11-05T08:33:05.884 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.010503637 2023-11-05T08:33:05.884 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.884 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:05.884 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.885 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.885 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.885 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(osd.7.22:1 3.bs0 3:d0f842ed:::smithi09039252-18:head [assert-version v2,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:05.885 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:05.259122+0000", 2023-11-05T08:33:05.885 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 0.38285400000000003, 2023-11-05T08:33:05.886 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0027184129999999998, 2023-11-05T08:33:05.886 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:05.886 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "started", 2023-11-05T08:33:05.886 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:05.886 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.887 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:05.887 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.259122+0000", 2023-11-05T08:33:05.887 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.887 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.887 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.888 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:05.888 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.259122+0000", 2023-11-05T08:33:05.888 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:05.888 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.889 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.889 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:05.889 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.259125+0000", 2023-11-05T08:33:05.889 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.83e-06 2023-11-05T08:33:05.889 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.889 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.890 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:05.890 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.259133+0000", 2023-11-05T08:33:05.890 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.4470000000000006e-06 2023-11-05T08:33:05.890 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.890 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.891 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:05.891 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.259155+0000", 2023-11-05T08:33:05.891 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.1467e-05 2023-11-05T08:33:05.891 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.891 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.892 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:05.892 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.259168+0000", 2023-11-05T08:33:05.892 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.3422e-05 2023-11-05T08:33:05.892 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.892 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.893 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:05.893 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.259236+0000", 2023-11-05T08:33:05.893 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.8553999999999994e-05 2023-11-05T08:33:05.893 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.893 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.893 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:05.894 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.259347+0000", 2023-11-05T08:33:05.894 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000110732 2023-11-05T08:33:05.894 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:05.894 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:05.895 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:05.895 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.261840+0000", 2023-11-05T08:33:05.895 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0024929610000000001 2023-11-05T08:33:05.895 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.895 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:05.895 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.896 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:05.896 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:05.896 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:33:05.896 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-11-05T08:33:05.898 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2023-11-05T08:33:05.898 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1: {} 2023-11-05T08:33:05.899 INFO:teuthology.orchestra.run.smithi002.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-11-05T08:33:05.899 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2: {} 2023-11-05T08:33:05.900 INFO:teuthology.orchestra.run.smithi002.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-11-05T08:33:05.900 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-11-05T08:33:05.900 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4: {} 2023-11-05T08:33:05.901 INFO:teuthology.orchestra.run.smithi002.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-11-05T08:33:05.901 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5: {} 2023-11-05T08:33:05.901 INFO:teuthology.orchestra.run.smithi002.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-11-05T08:33:05.902 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-05T08:33:05.902 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6: {} 2023-11-05T08:33:05.903 INFO:teuthology.orchestra.run.smithi002.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-11-05T08:33:05.903 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7: {} 2023-11-05T08:33:05.904 INFO:teuthology.orchestra.run.smithi002.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-11-05T08:33:05.904 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8: {} 2023-11-05T08:33:05.904 INFO:teuthology.orchestra.run.smithi002.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-11-05T08:33:05.905 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9: {} 2023-11-05T08:33:05.905 INFO:teuthology.orchestra.run.smithi002.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-11-05T08:33:05.906 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10: {} 2023-11-05T08:33:05.906 INFO:teuthology.orchestra.run.smithi002.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-11-05T08:33:05.907 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11: {} 2023-11-05T08:33:05.907 INFO:teuthology.orchestra.run.smithi002.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-11-05T08:33:05.908 INFO:teuthology.orchestra.run.smithi002.stdout:osd.12: {} 2023-11-05T08:33:05.908 INFO:teuthology.orchestra.run.smithi002.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-11-05T08:33:05.909 INFO:teuthology.orchestra.run.smithi002.stdout:osd.13: {} 2023-11-05T08:33:05.909 INFO:teuthology.orchestra.run.smithi002.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-11-05T08:33:05.918 INFO:teuthology.orchestra.run.smithi002.stdout:osd.14: {} 2023-11-05T08:33:05.919 INFO:teuthology.orchestra.run.smithi002.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-11-05T08:33:05.941 INFO:teuthology.orchestra.run.smithi002.stdout:osd.15: {} 2023-11-05T08:33:05.942 INFO:teuthology.orchestra.run.smithi002.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-11-05T08:33:05.946 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-05T08:33:05.947 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-05T08:33:06.004 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:06.001+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:06.036 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:06.036 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:06.051 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2023-11-05T08:33:06.105 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:33:06.103+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:33:06.175 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:06.176 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:06.190 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2023-11-05T08:33:06.204 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-05T08:33:06.239 INFO:tasks.rados.rados.0.smithi090.stdout:136: done (4 left) 2023-11-05T08:33:06.239 INFO:tasks.rados.rados.0.smithi090.stdout:140: done (3 left) 2023-11-05T08:33:06.240 INFO:tasks.rados.rados.0.smithi090.stdout:141: done (2 left) 2023-11-05T08:33:06.240 INFO:tasks.rados.rados.0.smithi090.stdout:142: done (1 left) 2023-11-05T08:33:06.240 INFO:tasks.rados.rados.0.smithi090.stdout:143: snap_remove snap 9 2023-11-05T08:33:06.306 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-05T08:33:06.317 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:06.317 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:06.333 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2023-11-05T08:33:06.408 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:33:06.408+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:33:06.460 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:06.460 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:06.475 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2023-11-05T08:33:06.519 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:33:06.518+0000 7fdc5cd58640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:33:06.600 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:06.600 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:06.610 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:33:06.605+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:33:06.614 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2023-11-05T08:33:06.711 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:33:06.711+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:33:06.781 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:06.781 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:06.796 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-11-05T08:33:06.811 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-05T08:33:06.913 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:33:06.912+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:33:06.933 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:06.934 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:06.949 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-11-05T08:33:07.014 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:33:07.009+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:33:07.077 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:07.077 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:07.091 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-11-05T08:33:07.115 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:33:07.112+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:33:07.216 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:33:07.214+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:33:07.249 INFO:tasks.rados.rados.0.smithi090.stdout:143: done (1 left) 2023-11-05T08:33:07.249 INFO:tasks.rados.rados.0.smithi090.stdout:144: snap_create 2023-11-05T08:33:07.257 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-11-05T08:33:07.262 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:07.263 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:07.277 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-11-05T08:33:07.317 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:33:07.314+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:33:07.401 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:07.401 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:07.415 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-11-05T08:33:07.418 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:07.412+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:07.518 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:33:07.517+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:33:07.534 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:07.535 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:07.550 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-11-05T08:33:07.619 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:33:07.616+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:33:07.720 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:33:07.718+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:33:07.726 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:07.726 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:07.741 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2023-11-05T08:33:07.821 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:33:07.817+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:33:07.861 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:07.861 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:07.876 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2023-11-05T08:33:07.922 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:33:07.920+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:33:07.947 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-05T08:33:08.023 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:33:08.021+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:33:08.039 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:08.039 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:08.054 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2023-11-05T08:33:08.087 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-11-05T08:33:08.106 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1: {} 2023-11-05T08:33:08.106 INFO:teuthology.orchestra.run.smithi002.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-11-05T08:33:08.126 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2: {} 2023-11-05T08:33:08.127 INFO:teuthology.orchestra.run.smithi002.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-11-05T08:33:08.127 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-11-05T08:33:08.142 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:08.141+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:08.144 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4: {} 2023-11-05T08:33:08.145 INFO:teuthology.orchestra.run.smithi002.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-11-05T08:33:08.163 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5: {} 2023-11-05T08:33:08.163 INFO:teuthology.orchestra.run.smithi002.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-11-05T08:33:08.183 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6: {} 2023-11-05T08:33:08.183 INFO:teuthology.orchestra.run.smithi002.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-11-05T08:33:08.203 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7: {} 2023-11-05T08:33:08.203 INFO:teuthology.orchestra.run.smithi002.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-11-05T08:33:08.222 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8: {} 2023-11-05T08:33:08.223 INFO:teuthology.orchestra.run.smithi002.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-11-05T08:33:08.224 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:33:08.224+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:33:08.226 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:08.226 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:08.240 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2023-11-05T08:33:08.242 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9: {} 2023-11-05T08:33:08.242 INFO:teuthology.orchestra.run.smithi002.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-11-05T08:33:08.254 INFO:teuthology.orchestra.run.smithi002.stderr:nodeep-scrub is set 2023-11-05T08:33:08.261 INFO:tasks.rados.rados.0.smithi090.stdout:144: done (1 left) 2023-11-05T08:33:08.261 INFO:tasks.rados.rados.0.smithi090.stdout:145: delete oid 50 current snap is 11 2023-11-05T08:33:08.266 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10: {} 2023-11-05T08:33:08.267 INFO:teuthology.orchestra.run.smithi002.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-11-05T08:33:08.268 INFO:tasks.rados.rados.0.smithi090.stdout:145: done (1 left) 2023-11-05T08:33:08.268 INFO:tasks.rados.rados.0.smithi090.stdout:146: read oid 10 snap -1 2023-11-05T08:33:08.268 INFO:tasks.rados.rados.0.smithi090.stdout:146: expect (ObjNum 9 snap 0 seq_num 9) 2023-11-05T08:33:08.268 INFO:tasks.rados.rados.0.smithi090.stdout:147: read oid 18 snap -1 2023-11-05T08:33:08.268 INFO:tasks.rados.rados.0.smithi090.stdout:147: expect (ObjNum 17 snap 0 seq_num 17) 2023-11-05T08:33:08.269 INFO:tasks.rados.rados.0.smithi090.stdout:148: snap_remove snap 10 2023-11-05T08:33:08.289 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11: {} 2023-11-05T08:33:08.289 INFO:teuthology.orchestra.run.smithi002.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-11-05T08:33:08.304 INFO:teuthology.orchestra.run.smithi002.stdout:osd.12: {} 2023-11-05T08:33:08.305 INFO:teuthology.orchestra.run.smithi002.stderr:osd.12: osd_enable_op_tracker = 'true' 2023-11-05T08:33:08.322 INFO:teuthology.orchestra.run.smithi002.stdout:osd.13: {} 2023-11-05T08:33:08.323 INFO:teuthology.orchestra.run.smithi002.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-11-05T08:33:08.330 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:33:08.329+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:33:08.343 INFO:teuthology.orchestra.run.smithi002.stdout:osd.14: {} 2023-11-05T08:33:08.343 INFO:teuthology.orchestra.run.smithi002.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-11-05T08:33:08.364 INFO:teuthology.orchestra.run.smithi002.stdout:osd.15: {} 2023-11-05T08:33:08.365 INFO:teuthology.orchestra.run.smithi002.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-11-05T08:33:08.368 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-05T08:33:08.369 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-05T08:33:08.414 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:33:08.414 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2023-11-05T08:33:08.414 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 0 2023-11-05T08:33:08.414 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:33:08.426 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:33:08.422+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:33:08.428 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2023-11-05T08:33:08.527 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:33:08.524+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:33:08.596 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:33:08.596 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2023-11-05T08:33:08.596 INFO:teuthology.orchestra.run.smithi053.stdout: "complaint_time": 30, 2023-11-05T08:33:08.596 INFO:teuthology.orchestra.run.smithi053.stdout: "num_blocked_ops": 0 2023-11-05T08:33:08.597 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:33:08.610 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2023-11-05T08:33:08.628 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:33:08.626+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:33:08.729 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:33:08.726+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:33:08.740 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:33:08.740 INFO:teuthology.orchestra.run.smithi053.stdout: "size": 20, 2023-11-05T08:33:08.740 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 600, 2023-11-05T08:33:08.740 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2023-11-05T08:33:08.741 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.741 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:08.741 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:28.710073+0000", 2023-11-05T08:33:08.741 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 40.027622682999997, 2023-11-05T08:33:08.741 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.052834791999999998, 2023-11-05T08:33:08.742 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:08.742 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:08.742 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:08.742 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:08.742 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:08.742 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 1 2023-11-05T08:33:08.743 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.743 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:08.743 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.743 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:08.743 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710073+0000", 2023-11-05T08:33:08.744 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:08.744 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.744 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.744 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:08.744 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710073+0000", 2023-11-05T08:33:08.744 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:08.745 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.745 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.745 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:08.745 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710076+0000", 2023-11-05T08:33:08.745 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1729999999999999e-06 2023-11-05T08:33:08.746 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.746 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.746 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:08.746 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710108+0000", 2023-11-05T08:33:08.746 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1458000000000003e-05 2023-11-05T08:33:08.746 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.747 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.747 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:08.747 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710140+0000", 2023-11-05T08:33:08.747 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2532e-05 2023-11-05T08:33:08.747 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.748 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.748 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:08.748 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710165+0000", 2023-11-05T08:33:08.748 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5205e-05 2023-11-05T08:33:08.748 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.749 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.749 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:08.749 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710311+0000", 2023-11-05T08:33:08.749 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014532799999999999 2023-11-05T08:33:08.749 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.749 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.750 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2023-11-05T08:33:08.750 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710327+0000", 2023-11-05T08:33:08.750 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6059e-05 2023-11-05T08:33:08.750 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.750 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.751 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:08.751 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.759348+0000", 2023-11-05T08:33:08.751 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.049021481999999998 2023-11-05T08:33:08.751 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.751 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.751 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:08.752 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.759880+0000", 2023-11-05T08:33:08.752 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00053197400000000005 2023-11-05T08:33:08.752 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.752 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.752 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:08.752 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.760478+0000", 2023-11-05T08:33:08.753 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00059839300000000002 2023-11-05T08:33:08.753 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.753 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.753 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:08.753 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.761955+0000", 2023-11-05T08:33:08.754 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001476991 2023-11-05T08:33:08.754 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.754 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.754 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:08.754 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.762611+0000", 2023-11-05T08:33:08.755 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00065598700000000002 2023-11-05T08:33:08.755 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.755 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.755 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:08.755 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.762782+0000", 2023-11-05T08:33:08.755 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00017064000000000001 2023-11-05T08:33:08.756 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.756 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.756 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:08.756 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.762908+0000", 2023-11-05T08:33:08.756 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012557000000000001 2023-11-05T08:33:08.757 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:08.757 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:08.757 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:08.757 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.757 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.758 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:08.758 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.151546+0000", 2023-11-05T08:33:08.758 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 39.586149335000002, 2023-11-05T08:33:08.758 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.018837386000000001, 2023-11-05T08:33:08.758 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:08.758 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:08.759 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:08.759 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:08.759 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:08.759 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 113 2023-11-05T08:33:08.759 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.760 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:08.760 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.760 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:08.760 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151546+0000", 2023-11-05T08:33:08.760 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:08.761 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.761 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.761 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:08.761 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151546+0000", 2023-11-05T08:33:08.761 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:08.762 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.762 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.762 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:08.762 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151547+0000", 2023-11-05T08:33:08.762 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9699999999999995e-07 2023-11-05T08:33:08.763 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.763 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.763 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:08.763 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151553+0000", 2023-11-05T08:33:08.763 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9830000000000001e-06 2023-11-05T08:33:08.763 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.764 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.764 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:08.764 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151562+0000", 2023-11-05T08:33:08.764 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.5060000000000001e-06 2023-11-05T08:33:08.764 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.765 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.766 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:08.766 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151569+0000", 2023-11-05T08:33:08.766 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9160000000000004e-06 2023-11-05T08:33:08.766 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.766 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.767 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:08.767 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.163944+0000", 2023-11-05T08:33:08.767 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012374703000000001 2023-11-05T08:33:08.767 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.767 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.767 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:08.768 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.164091+0000", 2023-11-05T08:33:08.768 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014739199999999999 2023-11-05T08:33:08.768 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.768 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.768 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:08.769 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.164295+0000", 2023-11-05T08:33:08.769 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00020334899999999999 2023-11-05T08:33:08.769 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.769 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.769 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:08.770 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.167412+0000", 2023-11-05T08:33:08.770 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.003117529 2023-11-05T08:33:08.770 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.770 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.770 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:08.770 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.170241+0000", 2023-11-05T08:33:08.771 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0028283890000000002 2023-11-05T08:33:08.771 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.771 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.771 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:08.771 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.170326+0000", 2023-11-05T08:33:08.772 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.5581999999999995e-05 2023-11-05T08:33:08.772 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.772 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.772 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:08.772 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.170384+0000", 2023-11-05T08:33:08.772 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7340000000000003e-05 2023-11-05T08:33:08.773 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:08.773 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:08.773 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:08.773 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.773 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.773 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:08.774 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.151583+0000", 2023-11-05T08:33:08.774 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 39.586112890000003, 2023-11-05T08:33:08.774 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.020286660000000001, 2023-11-05T08:33:08.774 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:08.774 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:08.774 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:08.775 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:08.775 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:08.775 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 114 2023-11-05T08:33:08.775 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.775 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:08.776 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.776 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:08.776 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151583+0000", 2023-11-05T08:33:08.776 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:08.776 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.777 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.777 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:08.777 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151583+0000", 2023-11-05T08:33:08.777 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:08.777 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.778 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.778 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:08.778 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151583+0000", 2023-11-05T08:33:08.778 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.2600000000000002e-07 2023-11-05T08:33:08.778 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.778 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.779 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:08.779 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151589+0000", 2023-11-05T08:33:08.779 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5099999999999998e-06 2023-11-05T08:33:08.779 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.779 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.780 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:08.780 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151598+0000", 2023-11-05T08:33:08.780 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1730000000000001e-06 2023-11-05T08:33:08.780 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.780 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.780 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:08.781 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151608+0000", 2023-11-05T08:33:08.781 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0059e-05 2023-11-05T08:33:08.781 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.781 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.781 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:08.782 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.165015+0000", 2023-11-05T08:33:08.782 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.013407207000000001 2023-11-05T08:33:08.782 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.782 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.782 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:08.782 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.165268+0000", 2023-11-05T08:33:08.783 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00025284999999999999 2023-11-05T08:33:08.783 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.783 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.783 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:08.783 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.165635+0000", 2023-11-05T08:33:08.784 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00036695899999999998 2023-11-05T08:33:08.784 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.784 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.784 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:08.784 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.170105+0000", 2023-11-05T08:33:08.785 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0044696960000000004 2023-11-05T08:33:08.785 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.785 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.785 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:08.785 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.171719+0000", 2023-11-05T08:33:08.785 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001613855 2023-11-05T08:33:08.786 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.786 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.786 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:08.786 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.171810+0000", 2023-11-05T08:33:08.786 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1625000000000005e-05 2023-11-05T08:33:08.787 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.787 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.787 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:08.787 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.171869+0000", 2023-11-05T08:33:08.787 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.91e-05 2023-11-05T08:33:08.788 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:08.788 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:08.788 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:08.788 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.788 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.788 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:08.789 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.151660+0000", 2023-11-05T08:33:08.789 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 39.586035090999999, 2023-11-05T08:33:08.789 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022553719, 2023-11-05T08:33:08.789 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:08.789 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:08.790 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:08.790 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:08.790 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:08.790 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 115 2023-11-05T08:33:08.790 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.791 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:08.791 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.791 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:08.791 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151660+0000", 2023-11-05T08:33:08.791 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:08.791 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.792 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.792 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:08.792 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151660+0000", 2023-11-05T08:33:08.792 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:08.792 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.793 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.793 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:08.793 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151661+0000", 2023-11-05T08:33:08.793 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.5600000000000004e-07 2023-11-05T08:33:08.793 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.793 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.794 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:08.794 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151799+0000", 2023-11-05T08:33:08.794 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000137207 2023-11-05T08:33:08.794 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.794 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.795 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:08.795 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151814+0000", 2023-11-05T08:33:08.795 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4885000000000001e-05 2023-11-05T08:33:08.795 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.795 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.796 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:08.796 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151827+0000", 2023-11-05T08:33:08.796 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3413e-05 2023-11-05T08:33:08.796 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.796 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.797 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:08.797 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.167513+0000", 2023-11-05T08:33:08.797 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.015686091999999999 2023-11-05T08:33:08.797 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.797 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.798 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:08.798 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.167652+0000", 2023-11-05T08:33:08.798 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013890500000000001 2023-11-05T08:33:08.798 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.798 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.798 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:08.799 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.167860+0000", 2023-11-05T08:33:08.799 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00020838599999999999 2023-11-05T08:33:08.799 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.799 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.799 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:08.800 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.171602+0000", 2023-11-05T08:33:08.800 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0037417449999999999 2023-11-05T08:33:08.800 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.800 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.800 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:08.800 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174068+0000", 2023-11-05T08:33:08.801 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0024658010000000001 2023-11-05T08:33:08.801 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.801 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.801 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:08.801 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174152+0000", 2023-11-05T08:33:08.801 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.3752999999999996e-05 2023-11-05T08:33:08.802 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.802 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.802 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:08.802 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174214+0000", 2023-11-05T08:33:08.802 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.2576000000000005e-05 2023-11-05T08:33:08.803 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:08.803 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:08.803 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:08.803 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.803 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.803 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:08.804 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.151849+0000", 2023-11-05T08:33:08.804 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 39.585846734999997, 2023-11-05T08:33:08.804 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022629176000000001, 2023-11-05T08:33:08.804 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:08.804 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:08.805 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:08.805 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:08.805 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:08.805 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 116 2023-11-05T08:33:08.805 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.805 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:08.806 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.806 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:08.806 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151849+0000", 2023-11-05T08:33:08.806 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:08.806 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.807 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.807 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:08.807 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151849+0000", 2023-11-05T08:33:08.807 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:08.807 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.808 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.808 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:08.808 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151850+0000", 2023-11-05T08:33:08.808 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.037e-06 2023-11-05T08:33:08.808 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.808 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.809 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:08.809 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151858+0000", 2023-11-05T08:33:08.809 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.3389999999999993e-06 2023-11-05T08:33:08.809 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.809 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.810 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:08.810 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151868+0000", 2023-11-05T08:33:08.810 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6670000000000008e-06 2023-11-05T08:33:08.810 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.810 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.811 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:08.811 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151875+0000", 2023-11-05T08:33:08.811 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9229999999999996e-06 2023-11-05T08:33:08.811 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.811 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.811 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:08.812 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.168777+0000", 2023-11-05T08:33:08.812 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.016901884999999998 2023-11-05T08:33:08.812 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.812 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.812 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:08.813 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.169009+0000", 2023-11-05T08:33:08.813 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00023236400000000001 2023-11-05T08:33:08.813 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.813 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.813 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:08.813 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.169365+0000", 2023-11-05T08:33:08.814 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00035625300000000001 2023-11-05T08:33:08.814 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.814 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.814 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:08.814 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.171624+0000", 2023-11-05T08:33:08.815 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0022587169999999999 2023-11-05T08:33:08.815 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.815 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.815 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:08.815 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174314+0000", 2023-11-05T08:33:08.816 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002690263 2023-11-05T08:33:08.816 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.816 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.816 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:08.816 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174405+0000", 2023-11-05T08:33:08.816 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1064999999999996e-05 2023-11-05T08:33:08.817 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.817 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.817 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:08.817 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174478+0000", 2023-11-05T08:33:08.817 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.2663000000000005e-05 2023-11-05T08:33:08.818 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:08.818 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:08.818 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:08.818 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.818 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.819 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:08.819 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.151886+0000", 2023-11-05T08:33:08.819 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 39.585809320000003, 2023-11-05T08:33:08.819 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.023174162000000002, 2023-11-05T08:33:08.819 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:08.819 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:08.820 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:08.820 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:08.820 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:08.820 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 117 2023-11-05T08:33:08.820 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.821 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:08.821 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.821 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:08.821 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151886+0000", 2023-11-05T08:33:08.821 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:08.822 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.822 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.822 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:08.822 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151886+0000", 2023-11-05T08:33:08.822 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:08.823 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.823 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.823 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:08.823 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151887+0000", 2023-11-05T08:33:08.823 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.2300000000000001e-07 2023-11-05T08:33:08.823 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.824 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.824 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:08.824 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151893+0000", 2023-11-05T08:33:08.824 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.19e-06 2023-11-05T08:33:08.824 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.825 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.825 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:08.825 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151905+0000", 2023-11-05T08:33:08.825 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1883e-05 2023-11-05T08:33:08.825 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.825 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.826 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:08.826 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151912+0000", 2023-11-05T08:33:08.826 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.0690000000000004e-06 2023-11-05T08:33:08.826 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.826 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.827 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:08.827 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.170509+0000", 2023-11-05T08:33:08.827 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.018596686000000001 2023-11-05T08:33:08.827 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.827 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.828 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:08.828 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.170791+0000", 2023-11-05T08:33:08.828 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00028221799999999999 2023-11-05T08:33:08.828 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.828 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.829 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:08.829 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.171137+0000", 2023-11-05T08:33:08.829 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000346454 2023-11-05T08:33:08.829 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.829 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.829 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:08.830 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.172413+0000", 2023-11-05T08:33:08.830 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0012752340000000001 2023-11-05T08:33:08.830 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.830 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.830 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:08.831 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174926+0000", 2023-11-05T08:33:08.831 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0025137050000000002 2023-11-05T08:33:08.831 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.831 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.831 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:08.831 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.175007+0000", 2023-11-05T08:33:08.832 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.0413000000000004e-05 2023-11-05T08:33:08.832 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.832 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.832 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:08.832 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.175060+0000", 2023-11-05T08:33:08.833 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3687000000000003e-05 2023-11-05T08:33:08.833 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:08.833 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:08.833 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:08.833 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.833 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.834 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:163 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:08.834 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.670890+0000", 2023-11-05T08:33:08.834 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 39.066805090999999, 2023-11-05T08:33:08.834 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.02132307, 2023-11-05T08:33:08.834 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:08.835 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:08.835 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:08.835 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:08.835 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:08.835 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 163 2023-11-05T08:33:08.835 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.836 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:08.836 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.836 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:08.836 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670890+0000", 2023-11-05T08:33:08.836 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:08.836 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.837 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.837 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:08.837 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670890+0000", 2023-11-05T08:33:08.837 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:08.837 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.838 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.838 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:08.838 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670891+0000", 2023-11-05T08:33:08.838 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3300000000000002e-07 2023-11-05T08:33:08.838 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.839 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.839 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:08.839 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670897+0000", 2023-11-05T08:33:08.839 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.4600000000000002e-06 2023-11-05T08:33:08.839 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.840 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.840 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:08.840 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670906+0000", 2023-11-05T08:33:08.840 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.3130000000000004e-06 2023-11-05T08:33:08.840 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.840 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.841 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:08.841 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670913+0000", 2023-11-05T08:33:08.841 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9859999999999997e-06 2023-11-05T08:33:08.841 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.841 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.842 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:08.842 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.687360+0000", 2023-11-05T08:33:08.842 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.016447244 2023-11-05T08:33:08.842 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.842 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.843 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:08.843 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.687588+0000", 2023-11-05T08:33:08.843 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00022757699999999999 2023-11-05T08:33:08.843 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.843 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.843 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:08.844 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.687946+0000", 2023-11-05T08:33:08.844 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00035789399999999999 2023-11-05T08:33:08.844 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.844 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.844 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:08.845 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.691929+0000", 2023-11-05T08:33:08.845 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0039835690000000002 2023-11-05T08:33:08.845 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.845 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.845 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:08.845 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.691970+0000", 2023-11-05T08:33:08.846 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.0351e-05 2023-11-05T08:33:08.846 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.846 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.846 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:08.846 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692111+0000", 2023-11-05T08:33:08.847 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014114499999999999 2023-11-05T08:33:08.847 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.847 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.847 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:08.847 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692214+0000", 2023-11-05T08:33:08.847 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000102898 2023-11-05T08:33:08.848 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:08.848 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:08.848 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:08.848 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.848 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.849 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:08.849 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.670924+0000", 2023-11-05T08:33:08.849 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 39.066771682999999, 2023-11-05T08:33:08.849 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.021621676999999999, 2023-11-05T08:33:08.849 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:08.849 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:08.850 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:08.850 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:08.850 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:08.850 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 164 2023-11-05T08:33:08.850 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.851 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:08.851 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.851 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:08.851 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670924+0000", 2023-11-05T08:33:08.851 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:08.852 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.852 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.852 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:08.852 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670924+0000", 2023-11-05T08:33:08.852 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:08.852 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.853 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.853 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:08.853 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670925+0000", 2023-11-05T08:33:08.853 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.8299999999999996e-07 2023-11-05T08:33:08.853 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.854 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.854 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:08.854 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671022+0000", 2023-11-05T08:33:08.854 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.7445e-05 2023-11-05T08:33:08.854 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.855 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.855 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:08.855 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671033+0000", 2023-11-05T08:33:08.855 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1206000000000001e-05 2023-11-05T08:33:08.855 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.855 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.856 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:08.856 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671045+0000", 2023-11-05T08:33:08.856 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1797000000000001e-05 2023-11-05T08:33:08.856 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.856 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.857 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:08.857 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.688874+0000", 2023-11-05T08:33:08.857 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017829424 2023-11-05T08:33:08.857 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.857 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.858 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:08.858 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.689097+0000", 2023-11-05T08:33:08.858 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000222828 2023-11-05T08:33:08.858 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.858 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.859 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:08.859 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.689444+0000", 2023-11-05T08:33:08.859 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00034672099999999998 2023-11-05T08:33:08.859 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.859 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.859 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:08.860 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692240+0000", 2023-11-05T08:33:08.860 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0027959030000000002 2023-11-05T08:33:08.860 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.860 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.860 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:08.861 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692339+0000", 2023-11-05T08:33:08.861 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.8801000000000005e-05 2023-11-05T08:33:08.861 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.861 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.861 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:08.861 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692435+0000", 2023-11-05T08:33:08.862 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6311999999999995e-05 2023-11-05T08:33:08.862 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.862 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.862 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:08.862 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692546+0000", 2023-11-05T08:33:08.863 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00011055700000000001 2023-11-05T08:33:08.863 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:08.863 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:08.863 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:08.863 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.864 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.864 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:08.864 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.671060+0000", 2023-11-05T08:33:08.864 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 39.066636017, 2023-11-05T08:33:08.864 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022080292000000001, 2023-11-05T08:33:08.865 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:08.865 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:08.865 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:08.865 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:08.865 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:08.866 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 165 2023-11-05T08:33:08.866 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.866 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:08.866 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.866 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:08.867 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671060+0000", 2023-11-05T08:33:08.867 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:08.867 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.867 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.867 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:08.868 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671060+0000", 2023-11-05T08:33:08.868 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:08.868 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.868 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.868 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:08.868 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671060+0000", 2023-11-05T08:33:08.869 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.1699999999999997e-07 2023-11-05T08:33:08.869 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.869 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.869 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:08.869 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671067+0000", 2023-11-05T08:33:08.869 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5690000000000001e-06 2023-11-05T08:33:08.870 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.870 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.870 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:08.870 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671076+0000", 2023-11-05T08:33:08.870 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.5270000000000004e-06 2023-11-05T08:33:08.871 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.871 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.871 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:08.871 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671084+0000", 2023-11-05T08:33:08.871 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.3459999999999996e-06 2023-11-05T08:33:08.871 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.872 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.872 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:08.872 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.690832+0000", 2023-11-05T08:33:08.872 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.019747793 2023-11-05T08:33:08.872 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.872 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.873 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:08.873 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.690971+0000", 2023-11-05T08:33:08.873 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000139245 2023-11-05T08:33:08.873 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.873 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.874 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:08.874 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.691178+0000", 2023-11-05T08:33:08.874 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000206892 2023-11-05T08:33:08.874 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.874 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.874 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:08.875 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692704+0000", 2023-11-05T08:33:08.875 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00152655 2023-11-05T08:33:08.875 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.875 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.875 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:08.875 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692904+0000", 2023-11-05T08:33:08.876 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000199376 2023-11-05T08:33:08.876 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.876 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.876 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:08.876 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.693044+0000", 2023-11-05T08:33:08.877 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013989199999999999 2023-11-05T08:33:08.877 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.877 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.877 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:08.877 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.693140+0000", 2023-11-05T08:33:08.878 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6285000000000004e-05 2023-11-05T08:33:08.878 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:08.878 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:08.878 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:08.878 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.879 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.879 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:184 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:08.879 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.866152+0000", 2023-11-05T08:33:08.879 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 38.871543742, 2023-11-05T08:33:08.879 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.019116587000000001, 2023-11-05T08:33:08.879 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:08.880 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:08.880 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:08.880 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:08.880 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:08.880 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 184 2023-11-05T08:33:08.881 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.881 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:08.881 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.881 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:08.881 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866152+0000", 2023-11-05T08:33:08.882 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:08.882 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.882 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.882 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:08.882 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866152+0000", 2023-11-05T08:33:08.882 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:08.883 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.883 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.883 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:08.883 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866152+0000", 2023-11-05T08:33:08.883 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4600000000000004e-07 2023-11-05T08:33:08.884 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.884 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.884 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:08.884 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866158+0000", 2023-11-05T08:33:08.884 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5629999999999999e-06 2023-11-05T08:33:08.885 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.885 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.885 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:08.885 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866167+0000", 2023-11-05T08:33:08.885 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.2229999999999996e-06 2023-11-05T08:33:08.886 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.886 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.886 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:08.886 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866175+0000", 2023-11-05T08:33:08.886 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.3699999999999997e-06 2023-11-05T08:33:08.887 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.887 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.887 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:08.887 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.878285+0000", 2023-11-05T08:33:08.887 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012110164 2023-11-05T08:33:08.887 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.888 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.888 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:08.888 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.878584+0000", 2023-11-05T08:33:08.888 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00029961300000000001 2023-11-05T08:33:08.888 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.888 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.889 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:08.889 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.878942+0000", 2023-11-05T08:33:08.889 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00035795100000000002 2023-11-05T08:33:08.889 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.889 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.890 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:08.890 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.883845+0000", 2023-11-05T08:33:08.890 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0049021450000000001 2023-11-05T08:33:08.890 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.890 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.891 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:08.891 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885018+0000", 2023-11-05T08:33:08.891 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0011738160000000001 2023-11-05T08:33:08.891 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.891 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.891 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:08.892 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885167+0000", 2023-11-05T08:33:08.892 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014895800000000001 2023-11-05T08:33:08.892 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.892 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.892 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:08.893 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885268+0000", 2023-11-05T08:33:08.893 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010113800000000001 2023-11-05T08:33:08.893 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:08.893 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:08.893 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:08.894 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.894 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.894 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:185 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:08.894 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.866212+0000", 2023-11-05T08:33:08.894 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 38.871484021999997, 2023-11-05T08:33:08.895 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.019448929, 2023-11-05T08:33:08.895 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:08.895 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:08.895 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:08.895 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:08.895 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:08.896 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 185 2023-11-05T08:33:08.896 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.896 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:08.896 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.896 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:08.897 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866212+0000", 2023-11-05T08:33:08.897 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:08.897 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.897 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.897 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:08.898 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866212+0000", 2023-11-05T08:33:08.898 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:08.898 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.898 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.898 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:08.898 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866213+0000", 2023-11-05T08:33:08.899 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0559999999999999e-06 2023-11-05T08:33:08.899 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.901 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.901 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:08.902 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866308+0000", 2023-11-05T08:33:08.902 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.4984999999999999e-05 2023-11-05T08:33:08.902 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.902 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.902 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:08.903 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866320+0000", 2023-11-05T08:33:08.903 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2639e-05 2023-11-05T08:33:08.903 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.903 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.903 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:08.903 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866328+0000", 2023-11-05T08:33:08.904 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.1240000000000005e-06 2023-11-05T08:33:08.904 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.904 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.904 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:08.904 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.879899+0000", 2023-11-05T08:33:08.904 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.013571059999999999 2023-11-05T08:33:08.905 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.905 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.905 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:08.905 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.880034+0000", 2023-11-05T08:33:08.905 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000134676 2023-11-05T08:33:08.906 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.906 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.906 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:08.906 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.880258+0000", 2023-11-05T08:33:08.906 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00022358100000000001 2023-11-05T08:33:08.907 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.907 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.907 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:08.907 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885299+0000", 2023-11-05T08:33:08.907 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.005041678 2023-11-05T08:33:08.907 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.908 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.908 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:08.908 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885436+0000", 2023-11-05T08:33:08.908 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000136472 2023-11-05T08:33:08.908 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.909 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.909 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:08.909 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885557+0000", 2023-11-05T08:33:08.909 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000121644 2023-11-05T08:33:08.909 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.909 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.910 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:08.910 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885660+0000", 2023-11-05T08:33:08.910 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000103014 2023-11-05T08:33:08.910 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:08.910 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:08.911 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:08.911 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.911 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.911 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:186 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:08.911 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.866343+0000", 2023-11-05T08:33:08.911 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 38.871352078999998, 2023-11-05T08:33:08.912 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.021691589000000001, 2023-11-05T08:33:08.912 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:08.912 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:08.912 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:08.912 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:08.913 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:08.913 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 186 2023-11-05T08:33:08.913 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.913 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:08.913 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.914 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:08.914 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866343+0000", 2023-11-05T08:33:08.914 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:08.914 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.914 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.915 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:08.915 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866343+0000", 2023-11-05T08:33:08.915 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:08.915 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.915 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.915 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:08.916 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866344+0000", 2023-11-05T08:33:08.916 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.6000000000000003e-07 2023-11-05T08:33:08.916 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.916 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.916 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:08.917 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866351+0000", 2023-11-05T08:33:08.917 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3090000000000001e-06 2023-11-05T08:33:08.917 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.917 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.917 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:08.917 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866360+0000", 2023-11-05T08:33:08.918 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.2599999999999994e-06 2023-11-05T08:33:08.918 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.918 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.918 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:08.918 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866367+0000", 2023-11-05T08:33:08.919 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.7599999999999997e-06 2023-11-05T08:33:08.919 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.919 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.919 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:08.919 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.882269+0000", 2023-11-05T08:33:08.919 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.015902218999999999 2023-11-05T08:33:08.920 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.920 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.920 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:08.920 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.882545+0000", 2023-11-05T08:33:08.920 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00027617199999999998 2023-11-05T08:33:08.921 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.921 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.921 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:08.921 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.882925+0000", 2023-11-05T08:33:08.921 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00037995900000000003 2023-11-05T08:33:08.921 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.922 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.922 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:08.922 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.887718+0000", 2023-11-05T08:33:08.922 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0047935069999999998 2023-11-05T08:33:08.922 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.923 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.923 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:08.923 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.887836+0000", 2023-11-05T08:33:08.923 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000117401 2023-11-05T08:33:08.923 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.923 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.924 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:08.924 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.887931+0000", 2023-11-05T08:33:08.924 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.4802000000000005e-05 2023-11-05T08:33:08.924 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.924 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.925 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:08.925 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888035+0000", 2023-11-05T08:33:08.925 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010444 2023-11-05T08:33:08.925 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:08.925 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:08.926 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:08.926 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.926 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.926 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:187 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:08.926 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.866377+0000", 2023-11-05T08:33:08.927 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 38.871318117999998, 2023-11-05T08:33:08.927 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022216751, 2023-11-05T08:33:08.927 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:08.927 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:08.927 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:08.927 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:08.928 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:08.928 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 187 2023-11-05T08:33:08.928 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.928 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:08.928 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.929 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:08.929 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866377+0000", 2023-11-05T08:33:08.929 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:08.929 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.929 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.930 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:08.930 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866377+0000", 2023-11-05T08:33:08.930 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:08.930 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.930 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.931 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:08.931 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866378+0000", 2023-11-05T08:33:08.931 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9999999999999997e-07 2023-11-05T08:33:08.931 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.931 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.931 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:08.932 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866384+0000", 2023-11-05T08:33:08.932 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.4099999999999999e-06 2023-11-05T08:33:08.932 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.932 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.932 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:08.933 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866393+0000", 2023-11-05T08:33:08.933 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9719999999999998e-06 2023-11-05T08:33:08.933 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.933 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.933 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:08.934 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866400+0000", 2023-11-05T08:33:08.934 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.2740000000000002e-06 2023-11-05T08:33:08.934 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.934 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.934 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:08.934 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.883936+0000", 2023-11-05T08:33:08.935 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017536513 2023-11-05T08:33:08.935 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.935 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.935 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:08.935 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.884061+0000", 2023-11-05T08:33:08.936 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012435300000000001 2023-11-05T08:33:08.936 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.936 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.936 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:08.936 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.884277+0000", 2023-11-05T08:33:08.936 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00021665500000000001 2023-11-05T08:33:08.937 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.937 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.937 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:08.937 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888193+0000", 2023-11-05T08:33:08.937 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0039160319999999998 2023-11-05T08:33:08.938 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.938 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.938 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:08.938 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888235+0000", 2023-11-05T08:33:08.938 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1267999999999999e-05 2023-11-05T08:33:08.939 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.939 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.939 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:08.939 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888373+0000", 2023-11-05T08:33:08.939 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000138789 2023-11-05T08:33:08.939 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.940 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.940 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:08.940 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888594+0000", 2023-11-05T08:33:08.940 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00022078500000000001 2023-11-05T08:33:08.940 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:08.941 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:08.941 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:08.941 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.941 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.941 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:188 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:08.941 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.866411+0000", 2023-11-05T08:33:08.942 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 38.871284856999999, 2023-11-05T08:33:08.942 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.023678187999999999, 2023-11-05T08:33:08.942 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:08.942 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:08.942 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:08.943 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:08.943 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:08.943 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 188 2023-11-05T08:33:08.943 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.943 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:08.943 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.944 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:08.944 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866411+0000", 2023-11-05T08:33:08.944 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:08.944 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.944 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.944 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:08.945 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866411+0000", 2023-11-05T08:33:08.945 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:08.945 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.945 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.945 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:08.946 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866411+0000", 2023-11-05T08:33:08.946 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.44e-07 2023-11-05T08:33:08.946 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.946 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.946 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:08.947 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866531+0000", 2023-11-05T08:33:08.947 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000119664 2023-11-05T08:33:08.947 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.947 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.947 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:08.947 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866545+0000", 2023-11-05T08:33:08.948 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3875e-05 2023-11-05T08:33:08.948 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.948 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.948 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:08.948 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866554+0000", 2023-11-05T08:33:08.949 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9099999999999994e-06 2023-11-05T08:33:08.949 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.949 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.949 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:08.949 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885784+0000", 2023-11-05T08:33:08.950 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.019230651000000001 2023-11-05T08:33:08.950 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.950 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.950 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:08.950 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.886020+0000", 2023-11-05T08:33:08.950 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00023538300000000001 2023-11-05T08:33:08.951 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.951 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.951 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:08.951 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.886386+0000", 2023-11-05T08:33:08.951 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00036659399999999998 2023-11-05T08:33:08.952 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.952 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.952 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:08.952 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888662+0000", 2023-11-05T08:33:08.952 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0022756019999999998 2023-11-05T08:33:08.953 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.953 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.953 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:08.953 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.889922+0000", 2023-11-05T08:33:08.953 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0012603149999999999 2023-11-05T08:33:08.953 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.954 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.954 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:08.954 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.890034+0000", 2023-11-05T08:33:08.954 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000111283 2023-11-05T08:33:08.954 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.955 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.955 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:08.955 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.890089+0000", 2023-11-05T08:33:08.955 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5266999999999998e-05 2023-11-05T08:33:08.955 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:08.955 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:08.956 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:08.956 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.956 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.956 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:189 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:08.956 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.866572+0000", 2023-11-05T08:33:08.957 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 38.871123898999997, 2023-11-05T08:33:08.957 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.024895937999999999, 2023-11-05T08:33:08.957 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:08.957 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:08.957 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:08.958 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:08.958 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:08.958 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 189 2023-11-05T08:33:08.958 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.958 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:08.958 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.959 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:08.959 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866572+0000", 2023-11-05T08:33:08.959 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:08.959 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.959 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.960 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:08.960 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866572+0000", 2023-11-05T08:33:08.960 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:08.960 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.960 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.961 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:08.961 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866572+0000", 2023-11-05T08:33:08.961 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.8000000000000005e-07 2023-11-05T08:33:08.961 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.961 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.961 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:08.962 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866581+0000", 2023-11-05T08:33:08.962 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9800000000000004e-06 2023-11-05T08:33:08.962 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.962 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.962 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:08.963 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866591+0000", 2023-11-05T08:33:08.963 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6059999999999992e-06 2023-11-05T08:33:08.963 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.963 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.963 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:08.963 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866598+0000", 2023-11-05T08:33:08.964 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.1029999999999996e-06 2023-11-05T08:33:08.964 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.964 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.964 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:08.964 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888850+0000", 2023-11-05T08:33:08.964 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022251615999999998 2023-11-05T08:33:08.965 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.965 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.965 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:08.965 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888988+0000", 2023-11-05T08:33:08.965 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000138436 2023-11-05T08:33:08.966 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.966 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.966 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:08.966 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.889206+0000", 2023-11-05T08:33:08.966 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00021818099999999999 2023-11-05T08:33:08.967 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.967 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.967 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:08.967 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.890621+0000", 2023-11-05T08:33:08.967 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001414219 2023-11-05T08:33:08.967 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.968 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.968 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:08.968 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.891316+0000", 2023-11-05T08:33:08.968 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00069570500000000002 2023-11-05T08:33:08.968 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.969 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.969 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:08.969 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.891406+0000", 2023-11-05T08:33:08.969 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9901999999999994e-05 2023-11-05T08:33:08.969 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.969 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.970 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:08.970 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.891468+0000", 2023-11-05T08:33:08.970 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1409999999999996e-05 2023-11-05T08:33:08.970 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:08.970 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:08.971 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:08.971 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.971 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.971 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:292 3.2s0 3:41aa5db9:::smithi09039252-7:head [delete] snapc 8=[8,7,6,5] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e41)", 2023-11-05T08:33:08.971 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.121023+0000", 2023-11-05T08:33:08.972 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 14.616672484, 2023-11-05T08:33:08.972 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.061679772000000001, 2023-11-05T08:33:08.972 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:08.972 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:08.972 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:08.972 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:08.973 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:08.973 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 292 2023-11-05T08:33:08.973 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.973 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:08.973 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.974 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:08.974 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.121023+0000", 2023-11-05T08:33:08.974 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:08.974 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.974 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.975 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:08.975 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.121023+0000", 2023-11-05T08:33:08.975 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:08.975 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.975 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.975 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:08.976 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.121026+0000", 2023-11-05T08:33:08.976 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0369999999999999e-06 2023-11-05T08:33:08.976 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.976 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.976 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:08.977 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.121039+0000", 2023-11-05T08:33:08.977 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3103e-05 2023-11-05T08:33:08.977 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.977 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.977 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:08.978 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.121086+0000", 2023-11-05T08:33:08.978 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7095000000000002e-05 2023-11-05T08:33:08.978 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.978 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.978 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:08.978 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.121131+0000", 2023-11-05T08:33:08.979 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4455e-05 2023-11-05T08:33:08.979 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.979 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.979 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:08.979 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.129136+0000", 2023-11-05T08:33:08.980 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0080047439999999994 2023-11-05T08:33:08.980 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.980 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.980 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2023-11-05T08:33:08.980 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.129142+0000", 2023-11-05T08:33:08.981 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3759999999999996e-06 2023-11-05T08:33:08.981 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.981 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.981 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:08.981 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.158095+0000", 2023-11-05T08:33:08.982 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.028953402 2023-11-05T08:33:08.982 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.982 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.982 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:08.982 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.158534+0000", 2023-11-05T08:33:08.982 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00043911800000000001 2023-11-05T08:33:08.983 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.983 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.983 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:08.983 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.160380+0000", 2023-11-05T08:33:08.983 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001845923 2023-11-05T08:33:08.984 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.984 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.984 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:08.984 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.177604+0000", 2023-11-05T08:33:08.984 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017223405000000001 2023-11-05T08:33:08.985 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.985 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.985 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:08.985 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.182556+0000", 2023-11-05T08:33:08.985 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0049522740000000004 2023-11-05T08:33:08.986 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.986 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.986 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:08.986 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.182703+0000", 2023-11-05T08:33:08.986 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014684 2023-11-05T08:33:08.986 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:08.987 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:08.987 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:08.987 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.987 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.987 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:335 3.18s0 3:1aa38538:::smithi09039252-11:head [stat,delete] snapc 10=[10,7] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2023-11-05T08:33:08.988 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:04.225103+0000", 2023-11-05T08:33:08.988 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4.5125926100000004, 2023-11-05T08:33:08.988 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.028941109999999999, 2023-11-05T08:33:08.988 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:08.988 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:08.988 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:08.989 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:08.989 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:08.989 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 335 2023-11-05T08:33:08.989 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.989 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:08.990 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.990 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:08.990 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.225103+0000", 2023-11-05T08:33:08.990 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:08.990 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.990 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.991 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:08.991 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.225103+0000", 2023-11-05T08:33:08.991 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:08.991 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.991 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.991 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:08.992 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.225106+0000", 2023-11-05T08:33:08.992 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0929999999999999e-06 2023-11-05T08:33:08.992 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.992 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.992 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:08.993 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.225118+0000", 2023-11-05T08:33:08.993 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2017e-05 2023-11-05T08:33:08.993 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.993 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.993 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:08.993 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.225155+0000", 2023-11-05T08:33:08.994 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.6711999999999997e-05 2023-11-05T08:33:08.994 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.994 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.994 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:08.994 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.225172+0000", 2023-11-05T08:33:08.994 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7367000000000002e-05 2023-11-05T08:33:08.995 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.995 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.995 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:08.995 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.225312+0000", 2023-11-05T08:33:08.995 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000140323 2023-11-05T08:33:08.996 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.996 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.996 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op must wait for map", 2023-11-05T08:33:08.996 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.225355+0000", 2023-11-05T08:33:08.996 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.2746e-05 2023-11-05T08:33:08.997 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.997 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.997 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:08.997 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.237049+0000", 2023-11-05T08:33:08.997 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011693360999999999 2023-11-05T08:33:08.997 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.998 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.998 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:08.998 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.237652+0000", 2023-11-05T08:33:08.998 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000603749 2023-11-05T08:33:08.998 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.999 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:08.999 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:08.999 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.245013+0000", 2023-11-05T08:33:08.999 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0073607000000000004 2023-11-05T08:33:08.999 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:08.999 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.000 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:09.000 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.249533+0000", 2023-11-05T08:33:09.000 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0045203939999999996 2023-11-05T08:33:09.000 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:09.000 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.001 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:09.001 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.253639+0000", 2023-11-05T08:33:09.001 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0041051730000000002 2023-11-05T08:33:09.001 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:09.001 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.002 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:09.002 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.254044+0000", 2023-11-05T08:33:09.002 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00040547499999999998 2023-11-05T08:33:09.002 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:09.002 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:09.002 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:09.003 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:09.003 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.003 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(osd.5.22:1 3.10s0 3:084c43b8:::smithi09039252-5:head [assert-version v10,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:09.003 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:05.258544+0000", 2023-11-05T08:33:09.003 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 3.479151924, 2023-11-05T08:33:09.004 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.024845459, 2023-11-05T08:33:09.004 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:09.004 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2023-11-05T08:33:09.004 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:09.004 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.005 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:09.005 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258544+0000", 2023-11-05T08:33:09.005 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:09.005 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:09.005 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.005 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:09.006 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258544+0000", 2023-11-05T08:33:09.006 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:09.006 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:09.006 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.006 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:09.007 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258547+0000", 2023-11-05T08:33:09.007 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0740000000000001e-06 2023-11-05T08:33:09.007 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:09.007 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.007 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:09.008 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258561+0000", 2023-11-05T08:33:09.008 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4525999999999999e-05 2023-11-05T08:33:09.008 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:09.008 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.008 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:09.009 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258601+0000", 2023-11-05T08:33:09.009 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.9433e-05 2023-11-05T08:33:09.009 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:09.009 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.009 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:09.009 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258625+0000", 2023-11-05T08:33:09.010 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3819e-05 2023-11-05T08:33:09.010 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:09.010 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.010 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:09.010 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258788+0000", 2023-11-05T08:33:09.011 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00016302600000000001 2023-11-05T08:33:09.011 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:09.011 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.011 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op must wait for map", 2023-11-05T08:33:09.011 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258812+0000", 2023-11-05T08:33:09.012 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4043e-05 2023-11-05T08:33:09.012 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:09.012 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.012 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:09.012 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.275660+0000", 2023-11-05T08:33:09.012 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.016848823999999998 2023-11-05T08:33:09.013 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:09.013 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.013 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:09.013 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.276035+0000", 2023-11-05T08:33:09.013 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00037483200000000002 2023-11-05T08:33:09.014 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:09.014 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.014 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:09.014 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.283389+0000", 2023-11-05T08:33:09.014 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.007353882 2023-11-05T08:33:09.014 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:09.015 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:09.015 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:09.015 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:09.015 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.015 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:351 3.18s0 3:1aa38538:::smithi09039252-11:head [append 0~491520] snapc 10=[7] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:09.016 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:05.264529+0000", 2023-11-05T08:33:09.016 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 3.4731666959999998, 2023-11-05T08:33:09.016 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.029015691, 2023-11-05T08:33:09.016 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:09.016 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:09.016 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:09.017 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:09.017 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:09.017 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 351 2023-11-05T08:33:09.017 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:09.017 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:09.018 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.018 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:09.018 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.264529+0000", 2023-11-05T08:33:09.018 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:09.018 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:09.019 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.019 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:09.019 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.264529+0000", 2023-11-05T08:33:09.019 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:09.019 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:09.019 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.020 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:09.020 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.264532+0000", 2023-11-05T08:33:09.020 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3730000000000001e-06 2023-11-05T08:33:09.020 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:09.020 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.021 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:09.021 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.266714+0000", 2023-11-05T08:33:09.021 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0021819880000000002 2023-11-05T08:33:09.021 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:09.021 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.022 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:09.022 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.266743+0000", 2023-11-05T08:33:09.022 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9218999999999999e-05 2023-11-05T08:33:09.022 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:09.022 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.023 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:09.023 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.266759+0000", 2023-11-05T08:33:09.023 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5102999999999999e-05 2023-11-05T08:33:09.023 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:09.023 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.023 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:09.024 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.266928+0000", 2023-11-05T08:33:09.024 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00016909299999999999 2023-11-05T08:33:09.024 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:09.024 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.024 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op must wait for map", 2023-11-05T08:33:09.025 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.266953+0000", 2023-11-05T08:33:09.025 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5344999999999999e-05 2023-11-05T08:33:09.025 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:09.025 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.025 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:09.026 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.276760+0000", 2023-11-05T08:33:09.026 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0098067109999999992 2023-11-05T08:33:09.026 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:09.026 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.026 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:09.027 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.277064+0000", 2023-11-05T08:33:09.027 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00030468299999999998 2023-11-05T08:33:09.027 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:09.027 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.027 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:09.027 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.279923+0000", 2023-11-05T08:33:09.028 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0028589119999999999 2023-11-05T08:33:09.028 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:09.028 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.028 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:09.028 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.292166+0000", 2023-11-05T08:33:09.029 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012242723 2023-11-05T08:33:09.029 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:09.029 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.029 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:09.029 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.292949+0000", 2023-11-05T08:33:09.029 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00078316699999999998 2023-11-05T08:33:09.030 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:09.030 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.030 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:09.030 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.293545+0000", 2023-11-05T08:33:09.030 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00059537399999999997 2023-11-05T08:33:09.031 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:09.031 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:09.031 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:09.031 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:09.031 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.032 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:354 3.18s0 3:1aa38538:::smithi09039252-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:09.032 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:05.281130+0000", 2023-11-05T08:33:09.032 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 3.456565103, 2023-11-05T08:33:09.032 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.021364054, 2023-11-05T08:33:09.032 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:09.033 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2023-11-05T08:33:09.033 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:09.033 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:09.033 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:09.033 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 354 2023-11-05T08:33:09.034 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:09.034 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:09.034 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.034 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:09.034 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.281130+0000", 2023-11-05T08:33:09.034 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:09.035 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:09.035 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.035 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:09.035 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.281130+0000", 2023-11-05T08:33:09.035 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:09.036 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:09.036 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.036 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:09.036 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.281131+0000", 2023-11-05T08:33:09.036 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.0299999999999998e-07 2023-11-05T08:33:09.037 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:09.037 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.037 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:09.037 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.281135+0000", 2023-11-05T08:33:09.037 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2799999999999999e-06 2023-11-05T08:33:09.037 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:09.038 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.038 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:09.038 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.281146+0000", 2023-11-05T08:33:09.038 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.119e-05 2023-11-05T08:33:09.038 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:09.039 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.039 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:09.039 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.281154+0000", 2023-11-05T08:33:09.039 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.7719999999999994e-06 2023-11-05T08:33:09.039 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:09.040 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.040 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:09.040 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.298629+0000", 2023-11-05T08:33:09.040 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017475195999999998 2023-11-05T08:33:09.040 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:09.041 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.041 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:09.041 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.298839+0000", 2023-11-05T08:33:09.041 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00020991799999999999 2023-11-05T08:33:09.041 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:09.041 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:09.042 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:09.042 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.302495+0000", 2023-11-05T08:33:09.042 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0036558950000000002 2023-11-05T08:33:09.042 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:09.042 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:09.043 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:09.043 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:09.043 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:09.043 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:33:09.045 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-11-05T08:33:09.045 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:33:09.042+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:33:09.145 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:33:09.140+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:33:09.175 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:33:09.176 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2023-11-05T08:33:09.176 INFO:teuthology.orchestra.run.smithi069.stdout: "num_ops": 0 2023-11-05T08:33:09.176 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:33:09.190 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-11-05T08:33:09.246 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:09.240+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:09.272 INFO:tasks.rados.rados.0.smithi090.stdout:148: done (3 left) 2023-11-05T08:33:09.272 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 31 current snap is 11 2023-11-05T08:33:09.272 INFO:tasks.rados.rados.0.smithi090.stdout:149: seq_num 77 ranges {0=524288,524288=524288,1048576=245760} 2023-11-05T08:33:09.280 INFO:tasks.rados.rados.0.smithi090.stdout:149: writing smithi09039252-31 from 0 to 524288 tid 1 2023-11-05T08:33:09.290 INFO:tasks.rados.rados.0.smithi090.stdout:149: writing smithi09039252-31 from 524288 to 1048576 tid 2 2023-11-05T08:33:09.295 INFO:tasks.rados.rados.0.smithi090.stdout:149: writing smithi09039252-31 from 1048576 to 1294336 tid 3 2023-11-05T08:33:09.296 INFO:tasks.rados.rados.0.smithi090.stdout:150: copy_from oid 40 from oid 15 current snap is 11 2023-11-05T08:33:09.296 INFO:tasks.rados.rados.0.smithi090.stdout:147: expect (ObjNum 17 snap 0 seq_num 17) 2023-11-05T08:33:09.315 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:33:09.316 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2023-11-05T08:33:09.316 INFO:teuthology.orchestra.run.smithi069.stdout: "complaint_time": 30, 2023-11-05T08:33:09.317 INFO:teuthology.orchestra.run.smithi069.stdout: "num_blocked_ops": 0 2023-11-05T08:33:09.317 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:33:09.317 INFO:tasks.rados.rados.0.smithi090.stdout:146: expect (ObjNum 9 snap 0 seq_num 9) 2023-11-05T08:33:09.324 INFO:tasks.rados.rados.0.smithi090.stdout:149: finishing write tid 1 to smithi09039252-31 2023-11-05T08:33:09.324 INFO:tasks.rados.rados.0.smithi090.stdout:149: finishing write tid 2 to smithi09039252-31 2023-11-05T08:33:09.325 INFO:tasks.rados.rados.0.smithi090.stdout:149: finishing write tid 3 to smithi09039252-31 2023-11-05T08:33:09.325 INFO:tasks.rados.rados.0.smithi090.stdout:149: finishing write tid 4 to smithi09039252-31 2023-11-05T08:33:09.325 INFO:tasks.rados.rados.0.smithi090.stdout:149: finishing write tid 5 to smithi09039252-31 2023-11-05T08:33:09.325 INFO:tasks.rados.rados.0.smithi090.stdout:149: oid 31 updating version 0 to 20 2023-11-05T08:33:09.325 INFO:tasks.rados.rados.0.smithi090.stdout:149: oid 31 updating version 20 to 22 2023-11-05T08:33:09.326 INFO:tasks.rados.rados.0.smithi090.stdout:149: oid 31 updating version 22 to 23 2023-11-05T08:33:09.326 INFO:tasks.rados.rados.0.smithi090.stdout:149: oid 31 version 23 is already newer than 21 2023-11-05T08:33:09.326 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 31 v 23 (ObjNum 77 snap 11 seq_num 77) dirty exists 2023-11-05T08:33:09.326 INFO:tasks.rados.rados.0.smithi090.stdout:149: left oid 31 (ObjNum 77 snap 11 seq_num 77) 2023-11-05T08:33:09.326 INFO:tasks.rados.rados.0.smithi090.stdout:150: finishing copy_from racing read to smithi09039252-40 2023-11-05T08:33:09.326 INFO:tasks.rados.rados.0.smithi090.stdout:146: done (4 left) 2023-11-05T08:33:09.327 INFO:tasks.rados.rados.0.smithi090.stdout:147: done (3 left) 2023-11-05T08:33:09.327 INFO:tasks.rados.rados.0.smithi090.stdout:149: done (2 left) 2023-11-05T08:33:09.327 INFO:tasks.rados.rados.0.smithi090.stdout:151: snap_create 2023-11-05T08:33:09.327 INFO:tasks.rados.rados.0.smithi090.stdout:150: finishing copy_from to smithi09039252-40 2023-11-05T08:33:09.328 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 40 v 2 (ObjNum 17 snap 0 seq_num 17) dirty exists 2023-11-05T08:33:09.330 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-11-05T08:33:09.375 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:33:09.374+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:33:09.448 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:33:09.446+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:33:09.449 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:33:09.449 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 20, 2023-11-05T08:33:09.450 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 600, 2023-11-05T08:33:09.450 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [ 2023-11-05T08:33:09.450 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.450 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(client.4281.0:115 1.0 e23/22 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 23'77, mlcod=23'73)", 2023-11-05T08:33:09.450 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:29.169871+0000", 2023-11-05T08:33:09.451 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 40.278196991000002, 2023-11-05T08:33:09.451 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.005638522, 2023-11-05T08:33:09.451 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:09.451 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:09.451 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:09.452 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.452 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:09.452 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.169871+0000", 2023-11-05T08:33:09.452 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.452 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.452 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.453 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:09.453 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.169871+0000", 2023-11-05T08:33:09.453 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.453 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.453 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.454 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:09.454 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.169873+0000", 2023-11-05T08:33:09.454 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.61e-06 2023-11-05T08:33:09.454 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.454 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.455 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:09.455 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.169934+0000", 2023-11-05T08:33:09.455 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.1292999999999997e-05 2023-11-05T08:33:09.455 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.455 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.455 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:09.456 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.169962+0000", 2023-11-05T08:33:09.456 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7716999999999999e-05 2023-11-05T08:33:09.456 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.456 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.457 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:09.457 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.169976+0000", 2023-11-05T08:33:09.457 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3815000000000001e-05 2023-11-05T08:33:09.457 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.457 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.458 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:09.458 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.170058+0000", 2023-11-05T08:33:09.458 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.2885000000000003e-05 2023-11-05T08:33:09.458 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.458 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.459 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:33:09.459 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.170089+0000", 2023-11-05T08:33:09.459 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0579e-05 2023-11-05T08:33:09.459 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.459 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.460 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:33:09.460 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.175416+0000", 2023-11-05T08:33:09.460 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0053266169999999996 2023-11-05T08:33:09.460 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.460 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.461 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:09.461 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.175510+0000", 2023-11-05T08:33:09.461 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.4005999999999994e-05 2023-11-05T08:33:09.461 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.461 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:09.461 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.462 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.462 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.462 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(client.4281.0:140 1.0 e23/22 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 23'98, mlcod=23'90)", 2023-11-05T08:33:09.462 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:29.364836+0000", 2023-11-05T08:33:09.462 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 40.083232115999998, 2023-11-05T08:33:09.463 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0063794710000000003, 2023-11-05T08:33:09.463 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:09.463 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:09.463 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:09.463 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.464 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:09.464 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.364836+0000", 2023-11-05T08:33:09.464 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.464 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.464 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.465 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:09.465 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.364836+0000", 2023-11-05T08:33:09.465 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.465 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.465 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.465 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:09.466 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.364838+0000", 2023-11-05T08:33:09.466 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.663e-06 2023-11-05T08:33:09.466 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.466 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.466 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:09.467 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.364935+0000", 2023-11-05T08:33:09.467 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.7565999999999994e-05 2023-11-05T08:33:09.467 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.467 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.467 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:09.468 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.364964+0000", 2023-11-05T08:33:09.468 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8626999999999999e-05 2023-11-05T08:33:09.468 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.468 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.468 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:09.468 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.364977+0000", 2023-11-05T08:33:09.469 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3412e-05 2023-11-05T08:33:09.469 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.469 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.469 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:09.469 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.365131+0000", 2023-11-05T08:33:09.470 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00015339300000000001 2023-11-05T08:33:09.470 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.470 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.470 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:33:09.470 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.365170+0000", 2023-11-05T08:33:09.471 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.9632000000000003e-05 2023-11-05T08:33:09.471 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.471 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.471 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:33:09.471 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.371117+0000", 2023-11-05T08:33:09.471 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0059463249999999997 2023-11-05T08:33:09.472 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.472 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.472 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:09.472 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.371215+0000", 2023-11-05T08:33:09.472 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.8852999999999997e-05 2023-11-05T08:33:09.473 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.473 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:09.473 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.473 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.473 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.473 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.fs1 35/30 ECSubWrite(tid=25, reqid=client.4691.0:228, at_version=35'13, trim_to=0'0, roll_forward_to=33'11))", 2023-11-05T08:33:09.474 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:48.124260+0000", 2023-11-05T08:33:09.474 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 21.323808596999999, 2023-11-05T08:33:09.474 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0069525890000000003, 2023-11-05T08:33:09.474 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:09.474 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:09.475 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:09.475 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.475 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:09.475 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.124260+0000", 2023-11-05T08:33:09.475 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.476 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.476 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.476 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:09.476 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.124260+0000", 2023-11-05T08:33:09.476 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.477 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.477 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.477 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:09.477 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.124260+0000", 2023-11-05T08:33:09.477 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.6300000000000001e-07 2023-11-05T08:33:09.478 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.478 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.478 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:09.478 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.125159+0000", 2023-11-05T08:33:09.478 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000898886 2023-11-05T08:33:09.478 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.479 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.479 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:09.479 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.125202+0000", 2023-11-05T08:33:09.479 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.3040999999999997e-05 2023-11-05T08:33:09.479 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.480 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.480 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:09.480 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.125227+0000", 2023-11-05T08:33:09.480 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4774e-05 2023-11-05T08:33:09.480 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.480 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.481 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:09.481 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.125405+0000", 2023-11-05T08:33:09.481 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00017825600000000001 2023-11-05T08:33:09.481 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.481 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.482 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:09.482 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.125435+0000", 2023-11-05T08:33:09.482 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0148e-05 2023-11-05T08:33:09.482 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.482 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.482 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:09.483 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.131127+0000", 2023-11-05T08:33:09.483 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0056919070000000004 2023-11-05T08:33:09.483 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.483 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.483 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:09.484 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.131212+0000", 2023-11-05T08:33:09.484 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.5214000000000001e-05 2023-11-05T08:33:09.484 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.484 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:09.484 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.484 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.485 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.485 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.4s1 36/30 ECSubWrite(tid=10, reqid=osd.8.0:10, at_version=36'8, trim_to=0'0, roll_forward_to=34'6))", 2023-11-05T08:33:09.485 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.079150+0000", 2023-11-05T08:33:09.485 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 20.368917998000001, 2023-11-05T08:33:09.485 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00708488, 2023-11-05T08:33:09.486 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:09.486 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:09.486 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:09.486 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.486 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:09.487 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079150+0000", 2023-11-05T08:33:09.487 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.487 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.487 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.487 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:09.488 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079150+0000", 2023-11-05T08:33:09.488 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.488 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.488 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.488 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:09.488 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079151+0000", 2023-11-05T08:33:09.489 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.8299999999999996e-07 2023-11-05T08:33:09.489 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.489 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.489 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:09.489 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079199+0000", 2023-11-05T08:33:09.490 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.7726999999999999e-05 2023-11-05T08:33:09.490 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.490 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.490 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:09.490 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079227+0000", 2023-11-05T08:33:09.490 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8772000000000002e-05 2023-11-05T08:33:09.491 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.491 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.491 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:09.491 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079250+0000", 2023-11-05T08:33:09.491 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2422e-05 2023-11-05T08:33:09.492 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.492 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.492 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:09.492 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079718+0000", 2023-11-05T08:33:09.492 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00046772399999999998 2023-11-05T08:33:09.493 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.493 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.493 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:09.493 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079741+0000", 2023-11-05T08:33:09.493 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3659000000000002e-05 2023-11-05T08:33:09.493 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.494 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.494 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:09.494 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.086158+0000", 2023-11-05T08:33:09.494 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0064169769999999999 2023-11-05T08:33:09.494 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.495 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.495 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:09.495 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.086235+0000", 2023-11-05T08:33:09.495 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.6816000000000005e-05 2023-11-05T08:33:09.495 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.496 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:09.496 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.496 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.496 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.496 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.8s2 36/30 ECSubWrite(tid=23, reqid=osd.5.0:23, at_version=36'15, trim_to=0'0, roll_forward_to=34'13))", 2023-11-05T08:33:09.496 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.079405+0000", 2023-11-05T08:33:09.497 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 20.368662871000002, 2023-11-05T08:33:09.497 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0067666469999999998, 2023-11-05T08:33:09.497 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:09.497 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:09.497 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:09.498 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.498 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:09.498 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079405+0000", 2023-11-05T08:33:09.498 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.498 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.499 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.499 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:09.499 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079405+0000", 2023-11-05T08:33:09.499 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.499 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.499 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.500 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:09.500 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079406+0000", 2023-11-05T08:33:09.500 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.9999999999999998e-07 2023-11-05T08:33:09.500 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.500 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.501 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:09.501 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079446+0000", 2023-11-05T08:33:09.501 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.0034999999999999e-05 2023-11-05T08:33:09.501 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.501 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.501 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:09.502 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079490+0000", 2023-11-05T08:33:09.502 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.4413999999999999e-05 2023-11-05T08:33:09.502 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.502 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.502 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:09.503 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079509+0000", 2023-11-05T08:33:09.503 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8828999999999999e-05 2023-11-05T08:33:09.503 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.503 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.503 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:09.504 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079677+0000", 2023-11-05T08:33:09.504 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00016785199999999999 2023-11-05T08:33:09.504 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.504 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.504 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:09.505 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079702+0000", 2023-11-05T08:33:09.505 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5072e-05 2023-11-05T08:33:09.505 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.505 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.505 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:09.505 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.086071+0000", 2023-11-05T08:33:09.506 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0063687170000000003 2023-11-05T08:33:09.506 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.506 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.506 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:09.506 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.086172+0000", 2023-11-05T08:33:09.507 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000101228 2023-11-05T08:33:09.507 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.507 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:09.507 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.507 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.507 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.508 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:274 3.es0 3:74767c3c:::smithi09039252-8:head [append 0~737280] snapc 8=[8,7,6,5,3] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:33:09.508 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:53.134268+0000", 2023-11-05T08:33:09.508 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 16.313799841000002, 2023-11-05T08:33:09.508 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.014769740999999999, 2023-11-05T08:33:09.508 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:09.509 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:09.509 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:33:09.509 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:33:09.509 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:09.509 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 274 2023-11-05T08:33:09.509 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.510 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:09.510 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.510 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:09.510 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.134268+0000", 2023-11-05T08:33:09.510 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.511 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.511 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.511 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:09.511 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.134268+0000", 2023-11-05T08:33:09.511 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.511 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.512 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.512 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:09.512 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.134270+0000", 2023-11-05T08:33:09.512 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8560000000000001e-06 2023-11-05T08:33:09.512 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.513 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.513 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:09.513 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.135843+0000", 2023-11-05T08:33:09.513 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0015724210000000001 2023-11-05T08:33:09.513 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.514 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.514 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:09.514 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.135876+0000", 2023-11-05T08:33:09.514 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.3895e-05 2023-11-05T08:33:09.514 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.514 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.515 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:09.515 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.135929+0000", 2023-11-05T08:33:09.515 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.2930000000000003e-05 2023-11-05T08:33:09.515 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.515 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.516 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:09.516 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.136066+0000", 2023-11-05T08:33:09.516 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000136673 2023-11-05T08:33:09.516 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.516 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.517 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:33:09.517 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.136335+0000", 2023-11-05T08:33:09.517 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00026863599999999999 2023-11-05T08:33:09.517 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.517 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.517 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:09.518 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.138586+0000", 2023-11-05T08:33:09.518 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0022515629999999998 2023-11-05T08:33:09.518 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.518 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.518 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:09.519 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.148630+0000", 2023-11-05T08:33:09.519 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.010043778 2023-11-05T08:33:09.519 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.519 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.519 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:33:09.519 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.148798+0000", 2023-11-05T08:33:09.520 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00016756799999999999 2023-11-05T08:33:09.520 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.520 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.520 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:09.520 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.149038+0000", 2023-11-05T08:33:09.521 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00024042099999999999 2023-11-05T08:33:09.521 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.521 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:09.521 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.521 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.521 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.522 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:275 3.es0 3:74767c3c:::smithi09039252-8:head [append 0~376832] snapc 8=[8,7,6,5,3] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:33:09.522 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:53.142766+0000", 2023-11-05T08:33:09.522 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 16.305302425000001, 2023-11-05T08:33:09.522 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0071270860000000004, 2023-11-05T08:33:09.522 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:09.523 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:09.523 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:33:09.523 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:33:09.523 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:09.523 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 275 2023-11-05T08:33:09.524 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.524 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:09.524 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.524 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:09.524 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.142766+0000", 2023-11-05T08:33:09.524 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.525 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.525 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.525 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:09.525 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.142766+0000", 2023-11-05T08:33:09.525 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.526 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.526 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.526 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:09.526 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.142768+0000", 2023-11-05T08:33:09.526 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3989999999999998e-06 2023-11-05T08:33:09.527 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.527 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.527 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:09.527 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143223+0000", 2023-11-05T08:33:09.527 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00045515200000000001 2023-11-05T08:33:09.528 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.528 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.528 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:09.528 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143247+0000", 2023-11-05T08:33:09.528 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3898999999999999e-05 2023-11-05T08:33:09.528 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.529 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.529 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:09.529 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143266+0000", 2023-11-05T08:33:09.529 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8372999999999998e-05 2023-11-05T08:33:09.529 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.529 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.530 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:09.530 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143492+0000", 2023-11-05T08:33:09.530 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000226319 2023-11-05T08:33:09.530 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.530 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.531 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:33:09.531 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143751+0000", 2023-11-05T08:33:09.531 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00025883200000000002 2023-11-05T08:33:09.531 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.531 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.532 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:09.532 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.145441+0000", 2023-11-05T08:33:09.532 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0016905620000000001 2023-11-05T08:33:09.532 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.532 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.532 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:09.533 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.149302+0000", 2023-11-05T08:33:09.533 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0038602860000000001 2023-11-05T08:33:09.533 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.533 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.533 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:33:09.534 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.149825+0000", 2023-11-05T08:33:09.534 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00052301300000000002 2023-11-05T08:33:09.534 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.534 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.534 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:09.534 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.149893+0000", 2023-11-05T08:33:09.535 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.8250999999999994e-05 2023-11-05T08:33:09.535 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.535 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:09.535 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.535 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.536 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.536 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:276 3.es0 3:74767c3c:::smithi09039252-8:head [setxattr _header (55) in=7b] snapc 8=[8,7,6,5,3] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:33:09.536 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:53.143364+0000", 2023-11-05T08:33:09.536 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 16.304704470000001, 2023-11-05T08:33:09.536 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0074923510000000004, 2023-11-05T08:33:09.537 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:09.537 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:09.537 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:33:09.537 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:33:09.537 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:09.537 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 276 2023-11-05T08:33:09.538 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.538 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:09.538 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.538 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:09.538 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143364+0000", 2023-11-05T08:33:09.539 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.539 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.539 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.539 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:09.539 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143364+0000", 2023-11-05T08:33:09.539 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.540 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.540 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.540 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:09.540 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143366+0000", 2023-11-05T08:33:09.540 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0420000000000001e-06 2023-11-05T08:33:09.541 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.541 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.541 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:09.541 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143375+0000", 2023-11-05T08:33:09.541 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.5429999999999999e-06 2023-11-05T08:33:09.542 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.542 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.542 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:09.542 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143407+0000", 2023-11-05T08:33:09.542 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.1563999999999998e-05 2023-11-05T08:33:09.542 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.543 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.543 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:09.543 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143421+0000", 2023-11-05T08:33:09.543 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4479e-05 2023-11-05T08:33:09.543 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.544 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.544 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:09.544 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.146951+0000", 2023-11-05T08:33:09.544 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.003530156 2023-11-05T08:33:09.544 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.545 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.545 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:33:09.545 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.147140+0000", 2023-11-05T08:33:09.545 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00018828000000000001 2023-11-05T08:33:09.545 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.546 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.546 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:09.546 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.148038+0000", 2023-11-05T08:33:09.546 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00089799300000000003 2023-11-05T08:33:09.546 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.546 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.547 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:09.547 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.149508+0000", 2023-11-05T08:33:09.547 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001470652 2023-11-05T08:33:09.547 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.547 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.548 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:33:09.548 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.150659+0000", 2023-11-05T08:33:09.548 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001150376 2023-11-05T08:33:09.548 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.548 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.548 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:09.549 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.150856+0000", 2023-11-05T08:33:09.549 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000197266 2023-11-05T08:33:09.549 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.549 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:09.549 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.549 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.550 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.550 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:277 3.es0 3:74767c3c:::smithi09039252-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:33:09.550 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:53.143452+0000", 2023-11-05T08:33:09.550 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 16.304615857999998, 2023-11-05T08:33:09.550 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0099967070000000005, 2023-11-05T08:33:09.551 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:09.551 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2023-11-05T08:33:09.551 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:33:09.551 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:33:09.551 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:09.551 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 277 2023-11-05T08:33:09.552 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.552 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:09.552 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.552 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:09.552 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143452+0000", 2023-11-05T08:33:09.553 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.553 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.553 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.553 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:09.553 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143452+0000", 2023-11-05T08:33:09.554 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.554 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.554 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.554 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:09.554 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143453+0000", 2023-11-05T08:33:09.555 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0640000000000001e-06 2023-11-05T08:33:09.555 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.555 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.555 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:09.555 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143459+0000", 2023-11-05T08:33:09.555 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.5330000000000002e-06 2023-11-05T08:33:09.556 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.556 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.556 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:09.556 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143491+0000", 2023-11-05T08:33:09.556 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2156000000000002e-05 2023-11-05T08:33:09.557 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.557 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.557 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:09.557 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143504+0000", 2023-11-05T08:33:09.557 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3156000000000001e-05 2023-11-05T08:33:09.557 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.558 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.558 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:09.558 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.150256+0000", 2023-11-05T08:33:09.558 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0067515980000000001 2023-11-05T08:33:09.558 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.559 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.559 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for rw locks", 2023-11-05T08:33:09.559 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.150342+0000", 2023-11-05T08:33:09.559 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.5951000000000004e-05 2023-11-05T08:33:09.559 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.560 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.560 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:09.560 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.151851+0000", 2023-11-05T08:33:09.560 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0015094590000000001 2023-11-05T08:33:09.560 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.560 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.561 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:33:09.561 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.151933+0000", 2023-11-05T08:33:09.561 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.1898999999999996e-05 2023-11-05T08:33:09.561 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.561 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.562 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:09.562 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.153449+0000", 2023-11-05T08:33:09.562 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0015158909999999999 2023-11-05T08:33:09.562 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.562 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:09.562 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.563 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.563 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.563 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(osd.8.22:1 3.es0 3:7a868416:::smithi09039252-27:head [assert-version v10,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2023-11-05T08:33:09.563 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.173321+0000", 2023-11-05T08:33:09.563 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 15.274746772, 2023-11-05T08:33:09.564 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0063558599999999996, 2023-11-05T08:33:09.564 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:09.564 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2023-11-05T08:33:09.564 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:09.564 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.564 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:09.565 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.173321+0000", 2023-11-05T08:33:09.565 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.565 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.565 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.565 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:09.566 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.173321+0000", 2023-11-05T08:33:09.566 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.566 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.566 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.566 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:09.567 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.173324+0000", 2023-11-05T08:33:09.567 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8569999999999999e-06 2023-11-05T08:33:09.567 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.567 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.567 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:09.568 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.173336+0000", 2023-11-05T08:33:09.568 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.166e-05 2023-11-05T08:33:09.568 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.568 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.568 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:09.568 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.173409+0000", 2023-11-05T08:33:09.569 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.3079999999999998e-05 2023-11-05T08:33:09.569 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.569 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.569 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:09.569 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.173430+0000", 2023-11-05T08:33:09.570 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0579000000000001e-05 2023-11-05T08:33:09.570 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.570 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.570 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:09.570 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.173579+0000", 2023-11-05T08:33:09.571 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00014947199999999999 2023-11-05T08:33:09.571 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.571 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.571 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:33:09.571 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.173982+0000", 2023-11-05T08:33:09.571 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00040297899999999998 2023-11-05T08:33:09.572 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.572 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.572 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:09.572 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.179677+0000", 2023-11-05T08:33:09.572 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0056952330000000001 2023-11-05T08:33:09.572 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.573 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:09.573 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.573 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.573 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.573 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.11s5 41/40 ECSubWrite(tid=65, reqid=client.4691.0:287, at_version=41'13, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:09.574 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.173811+0000", 2023-11-05T08:33:09.574 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 15.274257263999999, 2023-11-05T08:33:09.574 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0074165059999999998, 2023-11-05T08:33:09.574 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:09.574 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:09.574 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:09.575 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.575 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:09.575 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.173811+0000", 2023-11-05T08:33:09.575 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.575 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.576 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.576 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:09.576 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.173811+0000", 2023-11-05T08:33:09.576 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.576 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.576 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.577 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:09.577 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.173812+0000", 2023-11-05T08:33:09.577 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.485e-06 2023-11-05T08:33:09.577 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.577 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.578 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:09.578 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.174484+0000", 2023-11-05T08:33:09.578 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00067199399999999998 2023-11-05T08:33:09.578 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.578 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.579 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:09.579 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.174513+0000", 2023-11-05T08:33:09.579 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8271000000000001e-05 2023-11-05T08:33:09.579 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.579 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.580 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:09.580 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.174531+0000", 2023-11-05T08:33:09.580 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8433000000000001e-05 2023-11-05T08:33:09.580 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.580 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.580 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:09.581 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.174759+0000", 2023-11-05T08:33:09.581 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00022838599999999999 2023-11-05T08:33:09.581 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.581 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.581 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:09.582 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.174781+0000", 2023-11-05T08:33:09.582 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1273000000000001e-05 2023-11-05T08:33:09.582 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.582 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.582 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:09.583 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.181145+0000", 2023-11-05T08:33:09.583 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0063640140000000003 2023-11-05T08:33:09.583 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.583 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.583 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:09.583 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.181227+0000", 2023-11-05T08:33:09.584 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.2650000000000003e-05 2023-11-05T08:33:09.584 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.584 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:09.584 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.584 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.585 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.585 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.11s5 41/40 ECSubWrite(tid=66, reqid=client.4691.0:288, at_version=41'14, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:09.585 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.177831+0000", 2023-11-05T08:33:09.585 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 15.27023745, 2023-11-05T08:33:09.585 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0064294060000000004, 2023-11-05T08:33:09.586 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:09.586 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:09.586 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:09.586 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.586 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:09.586 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.177831+0000", 2023-11-05T08:33:09.587 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.587 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.587 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.587 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:09.587 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.177831+0000", 2023-11-05T08:33:09.588 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.588 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.588 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.588 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:09.588 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.177832+0000", 2023-11-05T08:33:09.588 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4419999999999999e-06 2023-11-05T08:33:09.589 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.589 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.589 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:09.589 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.178026+0000", 2023-11-05T08:33:09.589 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00019376800000000001 2023-11-05T08:33:09.590 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.590 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.590 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:09.590 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.178053+0000", 2023-11-05T08:33:09.590 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6650000000000001e-05 2023-11-05T08:33:09.591 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.591 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.591 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:09.591 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.178070+0000", 2023-11-05T08:33:09.591 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.7385e-05 2023-11-05T08:33:09.592 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.592 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.592 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:09.592 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.178199+0000", 2023-11-05T08:33:09.592 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00012860700000000001 2023-11-05T08:33:09.592 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.593 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.593 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:09.593 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.178256+0000", 2023-11-05T08:33:09.593 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.7639000000000002e-05 2023-11-05T08:33:09.593 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.594 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.594 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:09.594 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184156+0000", 2023-11-05T08:33:09.594 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0058995260000000004 2023-11-05T08:33:09.594 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.595 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.595 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:09.595 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184260+0000", 2023-11-05T08:33:09.595 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00010438900000000001 2023-11-05T08:33:09.595 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.595 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:09.596 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.596 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.596 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.596 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.11s5 41/40 ECSubWrite(tid=67, reqid=client.4691.0:289, at_version=41'15, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:09.596 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.180455+0000", 2023-11-05T08:33:09.597 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 15.267612725999999, 2023-11-05T08:33:09.597 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0062528530000000001, 2023-11-05T08:33:09.597 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:09.597 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:09.597 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:09.598 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.598 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:09.598 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.180455+0000", 2023-11-05T08:33:09.598 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.598 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.599 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.599 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:09.599 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.180455+0000", 2023-11-05T08:33:09.599 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.599 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.600 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.600 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:09.600 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.180457+0000", 2023-11-05T08:33:09.600 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4580000000000001e-06 2023-11-05T08:33:09.600 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.601 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.601 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:09.601 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.180489+0000", 2023-11-05T08:33:09.601 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2292999999999997e-05 2023-11-05T08:33:09.601 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.601 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.602 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:09.602 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.180511+0000", 2023-11-05T08:33:09.602 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2357e-05 2023-11-05T08:33:09.602 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.602 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.603 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:09.603 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.180528+0000", 2023-11-05T08:33:09.603 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6138999999999999e-05 2023-11-05T08:33:09.603 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.603 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.603 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:09.604 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.180734+0000", 2023-11-05T08:33:09.604 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00020627199999999999 2023-11-05T08:33:09.604 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.604 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.604 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:09.605 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.180756+0000", 2023-11-05T08:33:09.605 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2600999999999999e-05 2023-11-05T08:33:09.605 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.605 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.605 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:09.606 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.186628+0000", 2023-11-05T08:33:09.606 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0058710419999999999 2023-11-05T08:33:09.606 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.606 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.606 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:09.607 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.186708+0000", 2023-11-05T08:33:09.607 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.0691000000000004e-05 2023-11-05T08:33:09.607 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.607 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:09.607 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.607 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.608 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.608 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.2s1 41/40 ECSubWrite(tid=189, reqid=osd.13.0:188, at_version=0'0, trim_to=0'0, roll_forward_to=41'11))", 2023-11-05T08:33:09.608 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.184426+0000", 2023-11-05T08:33:09.608 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 15.263641614999999, 2023-11-05T08:33:09.608 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0057920330000000003, 2023-11-05T08:33:09.609 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:09.609 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:09.609 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:09.609 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.609 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:09.610 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184426+0000", 2023-11-05T08:33:09.610 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.610 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.610 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.610 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:09.611 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184426+0000", 2023-11-05T08:33:09.611 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.611 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.611 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.611 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:09.611 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184428+0000", 2023-11-05T08:33:09.612 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5510000000000001e-06 2023-11-05T08:33:09.612 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.612 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.612 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:09.612 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184451+0000", 2023-11-05T08:33:09.612 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2608000000000001e-05 2023-11-05T08:33:09.613 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.613 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.613 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:09.613 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184482+0000", 2023-11-05T08:33:09.613 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.1458000000000003e-05 2023-11-05T08:33:09.614 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.614 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.614 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:09.614 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184520+0000", 2023-11-05T08:33:09.614 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.7815000000000003e-05 2023-11-05T08:33:09.614 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.615 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.615 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:09.615 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184645+0000", 2023-11-05T08:33:09.615 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00012552099999999999 2023-11-05T08:33:09.615 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.616 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.616 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:09.616 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184667+0000", 2023-11-05T08:33:09.616 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1997e-05 2023-11-05T08:33:09.616 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.616 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.617 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:09.617 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.190155+0000", 2023-11-05T08:33:09.617 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0054879359999999997 2023-11-05T08:33:09.617 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.617 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.618 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:09.618 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.190219+0000", 2023-11-05T08:33:09.618 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.3146999999999997e-05 2023-11-05T08:33:09.618 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.618 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:09.619 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.619 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.619 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.619 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.14s1 41/40 ECSubWrite(tid=17, reqid=client.4691.0:278, at_version=41'14, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:09.619 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.185630+0000", 2023-11-05T08:33:09.620 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 15.262438263, 2023-11-05T08:33:09.620 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0070243399999999996, 2023-11-05T08:33:09.620 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:09.620 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:09.620 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:09.621 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.621 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:09.621 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.185630+0000", 2023-11-05T08:33:09.621 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.621 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.621 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.622 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:09.622 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.185630+0000", 2023-11-05T08:33:09.622 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.622 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.622 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.623 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:09.623 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.185631+0000", 2023-11-05T08:33:09.623 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3480000000000001e-06 2023-11-05T08:33:09.623 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.623 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.623 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:09.624 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.186192+0000", 2023-11-05T08:33:09.624 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00056080799999999999 2023-11-05T08:33:09.624 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.624 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.624 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:09.625 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.186216+0000", 2023-11-05T08:33:09.625 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4048000000000001e-05 2023-11-05T08:33:09.625 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.625 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.625 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:09.625 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.186247+0000", 2023-11-05T08:33:09.626 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0505999999999999e-05 2023-11-05T08:33:09.626 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.626 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.626 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:09.626 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.186387+0000", 2023-11-05T08:33:09.627 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00014087999999999999 2023-11-05T08:33:09.627 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.627 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.627 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:09.627 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.186407+0000", 2023-11-05T08:33:09.628 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9160999999999998e-05 2023-11-05T08:33:09.628 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.628 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.628 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:09.628 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.192585+0000", 2023-11-05T08:33:09.628 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0061782080000000001 2023-11-05T08:33:09.629 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.629 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.629 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:09.629 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.192654+0000", 2023-11-05T08:33:09.629 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.9381000000000005e-05 2023-11-05T08:33:09.629 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.630 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:09.630 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.630 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.630 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.630 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.fs1 45/37 ECSubWrite(tid=40, reqid=client.4691.0:313, at_version=45'23, trim_to=0'0, roll_forward_to=38'21))", 2023-11-05T08:33:09.631 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:58.201456+0000", 2023-11-05T08:33:09.631 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 11.24661175, 2023-11-05T08:33:09.631 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0078928130000000003, 2023-11-05T08:33:09.631 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:09.631 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:09.632 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:09.632 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.632 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:09.632 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.201456+0000", 2023-11-05T08:33:09.632 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.633 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.633 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.633 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:09.633 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.201456+0000", 2023-11-05T08:33:09.633 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.633 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.634 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.634 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:09.634 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.201464+0000", 2023-11-05T08:33:09.634 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.4830000000000002e-06 2023-11-05T08:33:09.634 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.635 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.635 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:09.635 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.202475+0000", 2023-11-05T08:33:09.635 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001010911 2023-11-05T08:33:09.635 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.636 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.636 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:09.636 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.202500+0000", 2023-11-05T08:33:09.636 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5327999999999999e-05 2023-11-05T08:33:09.636 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.636 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.637 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:09.637 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.202520+0000", 2023-11-05T08:33:09.637 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9922999999999999e-05 2023-11-05T08:33:09.637 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.637 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.637 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:09.638 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.202653+0000", 2023-11-05T08:33:09.638 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00013329800000000001 2023-11-05T08:33:09.638 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.638 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.638 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:09.639 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.202672+0000", 2023-11-05T08:33:09.639 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8864000000000001e-05 2023-11-05T08:33:09.639 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.639 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.639 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:09.640 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.209293+0000", 2023-11-05T08:33:09.640 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0066211719999999998 2023-11-05T08:33:09.640 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.640 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.640 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:09.640 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.209349+0000", 2023-11-05T08:33:09.641 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.5834000000000001e-05 2023-11-05T08:33:09.641 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.641 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:09.641 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.641 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.641 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.642 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.fs1 45/37 ECSubWrite(tid=41, reqid=client.4691.0:314, at_version=45'24, trim_to=0'0, roll_forward_to=38'21))", 2023-11-05T08:33:09.642 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:58.203830+0000", 2023-11-05T08:33:09.642 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 11.244238466000001, 2023-11-05T08:33:09.642 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.007519492, 2023-11-05T08:33:09.642 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:09.643 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:09.643 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:09.643 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.643 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:09.643 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.203830+0000", 2023-11-05T08:33:09.643 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.644 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.644 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.644 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:09.644 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.203830+0000", 2023-11-05T08:33:09.644 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.645 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.645 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.645 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:09.645 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.203853+0000", 2023-11-05T08:33:09.645 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3842e-05 2023-11-05T08:33:09.645 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.646 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.646 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:09.646 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.204181+0000", 2023-11-05T08:33:09.646 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000327401 2023-11-05T08:33:09.646 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.647 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.647 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:09.647 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.204212+0000", 2023-11-05T08:33:09.647 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.1596e-05 2023-11-05T08:33:09.647 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.648 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.648 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:09.648 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.204235+0000", 2023-11-05T08:33:09.648 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2319000000000001e-05 2023-11-05T08:33:09.648 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.649 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.649 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:09.649 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.206582+0000", 2023-11-05T08:33:09.649 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0023471669999999998 2023-11-05T08:33:09.649 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.649 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.650 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:09.650 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.206606+0000", 2023-11-05T08:33:09.650 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3864999999999999e-05 2023-11-05T08:33:09.650 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.650 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.651 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:09.651 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.211313+0000", 2023-11-05T08:33:09.651 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0047070539999999996 2023-11-05T08:33:09.651 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.651 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.652 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:09.652 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.211349+0000", 2023-11-05T08:33:09.652 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.6248000000000003e-05 2023-11-05T08:33:09.652 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.652 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:09.652 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.653 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.653 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.653 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.fs1 45/37 ECSubWrite(tid=42, reqid=client.4691.0:315, at_version=45'25, trim_to=0'0, roll_forward_to=38'21))", 2023-11-05T08:33:09.653 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:58.205469+0000", 2023-11-05T08:33:09.653 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 11.242598904999999, 2023-11-05T08:33:09.653 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0069671580000000002, 2023-11-05T08:33:09.654 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:09.654 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:09.654 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:09.654 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.654 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:09.655 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.205469+0000", 2023-11-05T08:33:09.655 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.655 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.655 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.655 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:09.656 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.205469+0000", 2023-11-05T08:33:09.656 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.656 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.656 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.656 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:09.656 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.205471+0000", 2023-11-05T08:33:09.657 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6300000000000001e-06 2023-11-05T08:33:09.657 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.657 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.657 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:09.657 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.205521+0000", 2023-11-05T08:33:09.658 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.0593999999999997e-05 2023-11-05T08:33:09.658 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.658 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.658 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:09.658 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.205551+0000", 2023-11-05T08:33:09.659 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.9555000000000001e-05 2023-11-05T08:33:09.659 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.659 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.659 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:09.659 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.205571+0000", 2023-11-05T08:33:09.659 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0446e-05 2023-11-05T08:33:09.660 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.660 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.660 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:09.660 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.208849+0000", 2023-11-05T08:33:09.660 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0032774129999999999 2023-11-05T08:33:09.661 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.661 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.661 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:09.661 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.208871+0000", 2023-11-05T08:33:09.661 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2582000000000002e-05 2023-11-05T08:33:09.661 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.662 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.662 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:09.662 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.212407+0000", 2023-11-05T08:33:09.662 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0035357750000000001 2023-11-05T08:33:09.662 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.663 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.663 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:09.663 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.212436+0000", 2023-11-05T08:33:09.663 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.9162999999999998e-05 2023-11-05T08:33:09.663 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.664 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:09.664 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.664 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.664 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.664 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.18s2 51/45 ECSubWrite(tid=202, reqid=client.4691.0:335, at_version=51'23, trim_to=0'0, roll_forward_to=49'21))", 2023-11-05T08:33:09.664 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:04.243715+0000", 2023-11-05T08:33:09.665 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 5.204353115, 2023-11-05T08:33:09.665 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0057937880000000002, 2023-11-05T08:33:09.665 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:09.665 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:09.665 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:09.666 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.666 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:09.666 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:04.243715+0000", 2023-11-05T08:33:09.666 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.666 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.667 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.667 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:09.667 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:04.243715+0000", 2023-11-05T08:33:09.667 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.667 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.667 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.668 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:09.668 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:04.243717+0000", 2023-11-05T08:33:09.668 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.793e-06 2023-11-05T08:33:09.668 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.668 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.669 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:09.669 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:04.243788+0000", 2023-11-05T08:33:09.669 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.0727000000000002e-05 2023-11-05T08:33:09.669 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.669 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.670 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:09.670 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:04.243815+0000", 2023-11-05T08:33:09.670 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7682e-05 2023-11-05T08:33:09.670 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.670 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.671 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:09.671 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:04.243839+0000", 2023-11-05T08:33:09.671 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3569000000000001e-05 2023-11-05T08:33:09.671 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.671 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.672 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:09.672 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:04.243998+0000", 2023-11-05T08:33:09.672 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00015875700000000001 2023-11-05T08:33:09.672 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.672 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.673 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:09.673 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:04.244027+0000", 2023-11-05T08:33:09.673 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.9539e-05 2023-11-05T08:33:09.673 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.673 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.673 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:09.674 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:04.249418+0000", 2023-11-05T08:33:09.674 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0053905009999999998 2023-11-05T08:33:09.674 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.674 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.674 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:09.675 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:04.249509+0000", 2023-11-05T08:33:09.675 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.1219999999999997e-05 2023-11-05T08:33:09.675 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.675 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:09.675 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.675 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.676 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.676 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.18s2 52/45 ECSubWrite(tid=205, reqid=client.4691.0:351, at_version=52'24, trim_to=0'0, roll_forward_to=51'23))", 2023-11-05T08:33:09.676 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:05.282039+0000", 2023-11-05T08:33:09.676 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 4.1660291320000002, 2023-11-05T08:33:09.676 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0066848970000000004, 2023-11-05T08:33:09.676 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:09.677 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:09.677 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:09.677 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.677 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:09.677 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.282039+0000", 2023-11-05T08:33:09.677 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.678 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.678 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.678 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:09.678 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.282039+0000", 2023-11-05T08:33:09.678 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:09.679 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.679 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.679 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:09.679 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.282040+0000", 2023-11-05T08:33:09.679 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.1999999999999999e-07 2023-11-05T08:33:09.679 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.680 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.680 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:09.680 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.282608+0000", 2023-11-05T08:33:09.680 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00056786599999999999 2023-11-05T08:33:09.680 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.681 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.681 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:09.681 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.282651+0000", 2023-11-05T08:33:09.681 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.3089e-05 2023-11-05T08:33:09.681 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.681 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.682 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:09.682 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.282674+0000", 2023-11-05T08:33:09.682 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3329e-05 2023-11-05T08:33:09.682 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.682 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.683 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:09.683 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.282911+0000", 2023-11-05T08:33:09.683 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000237443 2023-11-05T08:33:09.683 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.683 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.683 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:09.684 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.282943+0000", 2023-11-05T08:33:09.684 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.1928999999999999e-05 2023-11-05T08:33:09.684 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.684 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.684 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:09.684 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.288631+0000", 2023-11-05T08:33:09.685 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0056872720000000002 2023-11-05T08:33:09.685 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:09.685 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:09.685 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:09.685 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.288724+0000", 2023-11-05T08:33:09.686 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.3249000000000001e-05 2023-11-05T08:33:09.686 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.686 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:09.686 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.686 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:09.686 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:09.687 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:33:09.688 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-11-05T08:33:09.689 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:33:09.686+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:33:09.779 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [3] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 7, 11, 15] 2023-11-05T08:33:09.779 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-11-05T08:33:09.779 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-05T08:33:09.789 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:33:09.786+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:33:09.861 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:33:09.861 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2023-11-05T08:33:09.862 INFO:teuthology.orchestra.run.smithi069.stdout: "num_ops": 0 2023-11-05T08:33:09.862 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:33:09.875 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-11-05T08:33:09.890 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:33:09.884+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:33:09.990 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:33:09.984+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:33:10.042 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:33:10.042 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2023-11-05T08:33:10.043 INFO:teuthology.orchestra.run.smithi069.stdout: "complaint_time": 30, 2023-11-05T08:33:10.043 INFO:teuthology.orchestra.run.smithi069.stdout: "num_blocked_ops": 0 2023-11-05T08:33:10.043 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:33:10.057 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-11-05T08:33:10.091 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:33:10.089+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:33:10.095 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:33:10.096 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-11-05T08:33:10.114 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":61,"stamp":"2023-11-05T08:33:09.267681+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":44065312,"num_objects":76,"num_object_clones":24,"num_object_copies":522,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":52,"num_objects_misplaced":11,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":4,"num_read":272,"num_read_kb":23613,"num_write":444,"num_write_kb":35820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1654784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":591,"ondisk_log_size":591,"up":197,"acting":197,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":197,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":514536,"kb_used_data":82100,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509434904,"statfs":{"total":1546188226560,"available":1545661341696,"internally_reserved":0,"allocated":84070400,"data_stored":99119533,"data_compressed":906936,"data_compressed_allocated":2043904,"data_compressed_original":4980736,"omap_allocated":25505,"internal_metadata":442473567},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3358720,"num_objects":6,"num_object_clones":5,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":-1,"num_read_kb":7304,"num_write":14,"num_write_kb":-800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-4,"num_bytes_recovered":-4866048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.036168"},"pg_stats":[{"pgid":"3.19","version":"39'22","reported_seq":104,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-05T08:33:06.235150+0000","last_change":"2023-11-05T08:33:05.235386+0000","last_active":"2023-11-05T08:33:06.235150+0000","last_peered":"2023-11-05T08:33:06.235150+0000","last_clean":"2023-11-05T08:33:06.235150+0000","last_became_active":"2023-11-05T08:32:59.211522+0000","last_became_peered":"2023-11-05T08:32:59.211522+0000","last_unstale":"2023-11-05T08:33:06.235150+0000","last_undegraded":"2023-11-05T08:33:06.235150+0000","last_fullsized":"2023-11-05T08:33:06.235150+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":5,"last_scrub":"39'22","last_scrub_stamp":"2023-11-05T08:33:01.870959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:01.870959+0000","objects_scrubbed":3,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:31.367486+0000","scrub_duration":0.0087276880000000008,"objects_trimmed":0,"snaptrim_duration":0.00096963500000000005,"stat_sum":{"num_bytes":2080768,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,11,9,13,7,1],"acting":[8,4,11,9,13,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"2"},{"start":"8","length":"1"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"}]},{"pgid":"3.18","version":"52'28","reported_seq":102,"reported_epoch":55,"state":"active+clean+remapped","last_fresh":"2023-11-05T08:33:08.263954+0000","last_change":"2023-11-05T08:33:07.923306+0000","last_active":"2023-11-05T08:33:08.263954+0000","last_peered":"2023-11-05T08:33:08.263954+0000","last_clean":"2023-11-05T08:33:08.263954+0000","last_became_active":"2023-11-05T08:32:59.217241+0000","last_became_peered":"2023-11-05T08:32:59.217241+0000","last_unstale":"2023-11-05T08:33:08.263954+0000","last_undegraded":"2023-11-05T08:33:08.263954+0000","last_fullsized":"2023-11-05T08:33:08.263954+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00186543,"stat_sum":{"num_bytes":2523136,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":40,"num_write_kb":2464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":458752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,2147483647,4,5,7,12],"acting":[13,6,2,4,5,7,12],"avail_no_missing":["13(0)","2(2)","4(3)","5(4)","6(1)","7(5)","12(6)"],"object_location_counts":[{"shards":"2(2),4(3),5(4),6(1),7(5),12(6),13(0)","objects":4}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"2"},{"start":"8","length":"1"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"}]},{"pgid":"3.17","version":"49'17","reported_seq":111,"reported_epoch":53,"state":"active+clean+remapped","last_fresh":"2023-11-05T08:33:06.438410+0000","last_change":"2023-11-05T08:33:05.262822+0000","last_active":"2023-11-05T08:33:06.438410+0000","last_peered":"2023-11-05T08:33:06.438410+0000","last_clean":"2023-11-05T08:33:06.438410+0000","last_became_active":"2023-11-05T08:33:02.240933+0000","last_became_peered":"2023-11-05T08:33:02.240933+0000","last_unstale":"2023-11-05T08:33:06.438410+0000","last_undegraded":"2023-11-05T08:33:06.438410+0000","last_fullsized":"2023-11-05T08:33:06.438410+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075446439999999997,"stat_sum":{"num_bytes":319488,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":1,"num_read":3,"num_read_kb":2,"num_write":5,"num_write_kb":312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2147483647,10,9,7,1,13],"acting":[5,6,10,9,7,1,13],"avail_no_missing":["5(0)","1(5)","6(1)","7(4)","9(3)","10(2)","13(6)"],"object_location_counts":[{"shards":"1(5),5(0),6(1),7(4),9(3),10(2),13(6)","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"2"},{"start":"8","length":"1"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"}]},{"pgid":"3.16","version":"51'19","reported_seq":86,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-05T08:33:07.688244+0000","last_change":"2023-11-05T08:33:07.689031+0000","last_active":"2023-11-05T08:33:07.688244+0000","last_peered":"2023-11-05T08:33:07.688244+0000","last_clean":"2023-11-05T08:33:07.688244+0000","last_became_active":"2023-11-05T08:32:59.210858+0000","last_became_peered":"2023-11-05T08:32:59.210858+0000","last_unstale":"2023-11-05T08:33:07.688244+0000","last_undegraded":"2023-11-05T08:33:07.688244+0000","last_fullsized":"2023-11-05T08:33:07.688244+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":5,"last_scrub":"46'13","last_scrub_stamp":"2023-11-05T08:33:00.659926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:00.659926+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:04.510724+0000","scrub_duration":0.0077151709999999998,"objects_trimmed":0,"snaptrim_duration":0.0029553159999999999,"stat_sum":{"num_bytes":1671168,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15,4,8,9,5,10],"acting":[11,15,4,8,9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"2"},{"start":"8","length":"1"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"}]},{"pgid":"3.15","version":"36'15","reported_seq":22,"reported_epoch":53,"state":"down+remapped","last_fresh":"2023-11-05T08:33:06.237939+0000","last_change":"2023-11-05T08:32:58.174545+0000","last_active":"2023-11-05T08:32:57.142227+0000","last_peered":"2023-11-05T08:32:49.086936+0000","last_clean":"2023-11-05T08:32:49.086936+0000","last_became_active":"2023-11-05T08:32:44.551939+0000","last_became_peered":"2023-11-05T08:32:44.551939+0000","last_unstale":"2023-11-05T08:33:06.237939+0000","last_undegraded":"2023-11-05T08:33:06.237939+0000","last_fullsized":"2023-11-05T08:33:06.237939+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1630208,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":795,"num_write":6,"num_write_kb":1592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,2147483647,13,14,2,15],"acting":[9,6,2147483647,13,2147483647,2,2147483647],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.14","version":"49'18","reported_seq":170,"reported_epoch":53,"state":"active+clean+remapped","last_fresh":"2023-11-05T08:33:06.235326+0000","last_change":"2023-11-05T08:33:05.236745+0000","last_active":"2023-11-05T08:33:06.235326+0000","last_peered":"2023-11-05T08:33:06.235326+0000","last_clean":"2023-11-05T08:33:06.235326+0000","last_became_active":"2023-11-05T08:33:02.227671+0000","last_became_peered":"2023-11-05T08:33:02.227671+0000","last_unstale":"2023-11-05T08:33:06.235326+0000","last_undegraded":"2023-11-05T08:33:06.235326+0000","last_fullsized":"2023-11-05T08:33:06.235326+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":5,"last_scrub":"49'18","last_scrub_stamp":"2023-11-05T08:33:04.812402+0000","last_deep_scrub":"49'18","last_deep_scrub_stamp":"2023-11-05T08:33:04.812402+0000","last_clean_scrub_stamp":"2023-11-05T08:33:04.812402+0000","objects_scrubbed":2,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:07.707149+0000","scrub_duration":0.0086643829999999995,"objects_trimmed":0,"snaptrim_duration":0.0020483540000000001,"stat_sum":{"num_bytes":2113536,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,9,15,1,13,8,4],"acting":[8,9,15,1,13,8,4],"avail_no_missing":["8(0)","1(3)","4(6)","8(5)","9(1)","13(4)","15(2)"],"object_location_counts":[{"shards":"1(3),4(6),8(0),8(5),9(1),13(4),15(2)","objects":2}],"blocked_by":[],"up_primary":9,"acting_primary":8,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"2"},{"start":"8","length":"1"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":37,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-05T08:33:05.269645+0000","last_change":"2023-11-05T08:32:31.942607+0000","last_active":"2023-11-05T08:33:05.269645+0000","last_peered":"2023-11-05T08:33:05.269645+0000","last_clean":"2023-11-05T08:33:05.269645+0000","last_became_active":"2023-11-05T08:32:31.941858+0000","last_became_peered":"2023-11-05T08:32:31.941858+0000","last_unstale":"2023-11-05T08:33:05.269645+0000","last_undegraded":"2023-11-05T08:33:05.269645+0000","last_fullsized":"2023-11-05T08:33:05.269645+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:48.710632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"35'9","reported_seq":82,"reported_epoch":55,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:33:08.264728+0000","last_change":"2023-11-05T08:32:54.153555+0000","last_active":"2023-11-05T08:33:08.264728+0000","last_peered":"2023-11-05T08:33:08.264728+0000","last_clean":"2023-11-05T08:32:48.102597+0000","last_became_active":"2023-11-05T08:32:54.153555+0000","last_became_peered":"2023-11-05T08:32:54.153555+0000","last_unstale":"2023-11-05T08:33:08.264728+0000","last_undegraded":"2023-11-05T08:32:54.121924+0000","last_fullsized":"2023-11-05T08:32:54.120000+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1089536,"num_objects":1,"num_object_clones":1,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":1064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,13,1,15,12,7,9],"acting":[2147483647,13,1,15,12,7,9],"avail_no_missing":["13(1)","1(2)","7(5)","9(6)","12(4)","15(3)"],"object_location_counts":[{"shards":"1(2),7(5),9(6),12(4),13(1),15(3)","objects":1}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":37,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-05T08:33:05.235101+0000","last_change":"2023-11-05T08:32:31.941613+0000","last_active":"2023-11-05T08:33:05.235101+0000","last_peered":"2023-11-05T08:33:05.235101+0000","last_clean":"2023-11-05T08:33:05.235101+0000","last_became_active":"2023-11-05T08:32:31.941207+0000","last_became_peered":"2023-11-05T08:32:31.941207+0000","last_unstale":"2023-11-05T08:33:05.235101+0000","last_undegraded":"2023-11-05T08:33:05.235101+0000","last_fullsized":"2023-11-05T08:33:05.235101+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:44.932076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"39'12","reported_seq":89,"reported_epoch":54,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:33:07.249576+0000","last_change":"2023-11-05T08:32:54.151639+0000","last_active":"2023-11-05T08:33:07.249576+0000","last_peered":"2023-11-05T08:33:07.249576+0000","last_clean":"2023-11-05T08:32:49.074431+0000","last_became_active":"2023-11-05T08:32:54.151639+0000","last_became_peered":"2023-11-05T08:32:54.151639+0000","last_unstale":"2023-11-05T08:33:07.249576+0000","last_undegraded":"2023-11-05T08:32:54.120694+0000","last_fullsized":"2023-11-05T08:32:54.119652+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030081180000000002,"stat_sum":{"num_bytes":319488,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,2147483647,8,5,11],"acting":[4,6,9,2147483647,8,5,11],"avail_no_missing":["4(0)","5(5)","6(1)","8(4)","9(2)","11(6)"],"object_location_counts":[{"shards":"4(0),5(5),6(1),8(4),9(2),11(6)","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":37,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-05T08:33:05.272406+0000","last_change":"2023-11-05T08:32:31.940904+0000","last_active":"2023-11-05T08:33:05.272406+0000","last_peered":"2023-11-05T08:33:05.272406+0000","last_clean":"2023-11-05T08:33:05.272406+0000","last_became_active":"2023-11-05T08:32:31.940312+0000","last_became_peered":"2023-11-05T08:32:31.940312+0000","last_unstale":"2023-11-05T08:33:05.272406+0000","last_undegraded":"2023-11-05T08:33:05.272406+0000","last_fullsized":"2023-11-05T08:33:05.272406+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:43.223063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"39'12","reported_seq":88,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:33:06.235468+0000","last_change":"2023-11-05T08:32:54.141915+0000","last_active":"2023-11-05T08:33:06.235468+0000","last_peered":"2023-11-05T08:33:06.235468+0000","last_clean":"2023-11-05T08:32:49.088117+0000","last_became_active":"2023-11-05T08:32:54.141915+0000","last_became_peered":"2023-11-05T08:32:54.141915+0000","last_unstale":"2023-11-05T08:33:06.235468+0000","last_undegraded":"2023-11-05T08:32:54.115420+0000","last_fullsized":"2023-11-05T08:32:54.114438+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.011061272,"stat_sum":{"num_bytes":1376256,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,4,1,2147483647,14,13],"acting":[8,2,4,1,2147483647,14,13],"avail_no_missing":["8(0)","1(3)","2(1)","4(2)","13(6)","14(5)"],"object_location_counts":[{"shards":"1(3),2(1),4(2),8(0),13(6),14(5)","objects":1}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":42,"reported_epoch":53,"state":"active+undersized","last_fresh":"2023-11-05T08:33:06.437417+0000","last_change":"2023-11-05T08:32:51.087583+0000","last_active":"2023-11-05T08:33:06.437417+0000","last_peered":"2023-11-05T08:33:06.437417+0000","last_clean":"2023-11-05T08:32:49.071396+0000","last_became_active":"2023-11-05T08:32:51.087583+0000","last_became_peered":"2023-11-05T08:32:51.087583+0000","last_unstale":"2023-11-05T08:33:06.437417+0000","last_undegraded":"2023-11-05T08:33:06.437417+0000","last_fullsized":"2023-11-05T08:32:51.085658+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:52.667515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"36'15","reported_seq":98,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:33:06.437306+0000","last_change":"2023-11-05T08:32:54.139002+0000","last_active":"2023-11-05T08:33:06.437306+0000","last_peered":"2023-11-05T08:33:06.437306+0000","last_clean":"2023-11-05T08:32:49.088591+0000","last_became_active":"2023-11-05T08:32:54.139002+0000","last_became_peered":"2023-11-05T08:32:54.139002+0000","last_unstale":"2023-11-05T08:33:06.437306+0000","last_undegraded":"2023-11-05T08:32:54.118261+0000","last_fullsized":"2023-11-05T08:32:54.117408+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.020413489999999999,"stat_sum":{"num_bytes":1630208,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":795,"num_write":6,"num_write_kb":1592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,4,13,7,2147483647,6],"acting":[15,8,4,13,7,2147483647,6],"avail_no_missing":["15(0)","4(2)","6(6)","7(4)","8(1)","13(3)"],"object_location_counts":[{"shards":"4(2),6(6),7(4),8(1),13(3),15(0)","objects":2}],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.3","version":"36'12","reported_seq":109,"reported_epoch":55,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:33:08.264104+0000","last_change":"2023-11-05T08:32:59.214416+0000","last_active":"2023-11-05T08:33:08.264104+0000","last_peered":"2023-11-05T08:33:08.264104+0000","last_clean":"2023-11-05T08:32:57.168145+0000","last_became_active":"2023-11-05T08:32:59.214416+0000","last_became_peered":"2023-11-05T08:32:59.214416+0000","last_unstale":"2023-11-05T08:33:08.264104+0000","last_undegraded":"2023-11-05T08:32:59.185849+0000","last_fullsized":"2023-11-05T08:32:59.184877+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0045341740000000002,"stat_sum":{"num_bytes":786432,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":4,"num_write_kb":768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,4,15,2,5,2147483647],"acting":[13,7,4,15,2,5,2147483647],"avail_no_missing":["13(0)","2(4)","4(2)","5(5)","7(1)","15(3)"],"object_location_counts":[{"shards":"2(4),4(2),5(5),7(1),13(0),15(3)","objects":2}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"2"},{"start":"6","length":"1"},{"start":"8","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":35,"reported_epoch":52,"state":"active+undersized","last_fresh":"2023-11-05T08:33:05.261979+0000","last_change":"2023-11-05T08:32:59.192112+0000","last_active":"2023-11-05T08:33:05.261979+0000","last_peered":"2023-11-05T08:33:05.261979+0000","last_clean":"2023-11-05T08:32:57.141279+0000","last_became_active":"2023-11-05T08:32:59.192112+0000","last_became_peered":"2023-11-05T08:32:59.192112+0000","last_unstale":"2023-11-05T08:33:05.261979+0000","last_undegraded":"2023-11-05T08:33:05.261979+0000","last_fullsized":"2023-11-05T08:32:59.189202+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:58.324718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12],"acting":[12],"avail_no_missing":["12"],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.0","version":"33'9","reported_seq":126,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-05T08:33:06.237704+0000","last_change":"2023-11-05T08:33:05.252295+0000","last_active":"2023-11-05T08:33:06.237704+0000","last_peered":"2023-11-05T08:33:06.237704+0000","last_clean":"2023-11-05T08:33:06.237704+0000","last_became_active":"2023-11-05T08:32:54.144790+0000","last_became_peered":"2023-11-05T08:32:54.144790+0000","last_unstale":"2023-11-05T08:33:06.237704+0000","last_undegraded":"2023-11-05T08:33:06.237704+0000","last_fullsized":"2023-11-05T08:33:06.237704+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'9","last_scrub_stamp":"2023-11-05T08:33:03.810524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:03.810524+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:08.393950+0000","scrub_duration":0.0066041449999999996,"objects_trimmed":0,"snaptrim_duration":0.001634824,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,15,1,5,6,4],"acting":[14,2,15,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"2"},{"start":"8","length":"1"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":39,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-05T08:33:07.921552+0000","last_change":"2023-11-05T08:32:31.941357+0000","last_active":"2023-11-05T08:33:07.921552+0000","last_peered":"2023-11-05T08:33:07.921552+0000","last_clean":"2023-11-05T08:33:07.921552+0000","last_became_active":"2023-11-05T08:32:31.940564+0000","last_became_peered":"2023-11-05T08:32:31.940564+0000","last_unstale":"2023-11-05T08:33:07.921552+0000","last_undegraded":"2023-11-05T08:33:07.921552+0000","last_fullsized":"2023-11-05T08:33:07.921552+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:58.168161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.1","version":"33'11","reported_seq":137,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-05T08:33:07.249723+0000","last_change":"2023-11-05T08:33:07.251024+0000","last_active":"2023-11-05T08:33:07.249723+0000","last_peered":"2023-11-05T08:33:07.249723+0000","last_clean":"2023-11-05T08:33:07.249723+0000","last_became_active":"2023-11-05T08:32:54.144895+0000","last_became_peered":"2023-11-05T08:32:54.144895+0000","last_unstale":"2023-11-05T08:33:07.249723+0000","last_undegraded":"2023-11-05T08:33:07.249723+0000","last_fullsized":"2023-11-05T08:33:07.249723+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'11","last_scrub_stamp":"2023-11-05T08:33:04.003557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:04.003557+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:32.738162+0000","scrub_duration":0.010761797,"objects_trimmed":0,"snaptrim_duration":0.0017874099999999999,"stat_sum":{"num_bytes":851968,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":289,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,5,10,2,13],"acting":[4,11,9,5,10,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"2"},{"start":"8","length":"1"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":35,"reported_epoch":52,"state":"active+undersized","last_fresh":"2023-11-05T08:33:05.269140+0000","last_change":"2023-11-05T08:32:59.200499+0000","last_active":"2023-11-05T08:33:05.269140+0000","last_peered":"2023-11-05T08:33:05.269140+0000","last_clean":"2023-11-05T08:32:57.169235+0000","last_became_active":"2023-11-05T08:32:59.200499+0000","last_became_peered":"2023-11-05T08:32:59.200499+0000","last_unstale":"2023-11-05T08:33:05.269140+0000","last_undegraded":"2023-11-05T08:33:05.269140+0000","last_fullsized":"2023-11-05T08:32:59.194018+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:35.798739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"55'16","reported_seq":110,"reported_epoch":55,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:33:08.264264+0000","last_change":"2023-11-05T08:32:59.207989+0000","last_active":"2023-11-05T08:33:08.264264+0000","last_peered":"2023-11-05T08:33:08.264264+0000","last_clean":"2023-11-05T08:32:57.163393+0000","last_became_active":"2023-11-05T08:32:59.207989+0000","last_became_peered":"2023-11-05T08:32:59.207989+0000","last_unstale":"2023-11-05T08:33:08.264264+0000","last_undegraded":"2023-11-05T08:32:59.181953+0000","last_fullsized":"2023-11-05T08:32:59.181602+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018755122999999999,"stat_sum":{"num_bytes":475136,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,2147483647,6,15,1,11],"acting":[13,2,2147483647,6,15,1,11],"avail_no_missing":["13(0)","1(5)","2(1)","6(3)","11(6)","15(4)"],"object_location_counts":[{"shards":"1(5),2(1),6(3),11(6),13(0),15(4)","objects":2}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"2"},{"start":"6","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":37,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-05T08:33:05.254755+0000","last_change":"2023-11-05T08:32:31.940561+0000","last_active":"2023-11-05T08:33:05.254755+0000","last_peered":"2023-11-05T08:33:05.254755+0000","last_clean":"2023-11-05T08:33:05.254755+0000","last_became_active":"2023-11-05T08:32:31.939705+0000","last_became_peered":"2023-11-05T08:32:31.939705+0000","last_unstale":"2023-11-05T08:33:05.254755+0000","last_undegraded":"2023-11-05T08:33:05.254755+0000","last_fullsized":"2023-11-05T08:33:05.254755+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:54.088838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"25'157","reported_seq":238,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-05T08:33:07.922348+0000","last_change":"2023-11-05T08:32:28.761556+0000","last_active":"2023-11-05T08:33:07.922348+0000","last_peered":"2023-11-05T08:33:07.922348+0000","last_clean":"2023-11-05T08:33:07.922348+0000","last_became_active":"2023-11-05T08:32:28.758834+0000","last_became_peered":"2023-11-05T08:32:28.758834+0000","last_unstale":"2023-11-05T08:33:07.922348+0000","last_undegraded":"2023-11-05T08:33:07.922348+0000","last_fullsized":"2023-11-05T08:33:07.922348+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:27.693099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:27.693099+0000","last_clean_scrub_stamp":"2023-11-05T08:32:27.693099+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:40.584365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.8","version":"52'17","reported_seq":96,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:33:06.436889+0000","last_change":"2023-11-05T08:32:54.151946+0000","last_active":"2023-11-05T08:33:06.436889+0000","last_peered":"2023-11-05T08:33:06.436889+0000","last_clean":"2023-11-05T08:32:49.090022+0000","last_became_active":"2023-11-05T08:32:54.151946+0000","last_became_peered":"2023-11-05T08:32:54.151946+0000","last_unstale":"2023-11-05T08:33:06.436889+0000","last_undegraded":"2023-11-05T08:32:54.119646+0000","last_fullsized":"2023-11-05T08:32:54.118811+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.013731910999999999,"stat_sum":{"num_bytes":1949696,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":16,"num_write_kb":1904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2147483647,2,1,14,15,6],"acting":[5,2147483647,2,1,14,15,6],"avail_no_missing":["5(0)","1(3)","2(2)","6(6)","14(4)","15(5)"],"object_location_counts":[{"shards":"1(3),2(2),5(0),6(6),14(4),15(5)","objects":3}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.9","version":"52'26","reported_seq":128,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:33:06.237728+0000","last_change":"2023-11-05T08:32:59.206152+0000","last_active":"2023-11-05T08:33:06.237728+0000","last_peered":"2023-11-05T08:33:06.237728+0000","last_clean":"2023-11-05T08:32:49.072716+0000","last_became_active":"2023-11-05T08:32:59.206152+0000","last_became_peered":"2023-11-05T08:32:59.206152+0000","last_unstale":"2023-11-05T08:33:06.237728+0000","last_undegraded":"2023-11-05T08:32:59.188430+0000","last_fullsized":"2023-11-05T08:32:59.187299+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015323719999999999,"stat_sum":{"num_bytes":3407872,"num_objects":6,"num_object_clones":2,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":21,"num_read_kb":8311,"num_write":31,"num_write_kb":3328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,2147483647,2147483647,15,8,10],"acting":[14,1,2147483647,2147483647,15,8,10],"avail_no_missing":["14(0)","1(1)","8(5)","10(6)","15(4)"],"object_location_counts":[{"shards":"1(1),8(5),10(6),14(0),15(4)","objects":6}],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.a","version":"52'21","reported_seq":103,"reported_epoch":54,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:33:07.249030+0000","last_change":"2023-11-05T08:32:59.213422+0000","last_active":"2023-11-05T08:33:07.249030+0000","last_peered":"2023-11-05T08:33:07.249030+0000","last_clean":"2023-11-05T08:32:57.149380+0000","last_became_active":"2023-11-05T08:32:59.213422+0000","last_became_peered":"2023-11-05T08:32:59.213422+0000","last_unstale":"2023-11-05T08:33:07.249030+0000","last_undegraded":"2023-11-05T08:32:59.183619+0000","last_fullsized":"2023-11-05T08:32:59.182610+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:08.742529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033597206999999997,"stat_sum":{"num_bytes":2392064,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":14,"num_read_kb":7541,"num_write":28,"num_write_kb":2336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,7,1,9,14,6,11],"acting":[2147483647,7,1,9,14,6,11],"avail_no_missing":["7(1)","1(2)","6(5)","9(3)","11(6)","14(4)"],"object_location_counts":[{"shards":"1(2),6(5),7(1),9(3),11(6),14(4)","objects":4}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"2"},{"start":"6","length":"1"}]},{"pgid":"3.b","version":"44'16","reported_seq":93,"reported_epoch":54,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:33:07.251276+0000","last_change":"2023-11-05T08:32:51.095575+0000","last_active":"2023-11-05T08:33:07.251276+0000","last_peered":"2023-11-05T08:33:07.251276+0000","last_clean":"2023-11-05T08:32:49.072970+0000","last_became_active":"2023-11-05T08:32:51.095575+0000","last_became_peered":"2023-11-05T08:32:51.095575+0000","last_unstale":"2023-11-05T08:33:07.251276+0000","last_undegraded":"2023-11-05T08:32:51.085902+0000","last_fullsized":"2023-11-05T08:32:51.084936+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:48.126395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021279290000000002,"stat_sum":{"num_bytes":2326528,"num_objects":5,"num_object_clones":1,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":1782,"num_write":14,"num_write_kb":2592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2147483647,1,12,6,7,9],"acting":[4,2147483647,1,12,6,7,9],"avail_no_missing":["4(0)","1(2)","6(4)","7(5)","9(6)","12(3)"],"object_location_counts":[{"shards":"1(2),4(0),6(4),7(5),9(6),12(3)","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.c","version":"33'2","reported_seq":73,"reported_epoch":54,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:33:07.685945+0000","last_change":"2023-11-05T08:32:51.099852+0000","last_active":"2023-11-05T08:33:07.685945+0000","last_peered":"2023-11-05T08:33:07.685945+0000","last_clean":"2023-11-05T08:32:49.087645+0000","last_became_active":"2023-11-05T08:32:51.099852+0000","last_became_peered":"2023-11-05T08:32:51.099852+0000","last_unstale":"2023-11-05T08:33:07.685945+0000","last_undegraded":"2023-11-05T08:32:51.089699+0000","last_fullsized":"2023-11-05T08:32:51.089041+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:46.112855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015630430000000001,"stat_sum":{"num_bytes":311296,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,2147483647,10,6,12,5],"acting":[11,8,2147483647,10,6,12,5],"avail_no_missing":["11(0)","5(6)","6(4)","8(1)","10(3)","12(5)"],"object_location_counts":[{"shards":"5(6),6(4),8(1),10(3),11(0),12(5)","objects":1}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.d","version":"52'16","reported_seq":92,"reported_epoch":54,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:33:07.249545+0000","last_change":"2023-11-05T08:32:59.204042+0000","last_active":"2023-11-05T08:33:07.249545+0000","last_peered":"2023-11-05T08:33:07.249545+0000","last_clean":"2023-11-05T08:32:49.088145+0000","last_became_active":"2023-11-05T08:32:59.204042+0000","last_became_peered":"2023-11-05T08:32:59.204042+0000","last_unstale":"2023-11-05T08:33:07.249545+0000","last_undegraded":"2023-11-05T08:32:59.184163+0000","last_fullsized":"2023-11-05T08:32:59.183315+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:06.536466+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.024583713,"stat_sum":{"num_bytes":1556480,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":5,"num_read_kb":3,"num_write":10,"num_write_kb":1520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2147483647,14,10,2147483647,8,4],"acting":[7,2147483647,14,10,2147483647,8,4],"avail_no_missing":["7(0)","4(6)","8(5)","10(3)","14(2)"],"object_location_counts":[{"shards":"4(6),7(0),8(5),10(3),14(2)","objects":4}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.e","version":"49'43","reported_seq":139,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-05T08:33:06.439346+0000","last_change":"2023-11-05T08:33:05.270374+0000","last_active":"2023-11-05T08:33:06.439346+0000","last_peered":"2023-11-05T08:33:06.439346+0000","last_clean":"2023-11-05T08:33:06.439346+0000","last_became_active":"2023-11-05T08:32:44.949185+0000","last_became_peered":"2023-11-05T08:32:44.949185+0000","last_unstale":"2023-11-05T08:33:06.439346+0000","last_undegraded":"2023-11-05T08:33:06.439346+0000","last_fullsized":"2023-11-05T08:33:06.439346+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:07.719221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018942200000000001,"stat_sum":{"num_bytes":4947968,"num_objects":8,"num_object_clones":2,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":19,"num_read_kb":1642,"num_write":23,"num_write_kb":4832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1,11,8,6,12],"acting":[2,9,1,11,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"2"},{"start":"8","length":"1"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"}]},{"pgid":"3.f","version":"48'29","reported_seq":104,"reported_epoch":54,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:33:07.686267+0000","last_change":"2023-11-05T08:32:51.104425+0000","last_active":"2023-11-05T08:33:07.686267+0000","last_peered":"2023-11-05T08:33:07.686267+0000","last_clean":"2023-11-05T08:32:49.098585+0000","last_became_active":"2023-11-05T08:32:51.104425+0000","last_became_peered":"2023-11-05T08:32:51.104425+0000","last_unstale":"2023-11-05T08:33:07.686267+0000","last_undegraded":"2023-11-05T08:32:51.091474+0000","last_fullsized":"2023-11-05T08:32:51.089784+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:48.366729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012093372,"stat_sum":{"num_bytes":5767168,"num_objects":9,"num_object_clones":5,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":16,"num_read_kb":1987,"num_write":40,"num_write_kb":5632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,14,13,8,10,2147483647],"acting":[11,2,14,13,8,10,2147483647],"avail_no_missing":["11(0)","2(1)","8(4)","10(5)","13(3)","14(2)"],"object_location_counts":[{"shards":"2(1),8(4),10(5),11(0),13(3),14(2)","objects":9}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.10","version":"33'9","reported_seq":76,"reported_epoch":55,"state":"active+clean+remapped","last_fresh":"2023-11-05T08:33:08.263810+0000","last_change":"2023-11-05T08:33:07.923811+0000","last_active":"2023-11-05T08:33:08.263810+0000","last_peered":"2023-11-05T08:33:08.263810+0000","last_clean":"2023-11-05T08:33:08.263810+0000","last_became_active":"2023-11-05T08:32:59.210399+0000","last_became_peered":"2023-11-05T08:32:59.210399+0000","last_unstale":"2023-11-05T08:33:08.263810+0000","last_undegraded":"2023-11-05T08:33:08.263810+0000","last_fullsized":"2023-11-05T08:33:08.263810+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":5,"last_scrub":"33'9","last_scrub_stamp":"2023-11-05T08:33:00.792248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:00.792248+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:24.881761+0000","scrub_duration":0.0073349119999999999,"objects_trimmed":0,"snaptrim_duration":0.0023893949999999999,"stat_sum":{"num_bytes":1605632,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":337,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,7,4,8,2147483647,11],"acting":[13,6,7,4,8,6,11],"avail_no_missing":["13(0)","4(3)","6(1)","6(5)","7(2)","8(4)","11(6)"],"object_location_counts":[{"shards":"4(3),6(1),6(5),7(2),8(4),11(6),13(0)","objects":3}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"2"},{"start":"8","length":"1"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"}]},{"pgid":"3.11","version":"49'19","reported_seq":143,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-05T08:33:06.438301+0000","last_change":"2023-11-05T08:33:05.239164+0000","last_active":"2023-11-05T08:33:06.438301+0000","last_peered":"2023-11-05T08:33:06.438301+0000","last_clean":"2023-11-05T08:33:06.438301+0000","last_became_active":"2023-11-05T08:32:59.214193+0000","last_became_peered":"2023-11-05T08:32:59.214193+0000","last_unstale":"2023-11-05T08:33:06.438301+0000","last_undegraded":"2023-11-05T08:33:06.438301+0000","last_fullsized":"2023-11-05T08:33:06.438301+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":5,"last_scrub":"46'17","last_scrub_stamp":"2023-11-05T08:33:00.070510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:00.070510+0000","objects_scrubbed":3,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:20.151011+0000","scrub_duration":0.0081678800000000006,"objects_trimmed":0,"snaptrim_duration":0.001375437,"stat_sum":{"num_bytes":1998848,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,15,9,5,6,1],"acting":[10,11,15,9,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"2"},{"start":"8","length":"1"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"}]},{"pgid":"3.12","version":"37'7","reported_seq":68,"reported_epoch":54,"state":"active+clean+remapped","last_fresh":"2023-11-05T08:33:07.248782+0000","last_change":"2023-11-05T08:33:07.249754+0000","last_active":"2023-11-05T08:33:07.248782+0000","last_peered":"2023-11-05T08:33:07.248782+0000","last_clean":"2023-11-05T08:33:07.248782+0000","last_became_active":"2023-11-05T08:33:00.206236+0000","last_became_peered":"2023-11-05T08:33:00.206236+0000","last_unstale":"2023-11-05T08:33:07.248782+0000","last_undegraded":"2023-11-05T08:33:07.248782+0000","last_fullsized":"2023-11-05T08:33:07.248782+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":5,"last_scrub":"37'7","last_scrub_stamp":"2023-11-05T08:33:02.650387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:02.650387+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:22.426134+0000","scrub_duration":0.0067664470000000001,"objects_trimmed":0,"snaptrim_duration":0.0018859669999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8,6,4,2147483647,15],"acting":[9,7,8,6,4,1,15],"avail_no_missing":["9(0)","1(5)","4(4)","6(3)","7(1)","8(2)","15(6)"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"2"},{"start":"8","length":"1"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"}]},{"pgid":"3.13","version":"41'14","reported_seq":137,"reported_epoch":54,"state":"active+clean+remapped","last_fresh":"2023-11-05T08:33:07.687914+0000","last_change":"2023-11-05T08:33:07.689341+0000","last_active":"2023-11-05T08:33:07.687914+0000","last_peered":"2023-11-05T08:33:07.687914+0000","last_clean":"2023-11-05T08:33:07.687914+0000","last_became_active":"2023-11-05T08:32:59.212969+0000","last_became_peered":"2023-11-05T08:32:59.212969+0000","last_unstale":"2023-11-05T08:33:07.687914+0000","last_undegraded":"2023-11-05T08:33:07.687914+0000","last_fullsized":"2023-11-05T08:33:07.687914+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'14","last_scrub_stamp":"2023-11-05T08:33:02.644028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:02.644028+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:20.082073+0000","scrub_duration":0.007386386,"objects_trimmed":0,"snaptrim_duration":0.0034524439999999998,"stat_sum":{"num_bytes":344064,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,9,13,2147483647,4,10],"acting":[11,7,9,13,2,4,10],"avail_no_missing":["11(0)","2(4)","4(5)","7(1)","9(2)","10(6)","13(3)"],"object_location_counts":[{"shards":"2(4),4(5),7(1),9(2),10(6),11(0),13(3)","objects":1}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"2"},{"start":"8","length":"1"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":43474944,"num_objects":74,"num_object_clones":24,"num_object_copies":518,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":52,"num_objects_misplaced":11,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":4,"num_read":136,"num_read_kb":23495,"num_write":241,"num_write_kb":32360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1654784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":75034624,"data_stored":92209152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":434,"ondisk_log_size":434,"up":182,"acting":182,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":13898,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":22,"seq":94489280523,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33032,"kb_used_data":6004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338808,"statfs":{"total":96636764160,"available":96602939392,"internally_reserved":0,"allocated":6148096,"data_stored":7779006,"data_compressed":66225,"data_compressed_allocated":163840,"data_compressed_original":774144,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,4,5,6,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79200000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.887}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0669999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94799999999999995}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3280000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.125}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.284}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4279999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2569999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2370000000000001}]}]},{"osd":15,"up_from":21,"seq":90194313227,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31732,"kb_used_data":4700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340108,"statfs":{"total":96636764160,"available":96604270592,"internally_reserved":0,"allocated":4812800,"data_stored":5455444,"data_compressed":59276,"data_compressed_allocated":131072,"data_compressed_original":262144,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,4,5,6,7,8,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3759999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.131}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96399999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1559999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9460000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4769999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.996}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.28}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9019999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5349999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.643}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.899}]}]},{"osd":14,"up_from":22,"seq":94489280522,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31300,"kb_used_data":4272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340540,"statfs":{"total":96636764160,"available":96604712960,"internally_reserved":0,"allocated":4374528,"data_stored":5951646,"data_compressed":59276,"data_compressed_allocated":131072,"data_compressed_original":262144,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,4,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.018}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.329}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2829999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2330000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2430000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.393}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7090000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.472}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6439999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8029999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313223,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":4392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":4497408,"data_stored":4339168,"data_compressed":21230,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,9,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]}]},{"osd":3,"up_from":21,"seq":90194313225,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30380,"kb_used_data":3356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341460,"statfs":{"total":96636764160,"available":96605655040,"internally_reserved":0,"allocated":3436544,"data_stored":4001321,"data_compressed":40011,"data_compressed_allocated":90112,"data_compressed_original":180224,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,4,5,6,7,9,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.333}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0740000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1479999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.361}]}]},{"osd":6,"up_from":21,"seq":90194313226,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33640,"kb_used_data":6612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338200,"statfs":{"total":96636764160,"available":96602316800,"internally_reserved":0,"allocated":6770688,"data_stored":7761492,"data_compressed":59276,"data_compressed_allocated":131072,"data_compressed_original":262144,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83399999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.875}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0029999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3999999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81799999999999995}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0629999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.097}]}]},{"osd":1,"up_from":22,"seq":94489280522,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33252,"kb_used_data":6224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338588,"statfs":{"total":96636764160,"available":96602714112,"internally_reserved":0,"allocated":6373376,"data_stored":7254174,"data_compressed":59276,"data_compressed_allocated":131072,"data_compressed_original":262144,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[2,4,5,6,7,8,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.873}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77500000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95099999999999996}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82299999999999995}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97499999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4790000000000001}]}]},{"osd":2,"up_from":22,"seq":94489280522,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":5152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":5275648,"data_stored":6849214,"data_compressed":66225,"data_compressed_allocated":163840,"data_compressed_original":774144,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.236}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.127}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0449999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81799999999999995}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.075}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.024}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2190000000000001}]}]},{"osd":4,"up_from":22,"seq":94489280522,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32536,"kb_used_data":5508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339304,"statfs":{"total":96636764160,"available":96603447296,"internally_reserved":0,"allocated":5640192,"data_stored":6442580,"data_compressed":59276,"data_compressed_allocated":131072,"data_compressed_original":262144,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,5,6,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88800000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91500000000000004}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81599999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79000000000000004}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83899999999999997}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89800000000000002}]}]},{"osd":5,"up_from":22,"seq":94489280522,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30388,"kb_used_data":3360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341452,"statfs":{"total":96636764160,"available":96605646848,"internally_reserved":0,"allocated":3440640,"data_stored":3358878,"data_compressed":59276,"data_compressed_allocated":131072,"data_compressed_original":262144,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,4,6,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79200000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.474}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.175}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95899999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3839999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5760000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99399999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2589999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4990000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4339999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.726}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.639}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.109}]}]},{"osd":7,"up_from":22,"seq":94489280523,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32148,"kb_used_data":5120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339692,"statfs":{"total":96636764160,"available":96603844608,"internally_reserved":0,"allocated":5242880,"data_stored":6221982,"data_compressed":59276,"data_compressed_allocated":131072,"data_compressed_original":262144,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,4,5,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1930000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0569999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93300000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1759999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88100000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77500000000000002}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.137}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96899999999999997}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0900000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]}]},{"osd":8,"up_from":22,"seq":94489280522,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34828,"kb_used_data":7800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337012,"statfs":{"total":96636764160,"available":96601100288,"internally_reserved":0,"allocated":7987200,"data_stored":10154142,"data_compressed":59276,"data_compressed_allocated":131072,"data_compressed_original":262144,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,4,5,6,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94999999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.873}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.536}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0629999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.089}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]}]},{"osd":9,"up_from":21,"seq":90194313226,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32392,"kb_used_data":5372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339448,"statfs":{"total":96636764160,"available":96603594752,"internally_reserved":0,"allocated":5500928,"data_stored":5958278,"data_compressed":61209,"data_compressed_allocated":135168,"data_compressed_original":270336,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,4,6,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.143}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.337}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1060000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2669999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.282}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6499999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3129999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4490000000000001}]}]},{"osd":10,"up_from":22,"seq":94489280522,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31892,"kb_used_data":4864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339948,"statfs":{"total":96636764160,"available":96604106752,"internally_reserved":0,"allocated":4980736,"data_stored":6377630,"data_compressed":59276,"data_compressed_allocated":131072,"data_compressed_original":262144,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,4,5,6,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.153}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1990000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.018}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85399999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.236}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2749999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3720000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3939999999999999}]}]},{"osd":11,"up_from":21,"seq":90194313227,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33196,"kb_used_data":6164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338644,"statfs":{"total":96636764160,"available":96602771456,"internally_reserved":0,"allocated":6311936,"data_stored":7540894,"data_compressed":59276,"data_compressed_allocated":131072,"data_compressed_original":262144,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,4,5,6,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91000000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3779999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7390000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2509999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0649999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.133}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0960000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.302}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.825}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.673}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.53}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6459999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8939999999999999}]}]},{"osd":12,"up_from":22,"seq":94489280522,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30228,"kb_used_data":3200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341612,"statfs":{"total":96636764160,"available":96605810688,"internally_reserved":0,"allocated":3276800,"data_stored":3673684,"data_compressed":59276,"data_compressed_allocated":131072,"data_compressed_original":262144,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,4,5,6,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0609999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.294}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.21}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5289999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.476}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.506}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.827}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.714}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6949,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6949,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4165632,"data_stored":4165632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5791744,"data_stored":6868992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4640768,"data_stored":5935104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2977792,"data_stored":3723264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5062656,"data_stored":6057984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2859008,"data_stored":2973696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":6193152,"data_stored":7376896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4661248,"data_stored":5836800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":7405568,"data_stored":9768960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4907008,"data_stored":5562368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4399104,"data_stored":5992448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":5730304,"data_stored":7155712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":2699264,"data_stored":3289088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":5513216,"data_stored":6864896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":3792896,"data_stored":5566464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":4235264,"data_stored":5070848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-05T08:33:10.185 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:33:10.186 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 20, 2023-11-05T08:33:10.186 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 600, 2023-11-05T08:33:10.186 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [ 2023-11-05T08:33:10.186 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.186 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.5s6 35/30 ECSubWrite(tid=20, reqid=client.4691.0:221, at_version=35'11, trim_to=0'0, roll_forward_to=33'9))", 2023-11-05T08:33:10.187 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:48.086545+0000", 2023-11-05T08:33:10.187 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 22.096326512000001, 2023-11-05T08:33:10.187 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.006420767, 2023-11-05T08:33:10.187 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:10.187 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:10.188 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:10.188 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.188 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:10.188 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.086545+0000", 2023-11-05T08:33:10.188 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.188 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.189 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.189 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:10.189 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.086545+0000", 2023-11-05T08:33:10.189 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.189 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.189 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.190 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:10.190 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.086545+0000", 2023-11-05T08:33:10.190 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.2300000000000002e-07 2023-11-05T08:33:10.190 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.190 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.191 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:10.191 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.086937+0000", 2023-11-05T08:33:10.191 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000391522 2023-11-05T08:33:10.191 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.191 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.191 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:10.192 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.086963+0000", 2023-11-05T08:33:10.192 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6114000000000001e-05 2023-11-05T08:33:10.192 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.192 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.192 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:10.192 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.086981+0000", 2023-11-05T08:33:10.193 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8289000000000001e-05 2023-11-05T08:33:10.193 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.193 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.193 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:10.193 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.087151+0000", 2023-11-05T08:33:10.194 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000169603 2023-11-05T08:33:10.194 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.194 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.194 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:10.194 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.087176+0000", 2023-11-05T08:33:10.194 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5488000000000001e-05 2023-11-05T08:33:10.195 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.195 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.195 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:10.195 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.092911+0000", 2023-11-05T08:33:10.195 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0057347580000000004 2023-11-05T08:33:10.195 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.196 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.196 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:10.196 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.092966+0000", 2023-11-05T08:33:10.196 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.4570000000000001e-05 2023-11-05T08:33:10.196 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.197 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:10.197 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.197 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.197 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.197 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.5s6 36/30 ECSubWrite(tid=25, reqid=osd.15.0:25, at_version=36'15, trim_to=0'0, roll_forward_to=35'13))", 2023-11-05T08:33:10.197 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.072943+0000", 2023-11-05T08:33:10.198 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 21.109928239999999, 2023-11-05T08:33:10.198 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0042426479999999999, 2023-11-05T08:33:10.198 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:10.198 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:10.199 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:10.199 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.199 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:10.199 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.072943+0000", 2023-11-05T08:33:10.199 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.199 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.200 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.200 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:10.200 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.072943+0000", 2023-11-05T08:33:10.200 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.200 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.200 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.201 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:10.201 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.072944+0000", 2023-11-05T08:33:10.201 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0669999999999999e-06 2023-11-05T08:33:10.201 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.201 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.202 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:10.202 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.073007+0000", 2023-11-05T08:33:10.202 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.2695999999999997e-05 2023-11-05T08:33:10.202 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.202 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.202 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:10.203 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.073043+0000", 2023-11-05T08:33:10.203 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.5595000000000001e-05 2023-11-05T08:33:10.203 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.203 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.203 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:10.203 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.073068+0000", 2023-11-05T08:33:10.204 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4888e-05 2023-11-05T08:33:10.204 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.204 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.204 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:10.204 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.073497+0000", 2023-11-05T08:33:10.205 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000429147 2023-11-05T08:33:10.205 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.205 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.205 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:10.205 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.073522+0000", 2023-11-05T08:33:10.205 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5791000000000001e-05 2023-11-05T08:33:10.206 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.206 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.206 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:10.206 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.077026+0000", 2023-11-05T08:33:10.206 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0035039440000000002 2023-11-05T08:33:10.207 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.207 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.207 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:10.207 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.077186+0000", 2023-11-05T08:33:10.207 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00015951999999999999 2023-11-05T08:33:10.208 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.208 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:10.208 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.208 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.208 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.209 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.8s6 36/30 ECSubWrite(tid=23, reqid=osd.5.0:23, at_version=36'15, trim_to=0'0, roll_forward_to=34'13))", 2023-11-05T08:33:10.209 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.079553+0000", 2023-11-05T08:33:10.209 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 21.103318114, 2023-11-05T08:33:10.209 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0063101340000000002, 2023-11-05T08:33:10.209 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:10.209 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:10.210 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:10.210 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.210 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:10.210 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079553+0000", 2023-11-05T08:33:10.210 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.211 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.211 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.211 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:10.211 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079553+0000", 2023-11-05T08:33:10.211 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.211 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.212 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.212 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:10.212 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079554+0000", 2023-11-05T08:33:10.212 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.1299999999999998e-07 2023-11-05T08:33:10.212 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.213 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.213 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:10.213 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079604+0000", 2023-11-05T08:33:10.213 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.0157000000000001e-05 2023-11-05T08:33:10.213 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.213 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.214 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:10.214 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079671+0000", 2023-11-05T08:33:10.214 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.6919000000000001e-05 2023-11-05T08:33:10.214 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.214 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.215 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:10.215 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079696+0000", 2023-11-05T08:33:10.215 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4192000000000001e-05 2023-11-05T08:33:10.215 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.215 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.215 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:10.216 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079889+0000", 2023-11-05T08:33:10.216 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00019337400000000001 2023-11-05T08:33:10.216 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.216 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.216 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:10.217 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079920+0000", 2023-11-05T08:33:10.217 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0682000000000002e-05 2023-11-05T08:33:10.217 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.217 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.217 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:10.217 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.085773+0000", 2023-11-05T08:33:10.218 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0058536889999999996 2023-11-05T08:33:10.218 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.218 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.218 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:10.218 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.085864+0000", 2023-11-05T08:33:10.218 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.0208000000000001e-05 2023-11-05T08:33:10.219 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.219 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:10.219 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.219 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.219 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.219 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.8s6 36/30 ECSubWrite(tid=24, reqid=osd.5.0:23, at_version=0'0, trim_to=0'0, roll_forward_to=36'15))", 2023-11-05T08:33:10.220 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.089082+0000", 2023-11-05T08:33:10.220 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 21.093789476000001, 2023-11-05T08:33:10.220 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0043041779999999997, 2023-11-05T08:33:10.220 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:10.220 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:10.220 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:10.221 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.221 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:10.221 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089082+0000", 2023-11-05T08:33:10.221 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.221 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.221 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.222 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:10.222 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089082+0000", 2023-11-05T08:33:10.222 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.222 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.222 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.222 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:10.223 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089084+0000", 2023-11-05T08:33:10.223 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2840000000000001e-06 2023-11-05T08:33:10.223 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.223 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.223 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:10.223 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089114+0000", 2023-11-05T08:33:10.224 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.9490999999999999e-05 2023-11-05T08:33:10.224 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.224 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.224 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:10.224 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089157+0000", 2023-11-05T08:33:10.224 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.2818000000000002e-05 2023-11-05T08:33:10.225 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.225 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.225 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:10.225 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089180+0000", 2023-11-05T08:33:10.225 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3311000000000001e-05 2023-11-05T08:33:10.225 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.226 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.226 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:10.226 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089358+0000", 2023-11-05T08:33:10.226 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00017781900000000001 2023-11-05T08:33:10.226 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.227 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.227 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:10.227 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089381+0000", 2023-11-05T08:33:10.227 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2853e-05 2023-11-05T08:33:10.227 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.228 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.228 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:10.228 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.093288+0000", 2023-11-05T08:33:10.228 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0039075949999999998 2023-11-05T08:33:10.228 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.229 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.229 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:10.229 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.093386+0000", 2023-11-05T08:33:10.229 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.8006999999999999e-05 2023-11-05T08:33:10.229 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.229 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:10.230 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.230 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.230 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.230 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.es5 40/30 ECSubWrite(tid=35, reqid=client.4691.0:274, at_version=40'23, trim_to=0'0, roll_forward_to=36'21))", 2023-11-05T08:33:10.230 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:53.138991+0000", 2023-11-05T08:33:10.231 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 17.04388101, 2023-11-05T08:33:10.231 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0046256200000000004, 2023-11-05T08:33:10.231 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:10.231 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:10.231 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:10.232 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.232 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:10.232 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.138991+0000", 2023-11-05T08:33:10.232 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.232 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.232 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.233 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:10.233 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.138991+0000", 2023-11-05T08:33:10.233 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.233 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.233 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.233 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:10.234 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.138991+0000", 2023-11-05T08:33:10.234 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.0800000000000004e-07 2023-11-05T08:33:10.234 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.234 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.234 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:10.234 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.139123+0000", 2023-11-05T08:33:10.235 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00013216399999999999 2023-11-05T08:33:10.235 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.235 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.235 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:10.235 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.139158+0000", 2023-11-05T08:33:10.236 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.4743999999999998e-05 2023-11-05T08:33:10.236 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.236 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.236 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:10.236 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.139185+0000", 2023-11-05T08:33:10.236 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6653e-05 2023-11-05T08:33:10.237 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.237 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.237 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:10.237 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.139270+0000", 2023-11-05T08:33:10.237 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.4992000000000002e-05 2023-11-05T08:33:10.238 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.238 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.238 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:10.238 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.139281+0000", 2023-11-05T08:33:10.238 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1282e-05 2023-11-05T08:33:10.239 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.239 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.239 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:10.239 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143543+0000", 2023-11-05T08:33:10.239 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0042614330000000002 2023-11-05T08:33:10.239 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.240 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.240 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:10.240 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143616+0000", 2023-11-05T08:33:10.240 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.3644000000000002e-05 2023-11-05T08:33:10.240 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.241 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:10.241 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.241 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.241 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.241 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.es5 41/30 ECSubWrite(tid=40, reqid=osd.2.0:40, at_version=41'27, trim_to=0'0, roll_forward_to=40'25))", 2023-11-05T08:33:10.242 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.123345+0000", 2023-11-05T08:33:10.242 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 16.059526111, 2023-11-05T08:33:10.242 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0086456399999999996, 2023-11-05T08:33:10.242 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:10.242 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:10.243 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:10.243 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.243 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:10.243 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.123345+0000", 2023-11-05T08:33:10.243 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.243 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.244 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.244 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:10.244 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.123345+0000", 2023-11-05T08:33:10.244 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.244 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.245 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.245 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:10.245 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.123346+0000", 2023-11-05T08:33:10.245 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.3000000000000001e-07 2023-11-05T08:33:10.245 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.246 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.246 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:10.246 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.123401+0000", 2023-11-05T08:33:10.246 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.4583999999999998e-05 2023-11-05T08:33:10.246 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.246 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.247 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:10.247 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.123432+0000", 2023-11-05T08:33:10.247 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.1488000000000001e-05 2023-11-05T08:33:10.247 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.247 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.248 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:10.248 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.123458+0000", 2023-11-05T08:33:10.248 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5925999999999999e-05 2023-11-05T08:33:10.248 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.248 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.248 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:10.249 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.123705+0000", 2023-11-05T08:33:10.249 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00024677599999999998 2023-11-05T08:33:10.249 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.249 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.249 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:10.250 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.123734+0000", 2023-11-05T08:33:10.250 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.9002000000000001e-05 2023-11-05T08:33:10.250 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.250 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.250 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:10.250 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131911+0000", 2023-11-05T08:33:10.251 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0081774080000000006 2023-11-05T08:33:10.251 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.251 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.251 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:10.251 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131991+0000", 2023-11-05T08:33:10.252 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.9925999999999993e-05 2023-11-05T08:33:10.252 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.252 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:10.252 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.252 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.253 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.253 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.as5 41/33 ECSubWrite(tid=35, reqid=osd.7.0:35, at_version=41'12, trim_to=0'0, roll_forward_to=40'10))", 2023-11-05T08:33:10.253 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.129002+0000", 2023-11-05T08:33:10.253 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 16.053869564999999, 2023-11-05T08:33:10.253 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.010000177000000001, 2023-11-05T08:33:10.253 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:10.254 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:10.254 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:10.254 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.254 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:10.254 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.129002+0000", 2023-11-05T08:33:10.255 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.255 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.255 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.255 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:10.255 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.129002+0000", 2023-11-05T08:33:10.256 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.256 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.256 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.256 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:10.256 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.129003+0000", 2023-11-05T08:33:10.257 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.4000000000000001e-07 2023-11-05T08:33:10.257 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.257 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.257 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:10.257 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.129074+0000", 2023-11-05T08:33:10.258 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.1649999999999993e-05 2023-11-05T08:33:10.258 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.258 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.258 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:10.258 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.129105+0000", 2023-11-05T08:33:10.258 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.1161999999999997e-05 2023-11-05T08:33:10.259 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.259 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.259 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:10.259 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.129132+0000", 2023-11-05T08:33:10.259 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6874999999999999e-05 2023-11-05T08:33:10.260 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.260 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.260 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:10.260 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.133281+0000", 2023-11-05T08:33:10.260 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0041491239999999997 2023-11-05T08:33:10.261 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.261 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.261 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:10.261 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.133306+0000", 2023-11-05T08:33:10.261 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4408000000000001e-05 2023-11-05T08:33:10.261 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.262 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.262 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:10.262 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.138905+0000", 2023-11-05T08:33:10.262 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0055986899999999999 2023-11-05T08:33:10.262 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.262 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.263 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:10.263 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.139002+0000", 2023-11-05T08:33:10.263 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.7627999999999994e-05 2023-11-05T08:33:10.263 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.263 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:10.264 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.264 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.264 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.264 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.as5 41/33 ECSubWrite(tid=36, reqid=osd.7.0:36, at_version=41'14, trim_to=0'0, roll_forward_to=40'10))", 2023-11-05T08:33:10.264 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.131389+0000", 2023-11-05T08:33:10.265 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 16.051482576000002, 2023-11-05T08:33:10.265 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0079481799999999991, 2023-11-05T08:33:10.265 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:10.265 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:10.265 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:10.265 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.266 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:10.266 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131389+0000", 2023-11-05T08:33:10.266 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.266 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.266 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.266 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:10.267 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131389+0000", 2023-11-05T08:33:10.267 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.267 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.267 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.267 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:10.268 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131391+0000", 2023-11-05T08:33:10.268 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.753e-06 2023-11-05T08:33:10.268 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.268 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.268 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:10.268 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131444+0000", 2023-11-05T08:33:10.269 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.3363999999999999e-05 2023-11-05T08:33:10.269 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.269 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.269 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:10.269 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131540+0000", 2023-11-05T08:33:10.269 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.6075000000000004e-05 2023-11-05T08:33:10.270 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.270 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.270 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:10.270 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131580+0000", 2023-11-05T08:33:10.270 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.9351000000000003e-05 2023-11-05T08:33:10.271 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.271 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.271 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:10.271 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.135074+0000", 2023-11-05T08:33:10.271 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.003494259 2023-11-05T08:33:10.272 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.272 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.272 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:10.272 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.135103+0000", 2023-11-05T08:33:10.272 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.9342000000000001e-05 2023-11-05T08:33:10.272 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.273 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.273 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:10.273 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.139247+0000", 2023-11-05T08:33:10.273 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0041437940000000001 2023-11-05T08:33:10.273 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.274 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.274 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:10.274 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.139337+0000", 2023-11-05T08:33:10.274 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.0241999999999994e-05 2023-11-05T08:33:10.274 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.274 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:10.275 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.275 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.275 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.275 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.es5 41/30 ECSubWrite(tid=41, reqid=osd.2.0:40, at_version=0'0, trim_to=0'0, roll_forward_to=41'27))", 2023-11-05T08:33:10.275 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.149656+0000", 2023-11-05T08:33:10.276 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 16.033215393999999, 2023-11-05T08:33:10.276 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0061863129999999997, 2023-11-05T08:33:10.276 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:10.276 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:10.276 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:10.276 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.277 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:10.277 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.149656+0000", 2023-11-05T08:33:10.277 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.277 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.277 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.278 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:10.278 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.149656+0000", 2023-11-05T08:33:10.278 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.278 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.278 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.279 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:10.279 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.149658+0000", 2023-11-05T08:33:10.279 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.86e-06 2023-11-05T08:33:10.279 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.279 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.279 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:10.280 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.149687+0000", 2023-11-05T08:33:10.280 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8824999999999999e-05 2023-11-05T08:33:10.280 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.280 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.280 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:10.281 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.149719+0000", 2023-11-05T08:33:10.281 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2404999999999999e-05 2023-11-05T08:33:10.281 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.281 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.281 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:10.281 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.149742+0000", 2023-11-05T08:33:10.282 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2779e-05 2023-11-05T08:33:10.282 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.282 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.282 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:10.282 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.149992+0000", 2023-11-05T08:33:10.282 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00024954300000000002 2023-11-05T08:33:10.283 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.283 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.283 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:10.283 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.150017+0000", 2023-11-05T08:33:10.283 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5698e-05 2023-11-05T08:33:10.283 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.284 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.284 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:10.284 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.155762+0000", 2023-11-05T08:33:10.284 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0057443559999999999 2023-11-05T08:33:10.284 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.284 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.285 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:10.285 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.155842+0000", 2023-11-05T08:33:10.285 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.0846999999999994e-05 2023-11-05T08:33:10.285 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.285 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:10.286 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.286 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.286 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.286 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.2s3 41/40 ECSubWrite(tid=188, reqid=osd.13.0:188, at_version=41'11, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:10.286 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.170574+0000", 2023-11-05T08:33:10.287 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 16.012297551, 2023-11-05T08:33:10.287 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.006673193, 2023-11-05T08:33:10.287 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:10.287 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:10.287 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:10.287 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.288 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:10.288 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.170574+0000", 2023-11-05T08:33:10.288 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.288 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.288 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.289 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:10.289 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.170574+0000", 2023-11-05T08:33:10.289 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.289 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.289 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.289 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:10.290 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.170575+0000", 2023-11-05T08:33:10.290 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3629999999999999e-06 2023-11-05T08:33:10.290 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.290 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.290 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:10.290 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.170619+0000", 2023-11-05T08:33:10.291 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.4057999999999997e-05 2023-11-05T08:33:10.291 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.291 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.291 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:10.291 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.170643+0000", 2023-11-05T08:33:10.292 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4070000000000002e-05 2023-11-05T08:33:10.292 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.292 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.292 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:10.292 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.170665+0000", 2023-11-05T08:33:10.293 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1216000000000002e-05 2023-11-05T08:33:10.293 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.293 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.293 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:10.293 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.170789+0000", 2023-11-05T08:33:10.293 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000124416 2023-11-05T08:33:10.294 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.294 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.294 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:10.294 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.170810+0000", 2023-11-05T08:33:10.294 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0735999999999999e-05 2023-11-05T08:33:10.294 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.295 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.295 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:10.295 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.177176+0000", 2023-11-05T08:33:10.295 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0063664289999999998 2023-11-05T08:33:10.295 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.296 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.296 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:10.296 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.177247+0000", 2023-11-05T08:33:10.296 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.0905e-05 2023-11-05T08:33:10.296 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.296 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:10.297 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.297 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.297 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.297 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.2s3 41/40 ECSubWrite(tid=189, reqid=osd.13.0:188, at_version=0'0, trim_to=0'0, roll_forward_to=41'11))", 2023-11-05T08:33:10.297 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.184445+0000", 2023-11-05T08:33:10.297 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 15.998426391000001, 2023-11-05T08:33:10.298 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0050416910000000001, 2023-11-05T08:33:10.298 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:10.298 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:10.298 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:10.298 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.299 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:10.299 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184445+0000", 2023-11-05T08:33:10.299 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.299 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.299 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.300 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:10.300 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184445+0000", 2023-11-05T08:33:10.300 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.300 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.300 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.300 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:10.301 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184447+0000", 2023-11-05T08:33:10.301 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.688e-06 2023-11-05T08:33:10.301 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.301 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.301 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:10.301 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184471+0000", 2023-11-05T08:33:10.302 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3745999999999999e-05 2023-11-05T08:33:10.302 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.302 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.302 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:10.302 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184531+0000", 2023-11-05T08:33:10.303 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.0603999999999998e-05 2023-11-05T08:33:10.303 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.303 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.303 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:10.303 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184551+0000", 2023-11-05T08:33:10.303 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0239000000000001e-05 2023-11-05T08:33:10.304 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.304 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.304 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:10.304 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184713+0000", 2023-11-05T08:33:10.304 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00016179399999999999 2023-11-05T08:33:10.304 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.305 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.305 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:10.305 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184734+0000", 2023-11-05T08:33:10.305 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0455e-05 2023-11-05T08:33:10.305 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.305 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.306 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:10.306 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.189410+0000", 2023-11-05T08:33:10.306 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0046763350000000002 2023-11-05T08:33:10.306 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.306 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.307 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:10.307 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.189487+0000", 2023-11-05T08:33:10.307 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.6829999999999995e-05 2023-11-05T08:33:10.307 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.307 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:10.307 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.308 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.308 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.308 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.as5 42/33 ECSubWrite(tid=38, reqid=osd.7.0:38, at_version=42'16, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:33:10.308 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:55.127656+0000", 2023-11-05T08:33:10.308 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 15.055215603000001, 2023-11-05T08:33:10.309 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.016413512000000002, 2023-11-05T08:33:10.309 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:10.309 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:10.309 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:10.309 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.309 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:10.310 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127656+0000", 2023-11-05T08:33:10.310 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.310 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.310 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.310 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:10.310 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127656+0000", 2023-11-05T08:33:10.311 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.311 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.311 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.311 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:10.311 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127656+0000", 2023-11-05T08:33:10.311 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.2699999999999999e-07 2023-11-05T08:33:10.312 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.312 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.312 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:10.312 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127732+0000", 2023-11-05T08:33:10.312 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.5978999999999999e-05 2023-11-05T08:33:10.312 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.313 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.313 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:10.313 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127763+0000", 2023-11-05T08:33:10.313 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0672e-05 2023-11-05T08:33:10.313 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.313 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.314 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:10.314 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127785+0000", 2023-11-05T08:33:10.314 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2059e-05 2023-11-05T08:33:10.314 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.314 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.315 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:10.315 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127975+0000", 2023-11-05T08:33:10.315 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000189946 2023-11-05T08:33:10.315 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.315 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.315 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:10.316 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127999+0000", 2023-11-05T08:33:10.316 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3555999999999999e-05 2023-11-05T08:33:10.316 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.316 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.316 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:10.316 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.143967+0000", 2023-11-05T08:33:10.317 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.015968781000000001 2023-11-05T08:33:10.317 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.317 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.317 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:10.317 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.144069+0000", 2023-11-05T08:33:10.318 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000101992 2023-11-05T08:33:10.318 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.318 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:10.318 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.318 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.319 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.319 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.as5 42/33 ECSubWrite(tid=39, reqid=osd.7.0:39, at_version=42'18, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:33:10.319 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:55.127880+0000", 2023-11-05T08:33:10.319 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 15.054991118, 2023-11-05T08:33:10.319 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.019355170000000001, 2023-11-05T08:33:10.319 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:10.320 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:10.320 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:10.320 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.320 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:10.320 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127880+0000", 2023-11-05T08:33:10.320 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.321 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.321 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.321 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:10.321 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127880+0000", 2023-11-05T08:33:10.321 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.321 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.322 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.322 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:10.322 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127882+0000", 2023-11-05T08:33:10.322 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.187e-06 2023-11-05T08:33:10.322 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.323 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.323 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:10.323 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127934+0000", 2023-11-05T08:33:10.323 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.2553000000000003e-05 2023-11-05T08:33:10.323 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.324 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.324 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:10.324 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127961+0000", 2023-11-05T08:33:10.324 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6999e-05 2023-11-05T08:33:10.324 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.324 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.325 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:10.325 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127983+0000", 2023-11-05T08:33:10.325 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1778999999999999e-05 2023-11-05T08:33:10.325 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.325 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.326 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:10.326 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.129155+0000", 2023-11-05T08:33:10.326 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001172022 2023-11-05T08:33:10.326 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.326 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.326 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:10.327 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.129180+0000", 2023-11-05T08:33:10.327 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4841000000000002e-05 2023-11-05T08:33:10.327 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.327 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.327 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:10.329 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.147146+0000", 2023-11-05T08:33:10.329 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.017965990000000001 2023-11-05T08:33:10.329 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.329 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.329 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:10.330 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.147236+0000", 2023-11-05T08:33:10.330 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.9798999999999999e-05 2023-11-05T08:33:10.330 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.330 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:10.330 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.331 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.331 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.331 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.es5 42/30 ECSubWrite(tid=43, reqid=osd.2.0:43, at_version=42'29, trim_to=0'0, roll_forward_to=41'27))", 2023-11-05T08:33:10.331 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:55.134988+0000", 2023-11-05T08:33:10.331 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 15.047883302000001, 2023-11-05T08:33:10.332 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.012343942, 2023-11-05T08:33:10.332 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:10.332 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:10.332 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:10.332 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.332 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:10.333 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.134988+0000", 2023-11-05T08:33:10.333 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.333 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.333 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.333 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:10.334 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.134988+0000", 2023-11-05T08:33:10.334 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.334 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.334 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.334 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:10.334 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.134989+0000", 2023-11-05T08:33:10.335 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.6000000000000003e-07 2023-11-05T08:33:10.335 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.335 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.335 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:10.335 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.135039+0000", 2023-11-05T08:33:10.336 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.9879999999999997e-05 2023-11-05T08:33:10.336 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.336 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.336 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:10.336 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.135069+0000", 2023-11-05T08:33:10.337 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0445000000000001e-05 2023-11-05T08:33:10.337 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.337 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.337 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:10.337 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.135094+0000", 2023-11-05T08:33:10.338 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4646000000000001e-05 2023-11-05T08:33:10.338 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.338 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.338 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:10.338 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.135758+0000", 2023-11-05T08:33:10.338 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00066366799999999996 2023-11-05T08:33:10.339 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.339 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.339 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:10.339 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.135780+0000", 2023-11-05T08:33:10.339 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2509000000000001e-05 2023-11-05T08:33:10.339 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.340 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.340 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:10.340 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.147237+0000", 2023-11-05T08:33:10.340 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.011456401999999999 2023-11-05T08:33:10.340 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.340 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.341 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:10.341 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.147332+0000", 2023-11-05T08:33:10.341 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.5631999999999995e-05 2023-11-05T08:33:10.341 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.341 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:10.341 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.342 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.342 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.342 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.as5 42/33 ECSubWrite(tid=40, reqid=osd.7.0:39, at_version=0'0, trim_to=0'0, roll_forward_to=42'18))", 2023-11-05T08:33:10.342 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:55.152136+0000", 2023-11-05T08:33:10.342 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 15.030735906, 2023-11-05T08:33:10.343 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0053161390000000001, 2023-11-05T08:33:10.343 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:10.343 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:10.343 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:10.343 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.343 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:10.344 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.152136+0000", 2023-11-05T08:33:10.344 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.344 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.344 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.344 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:10.344 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.152136+0000", 2023-11-05T08:33:10.345 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.345 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.345 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.345 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:10.345 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.152138+0000", 2023-11-05T08:33:10.346 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.883e-06 2023-11-05T08:33:10.346 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.346 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.346 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:10.346 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.152165+0000", 2023-11-05T08:33:10.346 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7282000000000001e-05 2023-11-05T08:33:10.347 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.347 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.347 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:10.347 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.152204+0000", 2023-11-05T08:33:10.347 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.8964000000000002e-05 2023-11-05T08:33:10.348 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.348 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.348 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:10.348 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.152230+0000", 2023-11-05T08:33:10.348 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6446000000000001e-05 2023-11-05T08:33:10.348 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.349 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.349 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:10.349 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.152351+0000", 2023-11-05T08:33:10.349 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00012077 2023-11-05T08:33:10.349 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.349 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.350 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:10.350 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.152368+0000", 2023-11-05T08:33:10.350 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6599000000000002e-05 2023-11-05T08:33:10.350 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.350 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.350 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:10.351 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.157352+0000", 2023-11-05T08:33:10.351 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0049846939999999996 2023-11-05T08:33:10.351 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.352 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.352 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:10.352 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.157452+0000", 2023-11-05T08:33:10.352 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.9500999999999995e-05 2023-11-05T08:33:10.352 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.353 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:10.353 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.353 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.353 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.354 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.2s3 42/40 ECSubWrite(tid=193, reqid=osd.13.0:192, at_version=0'0, trim_to=0'0, roll_forward_to=42'13))", 2023-11-05T08:33:10.354 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:55.161223+0000", 2023-11-05T08:33:10.354 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 15.021648816000001, 2023-11-05T08:33:10.354 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0050713499999999996, 2023-11-05T08:33:10.355 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:10.355 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:10.355 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:10.355 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.355 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:10.356 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.161223+0000", 2023-11-05T08:33:10.356 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.356 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.356 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.356 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:10.357 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.161223+0000", 2023-11-05T08:33:10.357 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.357 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.357 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.357 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:10.358 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.161224+0000", 2023-11-05T08:33:10.358 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.26e-06 2023-11-05T08:33:10.358 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.358 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.358 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:10.359 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.161242+0000", 2023-11-05T08:33:10.359 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8153000000000001e-05 2023-11-05T08:33:10.359 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.359 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.359 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:10.359 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.161270+0000", 2023-11-05T08:33:10.360 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7538e-05 2023-11-05T08:33:10.360 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.360 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.360 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:10.360 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.161287+0000", 2023-11-05T08:33:10.361 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.7129000000000002e-05 2023-11-05T08:33:10.361 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.361 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.361 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:10.361 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.161467+0000", 2023-11-05T08:33:10.362 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00018033400000000001 2023-11-05T08:33:10.362 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.362 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.362 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:10.362 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.161483+0000", 2023-11-05T08:33:10.363 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5673000000000001e-05 2023-11-05T08:33:10.363 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.363 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.363 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:10.363 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.166210+0000", 2023-11-05T08:33:10.364 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0047273469999999998 2023-11-05T08:33:10.364 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.364 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.364 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:10.364 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.166294+0000", 2023-11-05T08:33:10.365 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.3916000000000001e-05 2023-11-05T08:33:10.365 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.365 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:10.365 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.365 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.366 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.366 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.18s1 51/45 ECSubWrite(tid=202, reqid=client.4691.0:335, at_version=51'23, trim_to=0'0, roll_forward_to=49'21))", 2023-11-05T08:33:10.366 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:04.244158+0000", 2023-11-05T08:33:10.366 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 5.9387130849999998, 2023-11-05T08:33:10.367 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0079120630000000004, 2023-11-05T08:33:10.367 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:10.367 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:10.367 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:10.367 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.368 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:10.368 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:04.244158+0000", 2023-11-05T08:33:10.368 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.368 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.368 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.368 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:10.369 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:04.244158+0000", 2023-11-05T08:33:10.369 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.369 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.369 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.369 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:10.370 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:04.244160+0000", 2023-11-05T08:33:10.370 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6670000000000001e-06 2023-11-05T08:33:10.370 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.370 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.370 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:10.371 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:04.244222+0000", 2023-11-05T08:33:10.371 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.1734000000000002e-05 2023-11-05T08:33:10.371 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.371 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.371 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:10.372 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:04.244256+0000", 2023-11-05T08:33:10.372 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.3822000000000002e-05 2023-11-05T08:33:10.372 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.372 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.372 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:10.372 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:04.244278+0000", 2023-11-05T08:33:10.373 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2419e-05 2023-11-05T08:33:10.373 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.373 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.373 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:10.373 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:04.246542+0000", 2023-11-05T08:33:10.374 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0022635989999999998 2023-11-05T08:33:10.374 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.374 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.374 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:10.374 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:04.246558+0000", 2023-11-05T08:33:10.375 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5936000000000001e-05 2023-11-05T08:33:10.375 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.375 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.375 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:10.375 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:04.251996+0000", 2023-11-05T08:33:10.375 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0054384029999999996 2023-11-05T08:33:10.376 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.376 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.376 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:10.376 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:04.252071+0000", 2023-11-05T08:33:10.376 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.4482999999999998e-05 2023-11-05T08:33:10.377 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.377 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:10.377 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.377 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.377 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.377 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.as5 52/45 ECSubWrite(tid=45, reqid=client.4691.0:349, at_version=52'21, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:10.378 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:05.263668+0000", 2023-11-05T08:33:10.378 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 4.9192036320000003, 2023-11-05T08:33:10.378 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0089752809999999999, 2023-11-05T08:33:10.378 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:10.378 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:10.379 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:10.379 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.379 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:10.379 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.263668+0000", 2023-11-05T08:33:10.379 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.379 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.380 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.380 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:10.380 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.263668+0000", 2023-11-05T08:33:10.380 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.380 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.381 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.381 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:10.381 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.263669+0000", 2023-11-05T08:33:10.381 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.3600000000000003e-07 2023-11-05T08:33:10.381 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.382 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.382 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:10.382 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.264055+0000", 2023-11-05T08:33:10.382 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00038651799999999997 2023-11-05T08:33:10.382 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.383 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.383 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:10.383 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.264091+0000", 2023-11-05T08:33:10.383 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.6089e-05 2023-11-05T08:33:10.383 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.384 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.384 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:10.384 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.264119+0000", 2023-11-05T08:33:10.384 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7665e-05 2023-11-05T08:33:10.384 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.385 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.385 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:10.385 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.264344+0000", 2023-11-05T08:33:10.385 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00022567400000000001 2023-11-05T08:33:10.385 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.386 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.386 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:10.386 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.264372+0000", 2023-11-05T08:33:10.386 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7118999999999999e-05 2023-11-05T08:33:10.386 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.387 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.387 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:10.387 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.272564+0000", 2023-11-05T08:33:10.387 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0081919829999999999 2023-11-05T08:33:10.388 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.388 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.388 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:10.388 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.272643+0000", 2023-11-05T08:33:10.388 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.9597000000000004e-05 2023-11-05T08:33:10.388 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.389 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:10.389 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.389 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.389 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.390 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.18s1 52/45 ECSubWrite(tid=205, reqid=client.4691.0:351, at_version=52'24, trim_to=0'0, roll_forward_to=51'23))", 2023-11-05T08:33:10.390 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:05.282816+0000", 2023-11-05T08:33:10.390 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 4.9000553419999999, 2023-11-05T08:33:10.390 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0041039559999999997, 2023-11-05T08:33:10.391 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:10.391 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:10.391 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:10.391 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.391 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:10.392 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.282816+0000", 2023-11-05T08:33:10.392 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.392 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.392 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.392 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:10.393 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.282816+0000", 2023-11-05T08:33:10.393 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.393 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.393 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.393 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:10.394 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.282818+0000", 2023-11-05T08:33:10.394 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3960000000000001e-06 2023-11-05T08:33:10.394 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.394 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.394 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:10.395 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.283296+0000", 2023-11-05T08:33:10.395 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00047850799999999999 2023-11-05T08:33:10.395 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.395 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.396 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:10.396 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.283323+0000", 2023-11-05T08:33:10.396 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7277999999999999e-05 2023-11-05T08:33:10.396 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.396 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.396 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:10.397 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.283344+0000", 2023-11-05T08:33:10.397 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0316e-05 2023-11-05T08:33:10.397 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.397 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.397 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:10.398 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.283534+0000", 2023-11-05T08:33:10.398 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000190223 2023-11-05T08:33:10.398 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.398 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.398 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:10.399 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.283566+0000", 2023-11-05T08:33:10.399 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.1841999999999998e-05 2023-11-05T08:33:10.399 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.399 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.399 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:10.400 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.286834+0000", 2023-11-05T08:33:10.400 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00326808 2023-11-05T08:33:10.400 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.400 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.400 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:10.400 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.286920+0000", 2023-11-05T08:33:10.401 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.6312999999999998e-05 2023-11-05T08:33:10.401 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.401 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:10.401 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.401 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.402 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.402 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.8s6 52/40 ECSubWrite(tid=38, reqid=client.4691.0:342, at_version=52'17, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:10.402 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:05.292740+0000", 2023-11-05T08:33:10.402 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 4.8901310919999998, 2023-11-05T08:33:10.402 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0041903890000000001, 2023-11-05T08:33:10.403 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:10.403 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:10.403 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:10.403 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.403 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:10.403 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.292740+0000", 2023-11-05T08:33:10.404 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.404 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.404 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.404 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:10.404 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.292740+0000", 2023-11-05T08:33:10.405 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:10.405 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.405 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.405 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:10.405 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.292741+0000", 2023-11-05T08:33:10.406 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.9100000000000003e-07 2023-11-05T08:33:10.406 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.406 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.406 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:10.406 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.293090+0000", 2023-11-05T08:33:10.406 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00034861499999999998 2023-11-05T08:33:10.407 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.407 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.407 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:10.407 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.293125+0000", 2023-11-05T08:33:10.407 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.4826999999999997e-05 2023-11-05T08:33:10.408 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.408 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.408 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:10.408 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.293145+0000", 2023-11-05T08:33:10.408 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0488000000000001e-05 2023-11-05T08:33:10.409 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.409 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.409 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:10.409 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.293335+0000", 2023-11-05T08:33:10.409 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000189438 2023-11-05T08:33:10.409 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.410 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.410 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:10.410 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.293365+0000", 2023-11-05T08:33:10.410 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0877999999999998e-05 2023-11-05T08:33:10.410 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.410 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.411 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:10.411 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.296862+0000", 2023-11-05T08:33:10.411 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.003496435 2023-11-05T08:33:10.411 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:10.411 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:10.412 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:10.412 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.296931+0000", 2023-11-05T08:33:10.412 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.8917000000000004e-05 2023-11-05T08:33:10.412 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.412 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:10.412 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.413 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:10.413 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:10.413 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:33:10.414 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-11-05T08:33:10.414 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-05T08:33:10.416 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:33:10.413+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:33:10.417 INFO:tasks.rados.rados.0.smithi090.stdout:150: done (2 left) 2023-11-05T08:33:10.417 INFO:tasks.rados.rados.0.smithi090.stdout:151: done (1 left) 2023-11-05T08:33:10.417 INFO:tasks.rados.rados.0.smithi090.stdout:152: read oid 10 snap -1 2023-11-05T08:33:10.418 INFO:tasks.rados.rados.0.smithi090.stdout:152: expect (ObjNum 9 snap 0 seq_num 9) 2023-11-05T08:33:10.418 INFO:tasks.rados.rados.0.smithi090.stdout:153: delete oid 34 current snap is 12 2023-11-05T08:33:10.418 INFO:tasks.rados.rados.0.smithi090.stdout:153: done (2 left) 2023-11-05T08:33:10.418 INFO:tasks.rados.rados.0.smithi090.stdout:154: snap_create 2023-11-05T08:33:10.418 INFO:tasks.rados.rados.0.smithi090.stdout:152: expect (ObjNum 9 snap 0 seq_num 9) 2023-11-05T08:33:10.419 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2023-11-05T08:33:10.515 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:33:10.512+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:33:10.550 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:33:10.550 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2023-11-05T08:33:10.551 INFO:teuthology.orchestra.run.smithi069.stdout: "num_ops": 0 2023-11-05T08:33:10.551 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:33:10.551 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-11-05T08:33:10.564 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2023-11-05T08:33:10.575 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1: {} 2023-11-05T08:33:10.575 INFO:teuthology.orchestra.run.smithi002.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-11-05T08:33:10.592 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2: {} 2023-11-05T08:33:10.593 INFO:teuthology.orchestra.run.smithi002.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-11-05T08:33:10.593 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-11-05T08:33:10.609 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4: {} 2023-11-05T08:33:10.609 INFO:teuthology.orchestra.run.smithi002.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-11-05T08:33:10.615 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~14s 2023-11-05T08:33:10.616 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:10.613+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:10.629 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5: {} 2023-11-05T08:33:10.630 INFO:teuthology.orchestra.run.smithi002.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-11-05T08:33:10.649 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6: {} 2023-11-05T08:33:10.649 INFO:teuthology.orchestra.run.smithi002.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-11-05T08:33:10.669 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7: {} 2023-11-05T08:33:10.670 INFO:teuthology.orchestra.run.smithi002.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-11-05T08:33:10.686 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8: {} 2023-11-05T08:33:10.686 INFO:teuthology.orchestra.run.smithi002.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-11-05T08:33:10.693 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:33:10.693 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2023-11-05T08:33:10.693 INFO:teuthology.orchestra.run.smithi069.stdout: "complaint_time": 30, 2023-11-05T08:33:10.693 INFO:teuthology.orchestra.run.smithi069.stdout: "num_blocked_ops": 0 2023-11-05T08:33:10.694 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:33:10.705 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9: {} 2023-11-05T08:33:10.706 INFO:teuthology.orchestra.run.smithi002.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-11-05T08:33:10.708 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2023-11-05T08:33:10.718 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:33:10.717+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:33:10.724 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10: {} 2023-11-05T08:33:10.724 INFO:teuthology.orchestra.run.smithi002.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-11-05T08:33:10.745 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11: {} 2023-11-05T08:33:10.746 INFO:teuthology.orchestra.run.smithi002.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-11-05T08:33:10.763 INFO:teuthology.orchestra.run.smithi002.stdout:osd.12: {} 2023-11-05T08:33:10.764 INFO:teuthology.orchestra.run.smithi002.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-11-05T08:33:10.785 INFO:teuthology.orchestra.run.smithi002.stdout:osd.13: {} 2023-11-05T08:33:10.785 INFO:teuthology.orchestra.run.smithi002.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-11-05T08:33:10.805 INFO:teuthology.orchestra.run.smithi002.stdout:osd.14: {} 2023-11-05T08:33:10.805 INFO:teuthology.orchestra.run.smithi002.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-11-05T08:33:10.806 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-05T08:33:10.802+0000 7f7a18f5a640 0 -- 172.21.15.2:0/536532264 >> [v2:172.21.15.90:6824/739151808,v1:172.21.15.90:6825/739151808] conn(0x7f7a14217050 msgr2=0x7f7a14219460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-11-05T08:33:10.818 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:33:10.812+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:33:10.831 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:10.832 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:10.846 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2023-11-05T08:33:10.918 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:10.912+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:10.967 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:10.967 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:10.981 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2023-11-05T08:33:11.019 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:33:11.016+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:33:11.031 INFO:teuthology.orchestra.run.smithi002.stdout:osd.15: {} 2023-11-05T08:33:11.031 INFO:teuthology.orchestra.run.smithi002.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-11-05T08:33:11.037 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-05T08:33:11.037 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-05T08:33:11.103 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:11.104 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:11.118 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2023-11-05T08:33:11.120 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:33:11.117+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:33:11.220 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:33:11.217+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:33:11.236 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:11.237 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:11.251 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-11-05T08:33:11.279 INFO:teuthology.orchestra.run.smithi002.stderr:noscrub is unset 2023-11-05T08:33:11.287 INFO:tasks.rados.rados.0.smithi090.stdout:152: done (2 left) 2023-11-05T08:33:11.287 INFO:tasks.rados.rados.0.smithi090.stdout:154: done (1 left) 2023-11-05T08:33:11.288 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 34 current snap is 13 2023-11-05T08:33:11.288 INFO:tasks.rados.rados.0.smithi090.stdout:155: seq_num 79 ranges {0=573440,573440=425984,999424=344064} 2023-11-05T08:33:11.291 INFO:tasks.rados.rados.0.smithi090.stdout:155: writing smithi09039252-34 from 0 to 573440 tid 1 2023-11-05T08:33:11.294 INFO:tasks.rados.rados.0.smithi090.stdout:155: writing smithi09039252-34 from 573440 to 999424 tid 2 2023-11-05T08:33:11.297 INFO:tasks.rados.rados.0.smithi090.stdout:155: writing smithi09039252-34 from 999424 to 1343488 tid 3 2023-11-05T08:33:11.297 INFO:tasks.rados.rados.0.smithi090.stdout:156: read oid 19 snap 11 2023-11-05T08:33:11.298 INFO:tasks.rados.rados.0.smithi090.stdout:156: expect (ObjNum 18 snap 0 seq_num 18) 2023-11-05T08:33:11.298 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 45 current snap is 13 2023-11-05T08:33:11.298 INFO:tasks.rados.rados.0.smithi090.stdout:157: seq_num 80 ranges {753664=589824,1343488=573440,1916928=196608} 2023-11-05T08:33:11.307 INFO:tasks.rados.rados.0.smithi090.stdout:157: writing smithi09039252-45 from 753664 to 1343488 tid 1 2023-11-05T08:33:11.312 INFO:tasks.rados.rados.0.smithi090.stdout:157: writing smithi09039252-45 from 1343488 to 1916928 tid 2 2023-11-05T08:33:11.314 INFO:tasks.rados.rados.0.smithi090.stdout:157: writing smithi09039252-45 from 1916928 to 2113536 tid 3 2023-11-05T08:33:11.314 INFO:tasks.rados.rados.0.smithi090.stdout:158: delete oid 21 current snap is 13 2023-11-05T08:33:11.314 INFO:tasks.rados.rados.0.smithi090.stdout:156: expect (ObjNum 18 snap 0 seq_num 18) 2023-11-05T08:33:11.321 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:11.317+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:11.329 INFO:tasks.rados.rados.0.smithi090.stdout:155: finishing write tid 1 to smithi09039252-34 2023-11-05T08:33:11.330 INFO:tasks.rados.rados.0.smithi090.stdout:155: finishing write tid 2 to smithi09039252-34 2023-11-05T08:33:11.330 INFO:tasks.rados.rados.0.smithi090.stdout:155: finishing write tid 3 to smithi09039252-34 2023-11-05T08:33:11.330 INFO:tasks.rados.rados.0.smithi090.stdout:155: finishing write tid 4 to smithi09039252-34 2023-11-05T08:33:11.330 INFO:tasks.rados.rados.0.smithi090.stdout:155: finishing write tid 5 to smithi09039252-34 2023-11-05T08:33:11.330 INFO:tasks.rados.rados.0.smithi090.stdout:155: oid 34 updating version 0 to 48 2023-11-05T08:33:11.331 INFO:tasks.rados.rados.0.smithi090.stdout:155: oid 34 version 48 is already newer than 46 2023-11-05T08:33:11.331 INFO:tasks.rados.rados.0.smithi090.stdout:155: oid 34 version 48 is already newer than 47 2023-11-05T08:33:11.331 INFO:tasks.rados.rados.0.smithi090.stdout:155: oid 34 updating version 48 to 49 2023-11-05T08:33:11.331 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 34 v 49 (ObjNum 79 snap 13 seq_num 79) dirty exists 2023-11-05T08:33:11.331 INFO:tasks.rados.rados.0.smithi090.stdout:155: left oid 34 (ObjNum 79 snap 13 seq_num 79) 2023-11-05T08:33:11.332 INFO:tasks.rados.rados.0.smithi090.stdout:157: finishing write tid 1 to smithi09039252-45 2023-11-05T08:33:11.332 INFO:tasks.rados.rados.0.smithi090.stdout:157: finishing write tid 2 to smithi09039252-45 2023-11-05T08:33:11.332 INFO:tasks.rados.rados.0.smithi090.stdout:157: finishing write tid 3 to smithi09039252-45 2023-11-05T08:33:11.332 INFO:tasks.rados.rados.0.smithi090.stdout:157: finishing write tid 4 to smithi09039252-45 2023-11-05T08:33:11.332 INFO:tasks.rados.rados.0.smithi090.stdout:157: finishing write tid 5 to smithi09039252-45 2023-11-05T08:33:11.333 INFO:tasks.rados.rados.0.smithi090.stdout:157: oid 45 updating version 0 to 27 2023-11-05T08:33:11.333 INFO:tasks.rados.rados.0.smithi090.stdout:157: oid 45 version 27 is already newer than 24 2023-11-05T08:33:11.333 INFO:tasks.rados.rados.0.smithi090.stdout:157: oid 45 version 27 is already newer than 25 2023-11-05T08:33:11.333 INFO:tasks.rados.rados.0.smithi090.stdout:157: oid 45 version 27 is already newer than 26 2023-11-05T08:33:11.333 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 45 v 27 (ObjNum 80 snap 13 seq_num 80) dirty exists 2023-11-05T08:33:11.333 INFO:tasks.rados.rados.0.smithi090.stdout:157: left oid 45 (ObjNum 80 snap 13 seq_num 80) 2023-11-05T08:33:11.334 INFO:tasks.rados.rados.0.smithi090.stdout:155: done (4 left) 2023-11-05T08:33:11.334 INFO:tasks.rados.rados.0.smithi090.stdout:156: done (3 left) 2023-11-05T08:33:11.334 INFO:tasks.rados.rados.0.smithi090.stdout:157: done (2 left) 2023-11-05T08:33:11.334 INFO:tasks.rados.rados.0.smithi090.stdout:158: done (1 left) 2023-11-05T08:33:11.334 INFO:tasks.rados.rados.0.smithi090.stdout:159: read oid 26 snap -1 2023-11-05T08:33:11.335 INFO:tasks.rados.rados.0.smithi090.stdout:159: expect (ObjNum 25 snap 0 seq_num 25) 2023-11-05T08:33:11.335 INFO:tasks.rados.rados.0.smithi090.stdout:160: read oid 39 snap -1 2023-11-05T08:33:11.335 INFO:tasks.rados.rados.0.smithi090.stdout:160: expect (ObjNum 38 snap 0 seq_num 38) 2023-11-05T08:33:11.335 INFO:tasks.rados.rados.0.smithi090.stdout:161: snap_remove snap 12 2023-11-05T08:33:11.393 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:11.393 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:11.408 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-11-05T08:33:11.421 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:33:11.418+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:33:11.522 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:11.521+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:11.566 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:11.567 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:11.581 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-11-05T08:33:11.623 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:33:11.622+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:33:11.704 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:11.704 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:11.718 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2023-11-05T08:33:11.724 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:33:11.721+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:33:11.825 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:11.821+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:11.840 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:11.840 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:11.854 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2023-11-05T08:33:11.926 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:33:11.920+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:33:11.978 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:11.978 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:11.992 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2023-11-05T08:33:12.035 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:33:12.032+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:33:12.128 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:33:12.126+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:33:12.156 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:12.156 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:12.171 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2023-11-05T08:33:12.229 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:33:12.226+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:33:12.286 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:12.287 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:12.291 INFO:tasks.rados.rados.0.smithi090.stdout:161: done (3 left) 2023-11-05T08:33:12.291 INFO:tasks.rados.rados.0.smithi090.stdout:162: snap_create 2023-11-05T08:33:12.291 INFO:tasks.rados.rados.0.smithi090.stdout:159: expect (ObjNum 25 snap 0 seq_num 25) 2023-11-05T08:33:12.301 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2023-11-05T08:33:12.310 INFO:tasks.rados.rados.0.smithi090.stdout:160: expect (ObjNum 38 snap 0 seq_num 38) 2023-11-05T08:33:12.329 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:33:12.326+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:33:12.417 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:12.417 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:12.431 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2023-11-05T08:33:12.432 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:33:12.429+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:33:12.531 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:33:12.528+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:33:12.596 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:12.597 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:12.610 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-11-05T08:33:12.633 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:33:12.629+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:33:12.730 INFO:teuthology.orchestra.run.smithi002.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-05T08:33:12.733 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-05T08:33:12.734 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-11-05T08:33:12.735 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:33:12.730+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:33:12.834 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:33:12.834+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:33:12.841 INFO:teuthology.orchestra.run.smithi002.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-05T08:33:12.844 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-05T08:33:12.845 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-11-05T08:33:12.935 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:33:12.934+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:33:12.957 INFO:teuthology.orchestra.run.smithi002.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-05T08:33:12.961 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-05T08:33:12.961 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-11-05T08:33:13.035 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:33:13.034+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:33:13.038 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-05T08:33:13.089 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:13.090 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:13.104 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-11-05T08:33:13.136 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:33:13.134+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:33:13.177 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-11-05T08:33:13.198 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1: {} 2023-11-05T08:33:13.199 INFO:teuthology.orchestra.run.smithi002.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-11-05T08:33:13.220 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2: {} 2023-11-05T08:33:13.220 INFO:teuthology.orchestra.run.smithi002.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-11-05T08:33:13.220 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-11-05T08:33:13.225 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:13.226 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:13.237 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:13.236+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:13.238 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4: {} 2023-11-05T08:33:13.238 INFO:teuthology.orchestra.run.smithi002.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-11-05T08:33:13.240 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-11-05T08:33:13.259 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5: {} 2023-11-05T08:33:13.259 INFO:teuthology.orchestra.run.smithi002.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-11-05T08:33:13.279 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6: {} 2023-11-05T08:33:13.279 INFO:teuthology.orchestra.run.smithi002.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-11-05T08:33:13.299 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7: {} 2023-11-05T08:33:13.300 INFO:teuthology.orchestra.run.smithi002.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-11-05T08:33:13.308 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-11-05T08:33:13.310 INFO:tasks.rados.rados.0.smithi090.stdout:159: done (3 left) 2023-11-05T08:33:13.310 INFO:tasks.rados.rados.0.smithi090.stdout:160: done (2 left) 2023-11-05T08:33:13.311 INFO:tasks.rados.rados.0.smithi090.stdout:162: done (1 left) 2023-11-05T08:33:13.311 INFO:tasks.rados.rados.0.smithi090.stdout:163: snap_create 2023-11-05T08:33:13.319 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8: {} 2023-11-05T08:33:13.320 INFO:teuthology.orchestra.run.smithi002.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-11-05T08:33:13.338 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:33:13.334+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:33:13.341 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9: {} 2023-11-05T08:33:13.341 INFO:teuthology.orchestra.run.smithi002.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-11-05T08:33:13.360 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:33:13.360 INFO:teuthology.orchestra.run.smithi002.stdout: "size": 20, 2023-11-05T08:33:13.360 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 600, 2023-11-05T08:33:13.361 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [ 2023-11-05T08:33:13.361 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.361 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.5s2 35/30 ECSubWrite(tid=20, reqid=client.4691.0:221, at_version=35'11, trim_to=0'0, roll_forward_to=33'9))", 2023-11-05T08:33:13.361 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:48.086738+0000", 2023-11-05T08:33:13.361 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 25.272993414999998, 2023-11-05T08:33:13.362 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0065328219999999998, 2023-11-05T08:33:13.362 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:13.362 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:13.362 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:13.362 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.363 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:13.363 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.086738+0000", 2023-11-05T08:33:13.363 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.363 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.363 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.364 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:13.364 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.086738+0000", 2023-11-05T08:33:13.364 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.364 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.364 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.365 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:13.365 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.086739+0000", 2023-11-05T08:33:13.365 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.22e-07 2023-11-05T08:33:13.365 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.365 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.365 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:13.366 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.087297+0000", 2023-11-05T08:33:13.366 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00055798899999999995 2023-11-05T08:33:13.366 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.366 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.366 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:13.366 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.087338+0000", 2023-11-05T08:33:13.367 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.1171000000000003e-05 2023-11-05T08:33:13.367 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.367 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.367 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:13.367 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.087364+0000", 2023-11-05T08:33:13.368 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.5826999999999999e-05 2023-11-05T08:33:13.368 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.368 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.369 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:13.369 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.087572+0000", 2023-11-05T08:33:13.369 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000208574 2023-11-05T08:33:13.369 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.369 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.369 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:13.370 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.087598+0000", 2023-11-05T08:33:13.370 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.5901000000000002e-05 2023-11-05T08:33:13.370 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.370 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.370 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:13.371 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.093224+0000", 2023-11-05T08:33:13.371 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0056260210000000001 2023-11-05T08:33:13.371 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.371 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.371 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:13.372 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.093271+0000", 2023-11-05T08:33:13.372 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.7117000000000003e-05 2023-11-05T08:33:13.372 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.372 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:13.372 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.372 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.373 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.373 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.5s2 35/30 ECSubWrite(tid=21, reqid=client.4691.0:222, at_version=35'12, trim_to=0'0, roll_forward_to=33'9))", 2023-11-05T08:33:13.373 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:48.091862+0000", 2023-11-05T08:33:13.373 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 25.267869942000001, 2023-11-05T08:33:13.373 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.003284333, 2023-11-05T08:33:13.374 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:13.374 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:13.374 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:13.374 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.374 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:13.374 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.091862+0000", 2023-11-05T08:33:13.375 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.375 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.375 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.375 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:13.375 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.091862+0000", 2023-11-05T08:33:13.375 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.376 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.376 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.376 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:13.376 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.091863+0000", 2023-11-05T08:33:13.376 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.237e-06 2023-11-05T08:33:13.377 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.377 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.377 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:13.377 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.092010+0000", 2023-11-05T08:33:13.377 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00014685699999999999 2023-11-05T08:33:13.377 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.378 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.378 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:13.378 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.092034+0000", 2023-11-05T08:33:13.378 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.4596e-05 2023-11-05T08:33:13.378 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.379 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.379 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:13.379 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.092047+0000", 2023-11-05T08:33:13.379 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.2841e-05 2023-11-05T08:33:13.379 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.379 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.380 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:13.380 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.092166+0000", 2023-11-05T08:33:13.380 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000118317 2023-11-05T08:33:13.380 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.380 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.380 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:13.381 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.092182+0000", 2023-11-05T08:33:13.381 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.5852999999999999e-05 2023-11-05T08:33:13.381 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.381 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.381 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:13.381 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.095105+0000", 2023-11-05T08:33:13.382 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00292312 2023-11-05T08:33:13.382 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.382 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.382 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:13.382 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.095146+0000", 2023-11-05T08:33:13.383 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.1511999999999998e-05 2023-11-05T08:33:13.383 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.383 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:13.383 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.383 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.383 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.384 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.ds6 36/30 ECSubWrite(tid=20, reqid=client.4691.0:233, at_version=36'10, trim_to=0'0, roll_forward_to=34'8))", 2023-11-05T08:33:13.384 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:49.067390+0000", 2023-11-05T08:33:13.384 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 24.292342035000001, 2023-11-05T08:33:13.384 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0062948609999999997, 2023-11-05T08:33:13.384 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:13.385 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:13.385 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:13.385 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.385 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:13.385 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.067390+0000", 2023-11-05T08:33:13.385 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.386 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.386 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.386 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:13.386 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.067390+0000", 2023-11-05T08:33:13.386 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.386 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.387 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.387 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:13.387 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.067390+0000", 2023-11-05T08:33:13.387 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.9700000000000003e-07 2023-11-05T08:33:13.387 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.388 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.388 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:13.388 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.067421+0000", 2023-11-05T08:33:13.388 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.1350000000000003e-05 2023-11-05T08:33:13.388 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.388 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.389 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:13.389 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.067439+0000", 2023-11-05T08:33:13.389 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.7833e-05 2023-11-05T08:33:13.389 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.389 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.389 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:13.390 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.067452+0000", 2023-11-05T08:33:13.390 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.2318e-05 2023-11-05T08:33:13.390 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.390 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.390 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:13.391 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.067542+0000", 2023-11-05T08:33:13.391 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.0628e-05 2023-11-05T08:33:13.391 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.391 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.391 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:13.392 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.067552+0000", 2023-11-05T08:33:13.392 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0275000000000001e-05 2023-11-05T08:33:13.392 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.392 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.392 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:13.392 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.073642+0000", 2023-11-05T08:33:13.393 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.006089779 2023-11-05T08:33:13.393 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.393 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.393 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:13.393 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.073685+0000", 2023-11-05T08:33:13.393 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.2381e-05 2023-11-05T08:33:13.394 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.394 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:13.394 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.394 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.394 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.394 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.3s2 36/30 ECSubWrite(tid=183, reqid=osd.13.0:183, at_version=36'12, trim_to=0'0, roll_forward_to=35'10))", 2023-11-05T08:33:13.395 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:49.071402+0000", 2023-11-05T08:33:13.395 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 24.288329595, 2023-11-05T08:33:13.395 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0050718050000000004, 2023-11-05T08:33:13.395 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:13.395 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:13.396 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:13.396 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.396 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:13.396 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.071402+0000", 2023-11-05T08:33:13.396 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.396 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.397 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.397 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:13.397 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.071402+0000", 2023-11-05T08:33:13.397 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.397 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.397 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.398 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:13.398 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.071403+0000", 2023-11-05T08:33:13.398 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.7599999999999998e-07 2023-11-05T08:33:13.398 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.398 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.398 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:13.399 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.071436+0000", 2023-11-05T08:33:13.399 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.3207999999999998e-05 2023-11-05T08:33:13.399 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.399 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.399 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:13.399 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.071453+0000", 2023-11-05T08:33:13.400 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.7042e-05 2023-11-05T08:33:13.400 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.400 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.400 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:13.400 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.071467+0000", 2023-11-05T08:33:13.401 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4384e-05 2023-11-05T08:33:13.401 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.401 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.401 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:13.401 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.071663+0000", 2023-11-05T08:33:13.401 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00019627300000000001 2023-11-05T08:33:13.402 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.402 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.402 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:13.402 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.071693+0000", 2023-11-05T08:33:13.402 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.9663e-05 2023-11-05T08:33:13.402 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.403 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.403 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:13.403 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.076421+0000", 2023-11-05T08:33:13.403 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.004727467 2023-11-05T08:33:13.403 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.404 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.404 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:13.404 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.076474+0000", 2023-11-05T08:33:13.404 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.3392e-05 2023-11-05T08:33:13.405 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.405 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:13.405 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.405 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.405 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.405 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.ds6 36/30 ECSubWrite(tid=21, reqid=client.4691.0:234, at_version=36'11, trim_to=0'0, roll_forward_to=34'8))", 2023-11-05T08:33:13.406 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:49.071464+0000", 2023-11-05T08:33:13.406 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 24.288267638000001, 2023-11-05T08:33:13.406 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0050888599999999997, 2023-11-05T08:33:13.407 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:13.407 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:13.407 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:13.407 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.407 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:13.408 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.071464+0000", 2023-11-05T08:33:13.408 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.408 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.408 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.408 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:13.409 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.071464+0000", 2023-11-05T08:33:13.409 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.409 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.409 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.409 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:13.410 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.071465+0000", 2023-11-05T08:33:13.410 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.2600000000000001e-07 2023-11-05T08:33:13.410 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.410 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.410 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:13.411 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.071490+0000", 2023-11-05T08:33:13.411 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.4879999999999999e-05 2023-11-05T08:33:13.411 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.411 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.411 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:13.412 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.071504+0000", 2023-11-05T08:33:13.412 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4484000000000001e-05 2023-11-05T08:33:13.412 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.412 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.412 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:13.412 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.071519+0000", 2023-11-05T08:33:13.413 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4878999999999999e-05 2023-11-05T08:33:13.413 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.413 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.413 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:13.413 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.072586+0000", 2023-11-05T08:33:13.414 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0010669290000000001 2023-11-05T08:33:13.414 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.414 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.414 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:13.414 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.072878+0000", 2023-11-05T08:33:13.415 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00029156000000000001 2023-11-05T08:33:13.415 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.415 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.415 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:13.415 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.076495+0000", 2023-11-05T08:33:13.415 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0036176649999999999 2023-11-05T08:33:13.416 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.416 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.416 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:13.416 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.076553+0000", 2023-11-05T08:33:13.416 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.7537000000000002e-05 2023-11-05T08:33:13.416 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.417 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:13.417 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.417 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.417 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.417 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:287 3.11s0 3:88a306be:::smithi09039252-16:head [append 0~770048] snapc 8=[8,7,6,5,3] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:33:13.418 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:53.182407+0000", 2023-11-05T08:33:13.418 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 20.177325008, 2023-11-05T08:33:13.418 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.011495974, 2023-11-05T08:33:13.418 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:13.418 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:13.419 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:13.419 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:13.419 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:13.419 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 287 2023-11-05T08:33:13.419 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.419 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:13.420 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.420 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:13.420 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.182407+0000", 2023-11-05T08:33:13.420 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.420 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.421 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.421 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:13.421 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.182407+0000", 2023-11-05T08:33:13.421 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.421 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.421 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.422 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:13.422 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.182409+0000", 2023-11-05T08:33:13.422 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.3590000000000002e-06 2023-11-05T08:33:13.422 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.422 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.423 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:13.423 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.183780+0000", 2023-11-05T08:33:13.423 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00137108 2023-11-05T08:33:13.423 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.423 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.424 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:13.424 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.183800+0000", 2023-11-05T08:33:13.424 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.9819000000000001e-05 2023-11-05T08:33:13.424 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.424 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.424 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:13.425 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.183810+0000", 2023-11-05T08:33:13.425 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.6940000000000007e-06 2023-11-05T08:33:13.425 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.425 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.425 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:13.426 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.183873+0000", 2023-11-05T08:33:13.426 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.3536999999999998e-05 2023-11-05T08:33:13.426 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.426 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.426 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for peered", 2023-11-05T08:33:13.427 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.183877+0000", 2023-11-05T08:33:13.427 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.1779999999999996e-06 2023-11-05T08:33:13.427 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.427 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.427 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:13.428 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.171366+0000", 2023-11-05T08:33:13.428 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.98748831999999997 2023-11-05T08:33:13.428 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.428 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.428 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:13.429 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.171750+0000", 2023-11-05T08:33:13.429 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00038434800000000001 2023-11-05T08:33:13.429 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.429 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.429 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:13.429 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.174288+0000", 2023-11-05T08:33:13.430 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.002537957 2023-11-05T08:33:13.430 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.430 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.430 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:13.430 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.181565+0000", 2023-11-05T08:33:13.431 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0072769530000000001 2023-11-05T08:33:13.431 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.431 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.431 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:33:13.431 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.193786+0000", 2023-11-05T08:33:13.432 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.012221037000000001 2023-11-05T08:33:13.432 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.432 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.432 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:13.432 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.193903+0000", 2023-11-05T08:33:13.433 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000116692 2023-11-05T08:33:13.433 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.433 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:13.433 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.433 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.434 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.434 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:288 3.11s0 3:88a306be:::smithi09039252-16:head [append 0~376832] snapc 8=[8,7,6,5,3] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:33:13.434 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:53.186355+0000", 2023-11-05T08:33:13.434 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 20.173377229, 2023-11-05T08:33:13.434 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0078397800000001, 2023-11-05T08:33:13.435 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:13.435 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:13.435 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:13.435 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:13.435 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:13.436 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 288 2023-11-05T08:33:13.436 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.436 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:13.436 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.436 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:13.437 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186355+0000", 2023-11-05T08:33:13.437 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.437 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.437 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.437 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:13.438 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186355+0000", 2023-11-05T08:33:13.438 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.438 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.438 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.439 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:13.439 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186355+0000", 2023-11-05T08:33:13.439 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.9400000000000004e-07 2023-11-05T08:33:13.439 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.439 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.440 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:13.440 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186649+0000", 2023-11-05T08:33:13.440 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00029353199999999999 2023-11-05T08:33:13.440 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.440 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.441 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:13.441 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186662+0000", 2023-11-05T08:33:13.441 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.2728000000000001e-05 2023-11-05T08:33:13.441 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.441 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.442 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:13.442 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186667+0000", 2023-11-05T08:33:13.442 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.8270000000000003e-06 2023-11-05T08:33:13.442 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.442 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.443 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:13.443 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186742+0000", 2023-11-05T08:33:13.443 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.4328999999999999e-05 2023-11-05T08:33:13.443 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.443 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.444 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for peered", 2023-11-05T08:33:13.444 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186744+0000", 2023-11-05T08:33:13.444 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.266e-06 2023-11-05T08:33:13.444 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.444 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.445 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:13.445 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.176861+0000", 2023-11-05T08:33:13.445 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.990116577 2023-11-05T08:33:13.445 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.445 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.445 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:13.446 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.177014+0000", 2023-11-05T08:33:13.446 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000153661 2023-11-05T08:33:13.446 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.446 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.446 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:13.447 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.178253+0000", 2023-11-05T08:33:13.447 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0012384399999999999 2023-11-05T08:33:13.447 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.447 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.447 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:13.448 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.181931+0000", 2023-11-05T08:33:13.448 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0036777730000000001 2023-11-05T08:33:13.448 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.448 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.448 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:33:13.449 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.194093+0000", 2023-11-05T08:33:13.449 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.012162885999999999 2023-11-05T08:33:13.449 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.449 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.449 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:13.450 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.194194+0000", 2023-11-05T08:33:13.450 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00010086699999999999 2023-11-05T08:33:13.450 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.450 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:13.450 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.451 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.451 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.451 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:289 3.11s0 3:88a306be:::smithi09039252-16:head [setxattr _header (56) in=7b] snapc 8=[8,7,6,5,3] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:33:13.451 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:53.186682+0000", 2023-11-05T08:33:13.451 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 20.173049882000001, 2023-11-05T08:33:13.452 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.008829143, 2023-11-05T08:33:13.452 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:13.452 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:13.452 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:13.452 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:13.453 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:13.453 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 289 2023-11-05T08:33:13.453 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.453 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:13.453 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.453 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:13.454 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186682+0000", 2023-11-05T08:33:13.454 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.454 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.454 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.455 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:13.455 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186682+0000", 2023-11-05T08:33:13.455 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.455 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.455 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.456 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:13.456 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186683+0000", 2023-11-05T08:33:13.456 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.8999999999999996e-07 2023-11-05T08:33:13.456 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.456 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.456 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:13.457 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186687+0000", 2023-11-05T08:33:13.457 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.933e-06 2023-11-05T08:33:13.457 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.457 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.457 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:13.458 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186697+0000", 2023-11-05T08:33:13.458 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.6120000000000005e-06 2023-11-05T08:33:13.458 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.458 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.458 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:13.459 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186704+0000", 2023-11-05T08:33:13.459 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.8929999999999999e-06 2023-11-05T08:33:13.459 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.459 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.459 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:13.460 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186817+0000", 2023-11-05T08:33:13.460 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000113229 2023-11-05T08:33:13.460 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.460 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.460 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for peered", 2023-11-05T08:33:13.461 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186819+0000", 2023-11-05T08:33:13.461 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.57e-06 2023-11-05T08:33:13.461 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.461 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.461 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:13.462 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.179928+0000", 2023-11-05T08:33:13.462 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.993109099 2023-11-05T08:33:13.462 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.462 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.462 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:13.463 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.180084+0000", 2023-11-05T08:33:13.463 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00015647 2023-11-05T08:33:13.463 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.463 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.463 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:13.464 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.181012+0000", 2023-11-05T08:33:13.464 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00092730299999999998 2023-11-05T08:33:13.464 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.464 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.464 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:13.464 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.183961+0000", 2023-11-05T08:33:13.465 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0029494510000000001 2023-11-05T08:33:13.465 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.465 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.465 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:33:13.465 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.195373+0000", 2023-11-05T08:33:13.466 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.011411607000000001 2023-11-05T08:33:13.466 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.466 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.466 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:13.466 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.195511+0000", 2023-11-05T08:33:13.467 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000138286 2023-11-05T08:33:13.467 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.467 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:13.467 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.467 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.467 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.468 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:290 3.11s0 3:88a306be:::smithi09039252-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:33:13.468 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:53.186775+0000", 2023-11-05T08:33:13.468 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 20.172956397, 2023-11-05T08:33:13.468 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0117209439999999, 2023-11-05T08:33:13.468 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:13.469 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "started", 2023-11-05T08:33:13.469 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:13.469 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:13.469 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:13.469 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 290 2023-11-05T08:33:13.470 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.470 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:13.470 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.470 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:13.470 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186775+0000", 2023-11-05T08:33:13.471 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.471 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.471 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.471 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:13.471 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186775+0000", 2023-11-05T08:33:13.471 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.472 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.472 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.472 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:13.472 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186776+0000", 2023-11-05T08:33:13.472 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.7999999999999993e-07 2023-11-05T08:33:13.473 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.473 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.473 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:13.473 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186780+0000", 2023-11-05T08:33:13.473 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.9249999999999997e-06 2023-11-05T08:33:13.473 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.474 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.474 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:13.474 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186792+0000", 2023-11-05T08:33:13.474 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.1841999999999999e-05 2023-11-05T08:33:13.474 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.475 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.475 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:13.475 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186800+0000", 2023-11-05T08:33:13.475 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.1559999999999995e-06 2023-11-05T08:33:13.475 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.476 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.476 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:13.476 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186855+0000", 2023-11-05T08:33:13.476 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.4892999999999998e-05 2023-11-05T08:33:13.476 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.477 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.477 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for peered", 2023-11-05T08:33:13.477 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186856+0000", 2023-11-05T08:33:13.477 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.1999999999999999e-07 2023-11-05T08:33:13.477 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.478 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.478 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:13.478 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.181764+0000", 2023-11-05T08:33:13.478 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.99490820300000005 2023-11-05T08:33:13.478 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.479 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.479 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for rw locks", 2023-11-05T08:33:13.479 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.181917+0000", 2023-11-05T08:33:13.479 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00015282300000000001 2023-11-05T08:33:13.479 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.479 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.480 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:13.480 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.196083+0000", 2023-11-05T08:33:13.480 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.014166247999999999 2023-11-05T08:33:13.480 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.481 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.481 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:13.481 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.196295+0000", 2023-11-05T08:33:13.481 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00021158599999999999 2023-11-05T08:33:13.481 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.482 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.482 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:13.482 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.198496+0000", 2023-11-05T08:33:13.482 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0022016679999999999 2023-11-05T08:33:13.482 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.483 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:13.483 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.483 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.483 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.483 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 41/40 ECSubWriteReply(tid=65, last_complete=41'13, committed=1, applied=1))", 2023-11-05T08:33:13.484 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:54.177163+0000", 2023-11-05T08:33:13.484 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 19.182568764999999, 2023-11-05T08:33:13.484 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0044112250000000004, 2023-11-05T08:33:13.484 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:13.484 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:13.485 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:13.485 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.485 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:13.485 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.177163+0000", 2023-11-05T08:33:13.485 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.486 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.486 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.486 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:13.486 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.177163+0000", 2023-11-05T08:33:13.486 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.487 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.487 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.487 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:13.487 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.177165+0000", 2023-11-05T08:33:13.487 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.716e-06 2023-11-05T08:33:13.487 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.488 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.488 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:13.488 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.177174+0000", 2023-11-05T08:33:13.488 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.9639999999999992e-06 2023-11-05T08:33:13.488 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.488 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.489 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:13.489 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.177198+0000", 2023-11-05T08:33:13.489 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.3849000000000002e-05 2023-11-05T08:33:13.489 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.489 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.489 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:13.490 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.177229+0000", 2023-11-05T08:33:13.490 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.1677999999999997e-05 2023-11-05T08:33:13.490 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.490 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.490 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:13.491 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.179757+0000", 2023-11-05T08:33:13.491 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0025279130000000001 2023-11-05T08:33:13.491 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.491 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.491 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:13.492 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.181574+0000", 2023-11-05T08:33:13.492 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0018171050000000001 2023-11-05T08:33:13.492 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.492 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:13.492 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.492 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.493 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.493 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.14s2 41/40 ECSubWrite(tid=17, reqid=client.4691.0:278, at_version=41'14, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:13.493 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:54.185928+0000", 2023-11-05T08:33:13.493 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 19.173803425999999, 2023-11-05T08:33:13.493 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0044635960000000002, 2023-11-05T08:33:13.493 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:13.494 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:13.494 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:13.494 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.494 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:13.494 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.185928+0000", 2023-11-05T08:33:13.495 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.495 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.495 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.495 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:13.495 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.185928+0000", 2023-11-05T08:33:13.496 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.496 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.496 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.496 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:13.496 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.185930+0000", 2023-11-05T08:33:13.496 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.6330000000000001e-06 2023-11-05T08:33:13.497 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.497 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.497 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:13.497 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.186149+0000", 2023-11-05T08:33:13.497 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000218777 2023-11-05T08:33:13.498 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.498 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.498 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:13.498 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.186179+0000", 2023-11-05T08:33:13.498 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.0338999999999999e-05 2023-11-05T08:33:13.499 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.499 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.499 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:13.499 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.186194+0000", 2023-11-05T08:33:13.499 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4749000000000001e-05 2023-11-05T08:33:13.500 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.500 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.500 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:13.500 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.186347+0000", 2023-11-05T08:33:13.501 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00015294700000000001 2023-11-05T08:33:13.501 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.501 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.501 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:13.501 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.186365+0000", 2023-11-05T08:33:13.502 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.8048999999999999e-05 2023-11-05T08:33:13.502 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.502 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.502 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:13.502 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.190305+0000", 2023-11-05T08:33:13.503 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0039397820000000002 2023-11-05T08:33:13.503 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.503 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.503 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:13.503 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.190392+0000", 2023-11-05T08:33:13.504 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.7319999999999997e-05 2023-11-05T08:33:13.504 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.504 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:13.504 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.504 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.504 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.505 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDPGPush(3.14s6 47/46 [PushOp(3:292b6c2e:::smithi09039252-40:7, version: 39'9, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:292b6c2e:::smithi09039252-40:7@39'9, size: 1671168, copy_subset: [], clone_subset: {}, snapset: 8=[]:{7: [7, 6, 5, 3], 8: [8]}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2023-11-05T08:33:13.505 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:00.235899+0000", 2023-11-05T08:33:13.505 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 13.123832348000001, 2023-11-05T08:33:13.505 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.003707288, 2023-11-05T08:33:13.506 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:13.506 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:13.506 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:13.506 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.506 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:13.507 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.235899+0000", 2023-11-05T08:33:13.507 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.507 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.507 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.507 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:13.508 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.235899+0000", 2023-11-05T08:33:13.508 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.508 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.508 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.508 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:13.508 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.235901+0000", 2023-11-05T08:33:13.509 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.6199999999999999e-06 2023-11-05T08:33:13.509 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.509 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.509 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:13.509 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.236430+0000", 2023-11-05T08:33:13.510 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00052928100000000004 2023-11-05T08:33:13.510 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.510 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.510 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:13.510 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.236500+0000", 2023-11-05T08:33:13.511 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.9247999999999998e-05 2023-11-05T08:33:13.511 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.511 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.511 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:13.511 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.236639+0000", 2023-11-05T08:33:13.511 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00013896500000000001 2023-11-05T08:33:13.512 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.512 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.512 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:13.512 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.236929+0000", 2023-11-05T08:33:13.512 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000290069 2023-11-05T08:33:13.513 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.513 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.513 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:13.513 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.239607+0000", 2023-11-05T08:33:13.513 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0026781050000000001 2023-11-05T08:33:13.514 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.514 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:13.514 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.514 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.514 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.515 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:332 3.1s0 3:863368a9:::smithi09039252-6:head [read 0~98304,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2023-11-05T08:33:13.515 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:04.226233+0000", 2023-11-05T08:33:13.515 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 9.1334992160000006, 2023-11-05T08:33:13.515 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0054114870000000004, 2023-11-05T08:33:13.515 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:13.515 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "started", 2023-11-05T08:33:13.516 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:13.516 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:13.516 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:13.516 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 332 2023-11-05T08:33:13.516 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.516 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:13.517 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.517 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:13.517 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226233+0000", 2023-11-05T08:33:13.517 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.517 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.518 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.518 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:13.518 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226233+0000", 2023-11-05T08:33:13.518 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.518 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.518 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.519 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:13.519 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226235+0000", 2023-11-05T08:33:13.519 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.0140000000000001e-06 2023-11-05T08:33:13.519 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.519 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.519 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:13.520 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226246+0000", 2023-11-05T08:33:13.520 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0992000000000001e-05 2023-11-05T08:33:13.520 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.520 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.520 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:13.520 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226271+0000", 2023-11-05T08:33:13.521 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.5732999999999999e-05 2023-11-05T08:33:13.521 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.521 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.521 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:13.521 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226290+0000", 2023-11-05T08:33:13.522 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.8793000000000001e-05 2023-11-05T08:33:13.522 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.522 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.522 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:13.522 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226800+0000", 2023-11-05T08:33:13.522 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00050957599999999997 2023-11-05T08:33:13.523 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.523 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.523 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:13.523 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.227301+0000", 2023-11-05T08:33:13.523 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00050143699999999998 2023-11-05T08:33:13.524 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.524 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.524 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:13.524 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.231644+0000", 2023-11-05T08:33:13.524 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0043429419999999998 2023-11-05T08:33:13.525 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.525 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:13.525 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.525 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.525 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.525 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:333 3.1s0 3:863368a9:::smithi09039252-6:head [read 0~98304,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2023-11-05T08:33:13.526 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:04.226324+0000", 2023-11-05T08:33:13.526 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 9.1334073149999995, 2023-11-05T08:33:13.526 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0064759040000000002, 2023-11-05T08:33:13.526 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:13.526 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "started", 2023-11-05T08:33:13.527 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:13.527 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:13.527 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:13.527 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 333 2023-11-05T08:33:13.527 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.528 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:13.528 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.528 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:13.528 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226324+0000", 2023-11-05T08:33:13.528 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.529 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.529 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.529 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:13.529 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226324+0000", 2023-11-05T08:33:13.529 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.529 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.530 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.530 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:13.530 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226326+0000", 2023-11-05T08:33:13.530 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.243e-06 2023-11-05T08:33:13.530 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.531 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.531 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:13.531 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226335+0000", 2023-11-05T08:33:13.531 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.2560000000000008e-06 2023-11-05T08:33:13.531 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.532 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.532 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:13.532 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226356+0000", 2023-11-05T08:33:13.532 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.1406000000000001e-05 2023-11-05T08:33:13.532 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.533 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.533 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:13.533 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226372+0000", 2023-11-05T08:33:13.533 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.5923e-05 2023-11-05T08:33:13.533 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.533 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.534 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:13.534 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.228444+0000", 2023-11-05T08:33:13.534 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0020712930000000001 2023-11-05T08:33:13.534 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.534 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.535 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:13.535 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.228711+0000", 2023-11-05T08:33:13.535 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00026721699999999999 2023-11-05T08:33:13.535 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.535 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.536 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:13.536 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.232800+0000", 2023-11-05T08:33:13.536 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0040895660000000002 2023-11-05T08:33:13.536 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.536 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:13.536 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.537 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.537 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.537 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:334 3.1s0 3:863368a9:::smithi09039252-6:head [read 0~98304,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2023-11-05T08:33:13.537 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:04.226390+0000", 2023-11-05T08:33:13.537 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 9.1333421139999995, 2023-11-05T08:33:13.538 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.017587591999999999, 2023-11-05T08:33:13.538 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:13.538 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "started", 2023-11-05T08:33:13.538 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:13.538 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:13.539 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:13.539 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 334 2023-11-05T08:33:13.539 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.539 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:13.539 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.540 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:13.540 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226390+0000", 2023-11-05T08:33:13.540 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.540 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.540 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.540 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:13.541 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226390+0000", 2023-11-05T08:33:13.541 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.541 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.541 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.541 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:13.542 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226390+0000", 2023-11-05T08:33:13.542 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.1999999999999998e-07 2023-11-05T08:33:13.542 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.542 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.542 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:13.543 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226396+0000", 2023-11-05T08:33:13.543 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.3229999999999997e-06 2023-11-05T08:33:13.543 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.543 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.543 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:13.544 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226409+0000", 2023-11-05T08:33:13.544 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.3096e-05 2023-11-05T08:33:13.544 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.544 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.544 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:13.545 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226417+0000", 2023-11-05T08:33:13.545 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.1300000000000001e-06 2023-11-05T08:33:13.545 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.545 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.545 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:13.545 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.233238+0000", 2023-11-05T08:33:13.546 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0068212999999999998 2023-11-05T08:33:13.546 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.546 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.546 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:13.546 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.233474+0000", 2023-11-05T08:33:13.547 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000235286 2023-11-05T08:33:13.547 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.547 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.547 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:13.547 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.243977+0000", 2023-11-05T08:33:13.548 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.010503637 2023-11-05T08:33:13.548 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.548 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:13.548 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.549 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.549 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.549 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:365 3.bs0 3:d0f842ed:::smithi09039252-18:head [read 0~294912,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2023-11-05T08:33:13.549 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:08.269803+0000", 2023-11-05T08:33:13.549 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 5.0899282640000001, 2023-11-05T08:33:13.550 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.005142087, 2023-11-05T08:33:13.550 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:13.550 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "started", 2023-11-05T08:33:13.550 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:13.550 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:13.551 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:13.551 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 365 2023-11-05T08:33:13.551 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.551 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:13.551 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.552 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:13.552 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.269803+0000", 2023-11-05T08:33:13.552 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.552 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.552 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.553 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:13.553 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.269803+0000", 2023-11-05T08:33:13.553 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.553 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.553 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.554 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:13.554 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.269806+0000", 2023-11-05T08:33:13.554 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.407e-06 2023-11-05T08:33:13.554 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.554 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.555 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:13.555 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.269818+0000", 2023-11-05T08:33:13.555 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.2549e-05 2023-11-05T08:33:13.555 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.555 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.556 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:13.556 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.269842+0000", 2023-11-05T08:33:13.556 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.3949000000000001e-05 2023-11-05T08:33:13.556 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.556 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.556 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:13.557 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.269853+0000", 2023-11-05T08:33:13.557 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0492000000000001e-05 2023-11-05T08:33:13.557 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.557 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.557 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:13.558 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.269949+0000", 2023-11-05T08:33:13.558 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.6068000000000002e-05 2023-11-05T08:33:13.558 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.558 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.558 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "op must wait for map", 2023-11-05T08:33:13.559 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.269962+0000", 2023-11-05T08:33:13.559 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.2598e-05 2023-11-05T08:33:13.559 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.559 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.559 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:13.560 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.272720+0000", 2023-11-05T08:33:13.560 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.002758758 2023-11-05T08:33:13.560 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.560 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.560 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:13.561 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.272857+0000", 2023-11-05T08:33:13.561 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00013655300000000001 2023-11-05T08:33:13.561 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.561 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.561 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:13.561 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.274946+0000", 2023-11-05T08:33:13.562 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0020887129999999999 2023-11-05T08:33:13.562 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.562 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:13.562 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.562 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.563 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.563 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:366 3.bs0 3:d0f842ed:::smithi09039252-18:head [read 0~294912,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2023-11-05T08:33:13.563 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:08.269879+0000", 2023-11-05T08:33:13.563 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 5.0898527629999997, 2023-11-05T08:33:13.563 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0056755659999999999, 2023-11-05T08:33:13.564 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:13.564 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "started", 2023-11-05T08:33:13.564 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:13.564 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:13.564 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:13.565 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 366 2023-11-05T08:33:13.565 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.565 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:13.565 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.565 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:13.566 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.269879+0000", 2023-11-05T08:33:13.566 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.566 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.566 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.566 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:13.566 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.269879+0000", 2023-11-05T08:33:13.567 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.567 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.567 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.567 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:13.567 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.269879+0000", 2023-11-05T08:33:13.568 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.5600000000000001e-07 2023-11-05T08:33:13.568 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.568 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.568 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:13.568 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.269882+0000", 2023-11-05T08:33:13.569 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.9849999999999998e-06 2023-11-05T08:33:13.569 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.569 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.569 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:13.569 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.269887+0000", 2023-11-05T08:33:13.570 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.9910000000000002e-06 2023-11-05T08:33:13.570 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.570 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.570 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:13.570 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.269891+0000", 2023-11-05T08:33:13.571 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.4690000000000002e-06 2023-11-05T08:33:13.571 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.571 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.571 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:13.571 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.270043+0000", 2023-11-05T08:33:13.572 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00015178500000000001 2023-11-05T08:33:13.572 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.572 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.572 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:33:13.572 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.270048+0000", 2023-11-05T08:33:13.573 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.3569999999999997e-06 2023-11-05T08:33:13.573 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.573 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.573 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:13.573 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.273302+0000", 2023-11-05T08:33:13.574 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0032543630000000001 2023-11-05T08:33:13.574 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.574 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.574 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:13.575 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.273413+0000", 2023-11-05T08:33:13.575 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000110215 2023-11-05T08:33:13.575 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.575 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.575 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:13.576 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.275555+0000", 2023-11-05T08:33:13.576 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.002142045 2023-11-05T08:33:13.576 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.576 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:13.576 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.577 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.577 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.577 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:367 3.bs0 3:d0f842ed:::smithi09039252-18:head [read 0~294912,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2023-11-05T08:33:13.577 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:08.269934+0000", 2023-11-05T08:33:13.577 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 5.0897974890000004, 2023-11-05T08:33:13.578 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.007346366, 2023-11-05T08:33:13.578 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:13.578 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "started", 2023-11-05T08:33:13.578 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:13.578 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:13.579 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:13.579 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 367 2023-11-05T08:33:13.579 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.579 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:13.579 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.580 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:13.580 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.269934+0000", 2023-11-05T08:33:13.580 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.580 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.580 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.581 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:13.581 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.269934+0000", 2023-11-05T08:33:13.581 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.581 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.581 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.582 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:13.582 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.269935+0000", 2023-11-05T08:33:13.582 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.03e-07 2023-11-05T08:33:13.582 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.582 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.583 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:13.583 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.269937+0000", 2023-11-05T08:33:13.583 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.283e-06 2023-11-05T08:33:13.583 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.583 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.584 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:13.584 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.269942+0000", 2023-11-05T08:33:13.584 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.9219999999999996e-06 2023-11-05T08:33:13.584 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.584 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.585 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:13.585 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.269945+0000", 2023-11-05T08:33:13.585 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.3519999999999998e-06 2023-11-05T08:33:13.585 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.585 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.585 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:13.586 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.270155+0000", 2023-11-05T08:33:13.586 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00021031899999999999 2023-11-05T08:33:13.586 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.586 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.587 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:33:13.587 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.270181+0000", 2023-11-05T08:33:13.587 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.5415999999999999e-05 2023-11-05T08:33:13.587 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.587 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.588 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:13.588 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.275639+0000", 2023-11-05T08:33:13.588 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00545788 2023-11-05T08:33:13.588 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.588 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.589 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:13.589 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.275752+0000", 2023-11-05T08:33:13.589 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000113107 2023-11-05T08:33:13.589 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.589 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.590 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:13.590 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.277281+0000", 2023-11-05T08:33:13.590 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001528784 2023-11-05T08:33:13.590 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.590 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:13.591 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.591 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.591 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.591 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.16s2 56/45 ECSubWrite(tid=63, reqid=client.4691.0:369, at_version=56'20, trim_to=0'0, roll_forward_to=51'19))", 2023-11-05T08:33:13.591 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:09.292106+0000", 2023-11-05T08:33:13.592 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 4.0676259559999997, 2023-11-05T08:33:13.592 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0033543240000000001, 2023-11-05T08:33:13.592 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:13.592 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:13.592 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:13.592 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.593 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:13.593 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.292106+0000", 2023-11-05T08:33:13.593 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.593 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.593 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.594 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:13.594 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.292106+0000", 2023-11-05T08:33:13.594 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.594 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.595 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.595 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:13.595 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.292106+0000", 2023-11-05T08:33:13.595 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.7999999999999995e-07 2023-11-05T08:33:13.595 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.595 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.596 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:13.596 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.292622+0000", 2023-11-05T08:33:13.596 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00051605499999999996 2023-11-05T08:33:13.596 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.597 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.597 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:13.597 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.292651+0000", 2023-11-05T08:33:13.597 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.8966e-05 2023-11-05T08:33:13.597 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.598 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.598 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:13.598 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.292673+0000", 2023-11-05T08:33:13.598 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.1999999999999999e-05 2023-11-05T08:33:13.598 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.599 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.599 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:13.599 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.292794+0000", 2023-11-05T08:33:13.599 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000120699 2023-11-05T08:33:13.599 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.600 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.600 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:13.600 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.292816+0000", 2023-11-05T08:33:13.600 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.1656999999999999e-05 2023-11-05T08:33:13.600 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.601 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.601 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:13.601 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.295384+0000", 2023-11-05T08:33:13.601 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0025678459999999999 2023-11-05T08:33:13.601 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.602 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.602 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:13.602 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.295460+0000", 2023-11-05T08:33:13.602 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.6520999999999995e-05 2023-11-05T08:33:13.602 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.603 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:13.603 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.603 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.603 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.603 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.16s2 56/45 ECSubWrite(tid=64, reqid=client.4691.0:370, at_version=56'21, trim_to=0'0, roll_forward_to=51'19))", 2023-11-05T08:33:13.603 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:09.297961+0000", 2023-11-05T08:33:13.604 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 4.0617708209999996, 2023-11-05T08:33:13.604 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0041275649999999997, 2023-11-05T08:33:13.604 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:13.604 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:13.604 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:13.605 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.605 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:13.605 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297961+0000", 2023-11-05T08:33:13.605 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.605 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.605 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.606 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:13.606 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297961+0000", 2023-11-05T08:33:13.606 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:13.606 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.606 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.606 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:13.607 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297963+0000", 2023-11-05T08:33:13.607 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.9800000000000001e-06 2023-11-05T08:33:13.607 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.607 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.607 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:13.608 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.298297+0000", 2023-11-05T08:33:13.608 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00033361099999999999 2023-11-05T08:33:13.608 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.608 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.608 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:13.608 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.298342+0000", 2023-11-05T08:33:13.609 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.5475000000000001e-05 2023-11-05T08:33:13.609 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.609 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.609 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:13.609 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.298367+0000", 2023-11-05T08:33:13.609 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.4579e-05 2023-11-05T08:33:13.610 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.610 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.610 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:13.610 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.298557+0000", 2023-11-05T08:33:13.610 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000190538 2023-11-05T08:33:13.611 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.611 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.611 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:13.611 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.298586+0000", 2023-11-05T08:33:13.611 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.9352999999999998e-05 2023-11-05T08:33:13.611 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.612 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.612 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:13.612 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.302002+0000", 2023-11-05T08:33:13.612 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0034157089999999998 2023-11-05T08:33:13.612 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:13.612 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:13.613 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:13.613 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.302088+0000", 2023-11-05T08:33:13.613 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.632e-05 2023-11-05T08:33:13.613 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.613 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:13.614 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.614 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:13.614 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:13.614 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:33:13.615 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10: {} 2023-11-05T08:33:13.616 INFO:teuthology.orchestra.run.smithi002.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-11-05T08:33:13.617 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2023-11-05T08:33:13.617 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11: {} 2023-11-05T08:33:13.618 INFO:teuthology.orchestra.run.smithi002.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-11-05T08:33:13.618 INFO:teuthology.orchestra.run.smithi002.stdout:osd.12: {} 2023-11-05T08:33:13.619 INFO:teuthology.orchestra.run.smithi002.stderr:osd.12: osd_enable_op_tracker = 'true' 2023-11-05T08:33:13.619 INFO:teuthology.orchestra.run.smithi002.stdout:osd.13: {} 2023-11-05T08:33:13.619 INFO:teuthology.orchestra.run.smithi002.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-11-05T08:33:13.620 INFO:teuthology.orchestra.run.smithi002.stdout:osd.14: {} 2023-11-05T08:33:13.620 INFO:teuthology.orchestra.run.smithi002.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-11-05T08:33:13.620 INFO:teuthology.orchestra.run.smithi002.stdout:osd.15: {} 2023-11-05T08:33:13.621 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:13.613+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:13.621 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-05T08:33:13.622 INFO:teuthology.orchestra.run.smithi002.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-11-05T08:33:13.622 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-05T08:33:13.716 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:33:13.714+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:33:13.788 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:33:13.789 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [], 2023-11-05T08:33:13.789 INFO:teuthology.orchestra.run.smithi002.stdout: "num_ops": 0 2023-11-05T08:33:13.789 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:33:13.804 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2023-11-05T08:33:13.817 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:33:13.814+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:33:13.918 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:33:13.912+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:33:13.931 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:33:13.931 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [], 2023-11-05T08:33:13.931 INFO:teuthology.orchestra.run.smithi002.stdout: "complaint_time": 30, 2023-11-05T08:33:13.931 INFO:teuthology.orchestra.run.smithi002.stdout: "num_blocked_ops": 0 2023-11-05T08:33:13.931 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:33:13.945 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2023-11-05T08:33:14.018 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-05T08:33:14.076 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:33:14.077 INFO:teuthology.orchestra.run.smithi002.stdout: "size": 20, 2023-11-05T08:33:14.077 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 600, 2023-11-05T08:33:14.077 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [ 2023-11-05T08:33:14.077 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.077 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.ds5 36/30 ECSubWrite(tid=22, reqid=client.4691.0:235, at_version=36'12, trim_to=0'0, roll_forward_to=34'8))", 2023-11-05T08:33:14.078 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:49.077540+0000", 2023-11-05T08:33:14.078 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 24.996719168999999, 2023-11-05T08:33:14.078 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.003308727, 2023-11-05T08:33:14.078 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:14.078 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:14.079 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:14.079 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.079 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:14.079 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.077540+0000", 2023-11-05T08:33:14.079 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.080 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.080 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.080 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:14.080 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.077540+0000", 2023-11-05T08:33:14.080 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.081 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.081 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.081 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:14.081 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.077541+0000", 2023-11-05T08:33:14.081 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4449999999999999e-06 2023-11-05T08:33:14.082 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.082 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.082 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:14.082 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.077573+0000", 2023-11-05T08:33:14.082 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.1736999999999998e-05 2023-11-05T08:33:14.083 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.083 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.083 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:14.083 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.078750+0000", 2023-11-05T08:33:14.083 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0011769739999999999 2023-11-05T08:33:14.084 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.084 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.084 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:14.084 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.078764+0000", 2023-11-05T08:33:14.084 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4225e-05 2023-11-05T08:33:14.085 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.085 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.085 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:14.085 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.078832+0000", 2023-11-05T08:33:14.085 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.7923999999999995e-05 2023-11-05T08:33:14.086 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.086 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.086 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:14.086 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.078843+0000", 2023-11-05T08:33:14.086 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0763e-05 2023-11-05T08:33:14.087 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.087 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.087 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:14.087 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.080792+0000", 2023-11-05T08:33:14.087 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00194875 2023-11-05T08:33:14.088 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.088 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.088 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:14.088 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.080849+0000", 2023-11-05T08:33:14.088 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.6909e-05 2023-11-05T08:33:14.089 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.089 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:14.089 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.089 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.089 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.090 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.ds5 36/30 ECSubWrite(tid=23, reqid=osd.7.0:23, at_version=36'14, trim_to=0'0, roll_forward_to=34'8))", 2023-11-05T08:33:14.090 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:49.078802+0000", 2023-11-05T08:33:14.090 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 24.995457447, 2023-11-05T08:33:14.090 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0035119359999999998, 2023-11-05T08:33:14.090 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:14.091 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:14.091 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:14.091 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.091 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:14.091 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.078802+0000", 2023-11-05T08:33:14.092 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.092 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.092 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.092 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:14.092 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.078802+0000", 2023-11-05T08:33:14.093 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.093 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.093 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.093 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:14.093 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.078802+0000", 2023-11-05T08:33:14.094 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.5899999999999996e-07 2023-11-05T08:33:14.094 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.094 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.094 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:14.094 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.078834+0000", 2023-11-05T08:33:14.094 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.1696999999999998e-05 2023-11-05T08:33:14.095 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.095 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.095 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:14.095 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.078845+0000", 2023-11-05T08:33:14.095 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0992000000000001e-05 2023-11-05T08:33:14.096 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.096 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.096 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:14.096 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.078851+0000", 2023-11-05T08:33:14.096 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.1360000000000002e-06 2023-11-05T08:33:14.096 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.097 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.097 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:14.097 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.079095+0000", 2023-11-05T08:33:14.097 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00024419600000000002 2023-11-05T08:33:14.097 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.097 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.098 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:14.098 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.079105+0000", 2023-11-05T08:33:14.098 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0341e-05 2023-11-05T08:33:14.098 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.099 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.099 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:14.099 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.082270+0000", 2023-11-05T08:33:14.099 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0031646719999999999 2023-11-05T08:33:14.099 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.100 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.100 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:14.100 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.082313+0000", 2023-11-05T08:33:14.100 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.3343000000000002e-05 2023-11-05T08:33:14.100 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.101 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:14.101 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.101 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.101 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.101 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.5s1 36/30 ECSubWrite(tid=26, reqid=osd.15.0:25, at_version=0'0, trim_to=0'0, roll_forward_to=36'15))", 2023-11-05T08:33:14.102 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:49.089242+0000", 2023-11-05T08:33:14.102 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 24.985017300999999, 2023-11-05T08:33:14.102 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.003392642, 2023-11-05T08:33:14.102 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:14.103 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:14.103 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:14.103 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.103 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:14.103 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.089242+0000", 2023-11-05T08:33:14.104 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.104 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.104 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.104 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:14.104 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.089242+0000", 2023-11-05T08:33:14.105 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.105 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.105 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.105 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:14.105 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.089242+0000", 2023-11-05T08:33:14.106 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.2500000000000004e-07 2023-11-05T08:33:14.106 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.106 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.106 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:14.106 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.089260+0000", 2023-11-05T08:33:14.107 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.7244999999999998e-05 2023-11-05T08:33:14.107 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.107 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.107 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:14.107 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.089273+0000", 2023-11-05T08:33:14.108 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.3668e-05 2023-11-05T08:33:14.108 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.108 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.108 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:14.108 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.089283+0000", 2023-11-05T08:33:14.109 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.5060000000000001e-06 2023-11-05T08:33:14.109 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.109 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.109 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:14.109 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.089343+0000", 2023-11-05T08:33:14.109 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.0588000000000003e-05 2023-11-05T08:33:14.110 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.110 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.110 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:14.110 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.089354+0000", 2023-11-05T08:33:14.110 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0688e-05 2023-11-05T08:33:14.111 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.111 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.111 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:14.111 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.092588+0000", 2023-11-05T08:33:14.111 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0032339439999999999 2023-11-05T08:33:14.111 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.112 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.112 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:14.112 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.092634+0000", 2023-11-05T08:33:14.112 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.6177999999999997e-05 2023-11-05T08:33:14.112 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.113 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:14.113 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.113 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.113 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.113 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:278 3.14s0 3:292b6c2e:::smithi09039252-40:head [copy-from ver 10] snapc 8=[8,7,6,5,3] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:33:14.114 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:53.143346+0000", 2023-11-05T08:33:14.114 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 20.930913175000001, 2023-11-05T08:33:14.114 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0524983919999999, 2023-11-05T08:33:14.114 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:14.114 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:14.115 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:14.115 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:14.115 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:14.115 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 278 2023-11-05T08:33:14.115 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.116 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:14.116 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.116 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:14.116 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143346+0000", 2023-11-05T08:33:14.116 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.117 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.117 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.117 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:14.117 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143346+0000", 2023-11-05T08:33:14.117 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.118 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.118 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.118 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:14.118 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143347+0000", 2023-11-05T08:33:14.118 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4470000000000001e-06 2023-11-05T08:33:14.119 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.119 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.119 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:14.119 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143353+0000", 2023-11-05T08:33:14.119 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.7790000000000001e-06 2023-11-05T08:33:14.120 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.120 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.120 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:14.120 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143367+0000", 2023-11-05T08:33:14.120 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4321000000000001e-05 2023-11-05T08:33:14.121 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.121 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.121 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:14.121 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143374+0000", 2023-11-05T08:33:14.121 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.8029999999999999e-06 2023-11-05T08:33:14.121 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.122 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.122 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:14.122 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143446+0000", 2023-11-05T08:33:14.122 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.2080000000000001e-05 2023-11-05T08:33:14.122 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.123 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.123 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for peered", 2023-11-05T08:33:14.123 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143450+0000", 2023-11-05T08:33:14.123 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.5449999999999999e-06 2023-11-05T08:33:14.123 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.124 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.124 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:14.124 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.169316+0000", 2023-11-05T08:33:14.124 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.025866494 2023-11-05T08:33:14.124 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.124 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.125 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:14.125 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.169711+0000", 2023-11-05T08:33:14.125 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00039465700000000003 2023-11-05T08:33:14.125 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.125 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.126 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:14.126 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.185825+0000", 2023-11-05T08:33:14.126 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.016113624 2023-11-05T08:33:14.126 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.126 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.127 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:14.127 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.190579+0000", 2023-11-05T08:33:14.127 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0047547370000000002 2023-11-05T08:33:14.127 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.127 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.128 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:33:14.128 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.195484+0000", 2023-11-05T08:33:14.128 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0049050200000000004 2023-11-05T08:33:14.128 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.128 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.129 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:14.129 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.195844+0000", 2023-11-05T08:33:14.129 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000359885 2023-11-05T08:33:14.129 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.129 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:14.130 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.130 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.130 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.130 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:279 3.14s0 3:292b6c2e:::smithi09039252-40:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:33:14.131 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:53.143390+0000", 2023-11-05T08:33:14.131 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 20.930869098999999, 2023-11-05T08:33:14.131 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0469768799999999, 2023-11-05T08:33:14.131 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:14.131 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "started", 2023-11-05T08:33:14.132 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:14.132 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:14.132 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:14.132 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 279 2023-11-05T08:33:14.132 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.132 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:14.133 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.133 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:14.133 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143390+0000", 2023-11-05T08:33:14.133 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.133 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.134 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.134 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:14.134 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143390+0000", 2023-11-05T08:33:14.134 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.134 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.134 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.135 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:14.135 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143390+0000", 2023-11-05T08:33:14.135 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.5999999999999999e-07 2023-11-05T08:33:14.135 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.135 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.136 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:14.136 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143392+0000", 2023-11-05T08:33:14.136 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.0209999999999998e-06 2023-11-05T08:33:14.136 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.136 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.137 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:14.137 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143399+0000", 2023-11-05T08:33:14.137 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.6869999999999997e-06 2023-11-05T08:33:14.137 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.137 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.137 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:14.138 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143405+0000", 2023-11-05T08:33:14.138 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.0619999999999997e-06 2023-11-05T08:33:14.138 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.138 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.138 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:14.138 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143483+0000", 2023-11-05T08:33:14.139 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.8268000000000003e-05 2023-11-05T08:33:14.139 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.139 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.139 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for peered", 2023-11-05T08:33:14.139 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143486+0000", 2023-11-05T08:33:14.139 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.322e-06 2023-11-05T08:33:14.140 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.140 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.140 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:14.140 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.172078+0000", 2023-11-05T08:33:14.140 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0285927589999999 2023-11-05T08:33:14.140 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.141 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.141 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for blocked object", 2023-11-05T08:33:14.141 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.172240+0000", 2023-11-05T08:33:14.141 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00016178899999999999 2023-11-05T08:33:14.141 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.141 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.142 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:14.142 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.189404+0000", 2023-11-05T08:33:14.142 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.017164220000000001 2023-11-05T08:33:14.142 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.142 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.142 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:14.143 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.189710+0000", 2023-11-05T08:33:14.143 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000305617 2023-11-05T08:33:14.143 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.143 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.143 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:14.144 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.190367+0000", 2023-11-05T08:33:14.144 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00065677499999999996 2023-11-05T08:33:14.144 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.144 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:14.144 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.144 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.145 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.145 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es4 41/30 ECSubWrite(tid=40, reqid=osd.2.0:40, at_version=41'27, trim_to=0'0, roll_forward_to=40'25))", 2023-11-05T08:33:14.145 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:54.120687+0000", 2023-11-05T08:33:14.145 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 19.953572314999999, 2023-11-05T08:33:14.145 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0097705859999999995, 2023-11-05T08:33:14.145 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:14.146 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:14.146 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:14.146 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.146 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:14.146 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120687+0000", 2023-11-05T08:33:14.147 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.147 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.147 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.147 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:14.147 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120687+0000", 2023-11-05T08:33:14.148 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.148 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.148 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.148 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:14.148 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120688+0000", 2023-11-05T08:33:14.149 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.64e-06 2023-11-05T08:33:14.149 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.149 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.149 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:14.149 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120741+0000", 2023-11-05T08:33:14.150 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.2274000000000001e-05 2023-11-05T08:33:14.150 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.150 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.150 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:14.150 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120770+0000", 2023-11-05T08:33:14.151 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.9768999999999999e-05 2023-11-05T08:33:14.151 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.151 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.151 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:14.152 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120795+0000", 2023-11-05T08:33:14.152 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.4805000000000001e-05 2023-11-05T08:33:14.152 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.152 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.152 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:14.153 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.124189+0000", 2023-11-05T08:33:14.153 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0033935279999999998 2023-11-05T08:33:14.153 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.153 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.153 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:14.154 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.124214+0000", 2023-11-05T08:33:14.154 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.5823e-05 2023-11-05T08:33:14.154 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.154 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.155 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:14.155 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.130330+0000", 2023-11-05T08:33:14.155 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0061156190000000001 2023-11-05T08:33:14.155 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.155 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.156 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:14.156 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.130457+0000", 2023-11-05T08:33:14.156 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00012712800000000001 2023-11-05T08:33:14.156 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.156 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:14.156 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.157 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.157 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.157 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es4 42/30 ECSubWrite(tid=43, reqid=osd.2.0:43, at_version=42'29, trim_to=0'0, roll_forward_to=41'27))", 2023-11-05T08:33:14.157 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:55.140218+0000", 2023-11-05T08:33:14.157 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 18.934040474, 2023-11-05T08:33:14.158 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0038969510000000001, 2023-11-05T08:33:14.158 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:14.158 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:14.158 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:14.158 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.159 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:14.159 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.140218+0000", 2023-11-05T08:33:14.159 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.159 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.159 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.160 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:14.160 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.140218+0000", 2023-11-05T08:33:14.160 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.160 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.160 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.161 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:14.161 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.140219+0000", 2023-11-05T08:33:14.161 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.7000000000000005e-07 2023-11-05T08:33:14.161 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.161 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.162 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:14.162 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.140283+0000", 2023-11-05T08:33:14.162 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.3738000000000005e-05 2023-11-05T08:33:14.162 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.162 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.163 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:14.163 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.140317+0000", 2023-11-05T08:33:14.163 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.4498000000000001e-05 2023-11-05T08:33:14.163 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.163 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.163 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:14.164 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.140342+0000", 2023-11-05T08:33:14.164 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.4742000000000001e-05 2023-11-05T08:33:14.164 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.164 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.164 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:14.165 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.140849+0000", 2023-11-05T08:33:14.165 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00050743999999999995 2023-11-05T08:33:14.165 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.165 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.165 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:14.166 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.140876+0000", 2023-11-05T08:33:14.166 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.6798999999999999e-05 2023-11-05T08:33:14.166 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.166 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.166 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:14.167 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.144042+0000", 2023-11-05T08:33:14.167 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0031655870000000001 2023-11-05T08:33:14.167 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.167 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.167 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:14.168 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.144115+0000", 2023-11-05T08:33:14.168 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.3577000000000004e-05 2023-11-05T08:33:14.168 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.168 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:14.168 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.169 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.169 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.169 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpReadReply(3.14s0 47/46 ECSubReadReply(tid=22, attrs_read=0))", 2023-11-05T08:33:14.169 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:00.232364+0000", 2023-11-05T08:33:14.169 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 13.841894501000001, 2023-11-05T08:33:14.169 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0035804690000000002, 2023-11-05T08:33:14.170 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:14.170 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:14.170 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:14.170 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.170 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:14.170 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.232364+0000", 2023-11-05T08:33:14.171 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.171 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.171 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.171 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:14.171 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.232364+0000", 2023-11-05T08:33:14.172 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.172 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.172 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.172 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:14.172 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.232366+0000", 2023-11-05T08:33:14.172 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.393e-06 2023-11-05T08:33:14.173 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.173 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.173 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:14.173 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.232910+0000", 2023-11-05T08:33:14.173 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000543938 2023-11-05T08:33:14.173 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.174 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.174 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:14.174 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.232931+0000", 2023-11-05T08:33:14.174 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.1599000000000001e-05 2023-11-05T08:33:14.174 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.175 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.175 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:14.175 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.232961+0000", 2023-11-05T08:33:14.175 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.9949000000000001e-05 2023-11-05T08:33:14.175 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.176 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.176 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:14.176 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.233087+0000", 2023-11-05T08:33:14.176 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00012517499999999999 2023-11-05T08:33:14.176 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.176 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.177 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:14.177 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.235945+0000", 2023-11-05T08:33:14.177 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.002858415 2023-11-05T08:33:14.177 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.177 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:14.177 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.178 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.178 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.178 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDPGPush(3.14s5 47/46 [PushOp(3:292b6c2e:::smithi09039252-40:7, version: 39'9, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:292b6c2e:::smithi09039252-40:7@39'9, size: 1671168, copy_subset: [], clone_subset: {}, snapset: 8=[]:{7: [7, 6, 5, 3], 8: [8]}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2023-11-05T08:33:14.178 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:00.235709+0000", 2023-11-05T08:33:14.178 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 13.838550221, 2023-11-05T08:33:14.179 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0033694910000000001, 2023-11-05T08:33:14.179 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:14.179 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:14.179 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:14.179 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.179 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:14.180 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.235709+0000", 2023-11-05T08:33:14.180 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.180 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.180 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.180 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:14.180 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.235709+0000", 2023-11-05T08:33:14.181 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.181 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.181 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.181 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:14.181 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.235709+0000", 2023-11-05T08:33:14.182 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.182 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.182 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.182 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:14.182 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.235709+0000", 2023-11-05T08:33:14.183 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.183 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.183 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.183 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:14.183 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.235844+0000", 2023-11-05T08:33:14.184 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000135079 2023-11-05T08:33:14.184 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.184 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.184 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:14.184 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.235948+0000", 2023-11-05T08:33:14.185 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000104583 2023-11-05T08:33:14.185 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.185 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.185 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:14.185 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.236166+0000", 2023-11-05T08:33:14.186 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00021716699999999999 2023-11-05T08:33:14.186 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.186 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.186 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:14.186 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.239078+0000", 2023-11-05T08:33:14.187 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0029126619999999999 2023-11-05T08:33:14.187 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.187 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:14.187 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.187 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.188 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.188 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDPGPushReply(3.14s0 47/46 [PushReplyOp(3:292b6c2e:::smithi09039252-40:7)])", 2023-11-05T08:33:14.188 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:00.249065+0000", 2023-11-05T08:33:14.188 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 13.825194234, 2023-11-05T08:33:14.188 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0037023899999999998, 2023-11-05T08:33:14.189 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:14.189 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:14.189 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:14.189 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.189 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:14.190 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.249065+0000", 2023-11-05T08:33:14.190 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.190 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.190 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.190 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:14.191 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.249065+0000", 2023-11-05T08:33:14.191 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.191 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.191 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.191 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:14.192 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.249066+0000", 2023-11-05T08:33:14.192 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.73e-06 2023-11-05T08:33:14.192 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.192 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.192 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:14.192 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.249080+0000", 2023-11-05T08:33:14.193 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.3196e-05 2023-11-05T08:33:14.193 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.193 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.193 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:14.193 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.249117+0000", 2023-11-05T08:33:14.194 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.7419000000000002e-05 2023-11-05T08:33:14.194 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.194 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.194 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:14.194 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.249134+0000", 2023-11-05T08:33:14.195 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.7079e-05 2023-11-05T08:33:14.195 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.195 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.195 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:14.195 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.249248+0000", 2023-11-05T08:33:14.196 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000113929 2023-11-05T08:33:14.196 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.196 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.196 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:14.196 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.252767+0000", 2023-11-05T08:33:14.197 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.003519037 2023-11-05T08:33:14.197 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.197 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:14.197 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.197 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.198 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.198 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpReadReply(3.14s0 47/46 ECSubReadReply(tid=24, attrs_read=0))", 2023-11-05T08:33:14.198 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:00.258165+0000", 2023-11-05T08:33:14.198 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 13.81609357, 2023-11-05T08:33:14.198 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.003129115, 2023-11-05T08:33:14.199 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:14.199 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:14.199 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:14.199 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.199 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:14.200 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.258165+0000", 2023-11-05T08:33:14.200 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.200 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.200 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.200 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:14.201 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.258165+0000", 2023-11-05T08:33:14.201 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.201 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.201 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.201 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:14.202 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.258167+0000", 2023-11-05T08:33:14.202 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.8339999999999999e-06 2023-11-05T08:33:14.202 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.202 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.202 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:14.202 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.258520+0000", 2023-11-05T08:33:14.203 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00035257399999999998 2023-11-05T08:33:14.203 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.203 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.203 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:14.203 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.258549+0000", 2023-11-05T08:33:14.204 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.9235999999999999e-05 2023-11-05T08:33:14.204 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.204 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.204 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:14.204 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.258562+0000", 2023-11-05T08:33:14.205 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.2921999999999999e-05 2023-11-05T08:33:14.205 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.205 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.205 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:14.205 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.258675+0000", 2023-11-05T08:33:14.206 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000113219 2023-11-05T08:33:14.206 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.206 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.206 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:14.206 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.261295+0000", 2023-11-05T08:33:14.207 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00261933 2023-11-05T08:33:14.207 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.207 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:14.207 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.207 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.208 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.208 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDPGPushReply(3.14s0 47/46 [PushReplyOp(3:292b6c2e:::smithi09039252-40:head)])", 2023-11-05T08:33:14.208 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:00.292862+0000", 2023-11-05T08:33:14.208 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 13.781397408, 2023-11-05T08:33:14.208 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0063631419999999996, 2023-11-05T08:33:14.209 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:14.209 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:14.209 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:14.209 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.209 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:14.210 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.292862+0000", 2023-11-05T08:33:14.210 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.210 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.210 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.210 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:14.211 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.292862+0000", 2023-11-05T08:33:14.211 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.211 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.211 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.211 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:14.211 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.292862+0000", 2023-11-05T08:33:14.212 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.0200000000000001e-07 2023-11-05T08:33:14.212 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.212 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.212 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:14.212 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.292867+0000", 2023-11-05T08:33:14.213 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.7829999999999998e-06 2023-11-05T08:33:14.213 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.213 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.213 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:14.213 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.292877+0000", 2023-11-05T08:33:14.214 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0312000000000001e-05 2023-11-05T08:33:14.214 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.214 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.214 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:14.214 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.292889+0000", 2023-11-05T08:33:14.214 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.2019e-05 2023-11-05T08:33:14.215 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.215 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.215 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:14.215 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.293399+0000", 2023-11-05T08:33:14.215 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00050963099999999997 2023-11-05T08:33:14.216 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.216 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.216 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:14.216 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.299225+0000", 2023-11-05T08:33:14.216 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0058256949999999997 2023-11-05T08:33:14.217 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.217 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:14.217 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.217 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.217 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.217 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.16s3 51/45 ECSubWrite(tid=58, reqid=osd.11.0:58, at_version=51'19, trim_to=0'0, roll_forward_to=49'17))", 2023-11-05T08:33:14.218 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:04.227640+0000", 2023-11-05T08:33:14.218 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 9.8466186180000008, 2023-11-05T08:33:14.218 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.003497839, 2023-11-05T08:33:14.218 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:14.218 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:14.219 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:14.219 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.219 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:14.219 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.227640+0000", 2023-11-05T08:33:14.219 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.220 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.220 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.220 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:14.220 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.227640+0000", 2023-11-05T08:33:14.220 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.220 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.221 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.221 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:14.221 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.227643+0000", 2023-11-05T08:33:14.221 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.2129999999999998e-06 2023-11-05T08:33:14.221 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.222 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.222 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:14.222 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.227704+0000", 2023-11-05T08:33:14.222 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.1748000000000006e-05 2023-11-05T08:33:14.222 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.223 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.223 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:14.223 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.227744+0000", 2023-11-05T08:33:14.223 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.9598999999999998e-05 2023-11-05T08:33:14.223 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.224 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.224 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:14.224 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.227769+0000", 2023-11-05T08:33:14.224 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.5182999999999999e-05 2023-11-05T08:33:14.224 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.225 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.225 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:14.225 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.227954+0000", 2023-11-05T08:33:14.225 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000185167 2023-11-05T08:33:14.225 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.226 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.226 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:14.226 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.227977+0000", 2023-11-05T08:33:14.226 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.2586e-05 2023-11-05T08:33:14.226 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.227 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.227 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:14.227 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.231028+0000", 2023-11-05T08:33:14.227 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0030515400000000002 2023-11-05T08:33:14.227 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.227 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.228 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:14.228 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.231138+0000", 2023-11-05T08:33:14.228 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00010980300000000001 2023-11-05T08:33:14.228 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.228 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:14.228 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.229 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.229 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.229 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.16s3 56/45 ECSubWrite(tid=63, reqid=client.4691.0:369, at_version=56'20, trim_to=0'0, roll_forward_to=51'19))", 2023-11-05T08:33:14.229 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:09.293412+0000", 2023-11-05T08:33:14.229 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 4.7808465130000002, 2023-11-05T08:33:14.230 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0049197820000000001, 2023-11-05T08:33:14.230 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:14.230 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:14.230 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:14.230 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.231 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:14.231 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.293412+0000", 2023-11-05T08:33:14.231 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.231 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.231 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.232 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:14.232 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.293412+0000", 2023-11-05T08:33:14.232 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.232 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.232 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.233 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:14.233 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.293413+0000", 2023-11-05T08:33:14.233 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.0299999999999998e-07 2023-11-05T08:33:14.233 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.233 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.234 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:14.234 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.293961+0000", 2023-11-05T08:33:14.234 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00054824799999999999 2023-11-05T08:33:14.234 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.234 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.235 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:14.235 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.293997+0000", 2023-11-05T08:33:14.235 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.5991999999999997e-05 2023-11-05T08:33:14.235 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.235 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.236 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:14.236 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.294022+0000", 2023-11-05T08:33:14.236 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.4865999999999999e-05 2023-11-05T08:33:14.236 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.236 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.237 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:14.237 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.294234+0000", 2023-11-05T08:33:14.237 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00021178399999999999 2023-11-05T08:33:14.237 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.237 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.238 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:14.238 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.294264+0000", 2023-11-05T08:33:14.238 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.0309000000000001e-05 2023-11-05T08:33:14.238 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.238 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.239 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:14.239 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.298238+0000", 2023-11-05T08:33:14.239 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0039737959999999999 2023-11-05T08:33:14.239 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.239 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.239 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:14.240 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.298332+0000", 2023-11-05T08:33:14.240 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.3984e-05 2023-11-05T08:33:14.240 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.240 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:14.240 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.240 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.241 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.241 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:374 3.14s0 3:292b6c2e:::smithi09039252-40:head [copy-from ver 2] snapc 15=[7] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2023-11-05T08:33:14.241 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:09.297419+0000", 2023-11-05T08:33:14.241 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 4.7768397650000001, 2023-11-05T08:33:14.241 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.03004254, 2023-11-05T08:33:14.242 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:14.242 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:14.242 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:14.242 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:14.242 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:14.243 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 374 2023-11-05T08:33:14.243 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.243 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:14.243 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.243 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:14.244 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297419+0000", 2023-11-05T08:33:14.244 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.244 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.244 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.244 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:14.245 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297419+0000", 2023-11-05T08:33:14.245 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.245 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.245 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.245 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:14.246 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297422+0000", 2023-11-05T08:33:14.246 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.717e-06 2023-11-05T08:33:14.246 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.246 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.247 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:14.247 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297437+0000", 2023-11-05T08:33:14.247 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4793e-05 2023-11-05T08:33:14.247 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.247 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.248 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:14.248 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297468+0000", 2023-11-05T08:33:14.248 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.1519000000000001e-05 2023-11-05T08:33:14.248 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.248 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.249 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:14.249 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297490+0000", 2023-11-05T08:33:14.249 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.1651999999999999e-05 2023-11-05T08:33:14.249 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.249 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.249 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:14.250 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.300337+0000", 2023-11-05T08:33:14.250 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0028475219999999999 2023-11-05T08:33:14.250 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.250 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.251 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:14.251 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.300717+0000", 2023-11-05T08:33:14.251 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00037937600000000001 2023-11-05T08:33:14.251 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.251 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.252 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:14.252 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.319194+0000", 2023-11-05T08:33:14.252 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.018477303 2023-11-05T08:33:14.252 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.252 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.252 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:14.253 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.322493+0000", 2023-11-05T08:33:14.253 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0032987250000000002 2023-11-05T08:33:14.253 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.253 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.253 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:33:14.253 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.326714+0000", 2023-11-05T08:33:14.254 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0042208319999999999 2023-11-05T08:33:14.254 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.254 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.254 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:14.254 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.327462+0000", 2023-11-05T08:33:14.255 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00074810100000000004 2023-11-05T08:33:14.255 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.255 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:14.255 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.255 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.256 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.256 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:375 3.14s0 3:292b6c2e:::smithi09039252-40:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e56)", 2023-11-05T08:33:14.256 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:09.297530+0000", 2023-11-05T08:33:14.256 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 4.7767289379999998, 2023-11-05T08:33:14.256 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.024433857999999999, 2023-11-05T08:33:14.257 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:14.257 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "started", 2023-11-05T08:33:14.257 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:14.257 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:14.257 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:14.258 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 375 2023-11-05T08:33:14.258 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.258 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:14.258 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.258 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:14.258 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297530+0000", 2023-11-05T08:33:14.259 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.259 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.259 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.259 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:14.260 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297530+0000", 2023-11-05T08:33:14.260 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.260 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.260 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.260 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:14.260 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297531+0000", 2023-11-05T08:33:14.261 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.48e-06 2023-11-05T08:33:14.261 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.261 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.261 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:14.262 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297540+0000", 2023-11-05T08:33:14.262 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.49e-06 2023-11-05T08:33:14.262 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.262 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.262 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:14.263 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297566+0000", 2023-11-05T08:33:14.263 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.6152e-05 2023-11-05T08:33:14.263 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.263 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.263 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:14.264 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297583+0000", 2023-11-05T08:33:14.264 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.6676000000000001e-05 2023-11-05T08:33:14.264 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.264 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.264 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:14.265 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.301232+0000", 2023-11-05T08:33:14.265 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.003649209 2023-11-05T08:33:14.265 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.265 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.265 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for blocked object", 2023-11-05T08:33:14.266 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.301367+0000", 2023-11-05T08:33:14.266 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00013454600000000001 2023-11-05T08:33:14.266 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.266 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.266 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:14.267 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.321294+0000", 2023-11-05T08:33:14.267 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.019927261000000002 2023-11-05T08:33:14.267 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.267 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.267 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:14.267 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.321567+0000", 2023-11-05T08:33:14.268 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00027297299999999998 2023-11-05T08:33:14.268 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.268 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.268 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:14.268 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.321964+0000", 2023-11-05T08:33:14.269 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00039707099999999998 2023-11-05T08:33:14.269 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.269 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:14.269 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.269 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.270 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.270 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.16s3 56/45 ECSubWrite(tid=64, reqid=client.4691.0:370, at_version=56'21, trim_to=0'0, roll_forward_to=51'19))", 2023-11-05T08:33:14.270 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:09.299985+0000", 2023-11-05T08:33:14.270 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 4.7742741219999996, 2023-11-05T08:33:14.270 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.003179776, 2023-11-05T08:33:14.271 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:14.271 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:14.271 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:14.271 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.271 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:14.272 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.299985+0000", 2023-11-05T08:33:14.272 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.272 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.272 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.272 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:14.272 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.299985+0000", 2023-11-05T08:33:14.273 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.273 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.273 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.273 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:14.273 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.299987+0000", 2023-11-05T08:33:14.273 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.86e-06 2023-11-05T08:33:14.274 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.274 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.274 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:14.274 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.300267+0000", 2023-11-05T08:33:14.274 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00028013499999999999 2023-11-05T08:33:14.275 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.275 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.275 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:14.275 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.300291+0000", 2023-11-05T08:33:14.275 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.4009999999999999e-05 2023-11-05T08:33:14.275 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.276 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.276 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:14.276 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.300305+0000", 2023-11-05T08:33:14.276 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4236000000000001e-05 2023-11-05T08:33:14.276 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.276 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.277 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:14.277 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.300411+0000", 2023-11-05T08:33:14.277 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000105834 2023-11-05T08:33:14.277 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.277 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.277 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:14.278 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.300443+0000", 2023-11-05T08:33:14.278 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.2279e-05 2023-11-05T08:33:14.278 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.278 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.278 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:14.279 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.303065+0000", 2023-11-05T08:33:14.279 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0026221579999999999 2023-11-05T08:33:14.279 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.279 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.279 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:14.279 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.303165+0000", 2023-11-05T08:33:14.280 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.9264000000000004e-05 2023-11-05T08:33:14.280 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.280 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:14.280 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.280 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.281 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.281 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.16s3 56/45 ECSubWrite(tid=66, reqid=client.4691.0:372, at_version=56'23, trim_to=0'0, roll_forward_to=56'20))", 2023-11-05T08:33:14.281 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:09.309455+0000", 2023-11-05T08:33:14.281 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 4.7648038939999999, 2023-11-05T08:33:14.281 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0032721600000000001, 2023-11-05T08:33:14.282 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:14.282 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:14.282 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:14.282 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.282 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:14.283 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.309455+0000", 2023-11-05T08:33:14.283 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.283 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.283 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.283 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:14.284 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.309455+0000", 2023-11-05T08:33:14.284 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.284 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.284 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.284 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:14.285 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.309456+0000", 2023-11-05T08:33:14.285 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.9299999999999996e-07 2023-11-05T08:33:14.285 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.285 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.285 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:14.286 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.309488+0000", 2023-11-05T08:33:14.286 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.1603000000000002e-05 2023-11-05T08:33:14.286 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.286 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.286 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:14.287 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.309508+0000", 2023-11-05T08:33:14.287 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.9993e-05 2023-11-05T08:33:14.287 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.287 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.287 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:14.288 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.309518+0000", 2023-11-05T08:33:14.288 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0842999999999999e-05 2023-11-05T08:33:14.288 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.288 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.288 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:14.289 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.310683+0000", 2023-11-05T08:33:14.289 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001164406 2023-11-05T08:33:14.289 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.289 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.289 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:14.290 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.310712+0000", 2023-11-05T08:33:14.290 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.9136e-05 2023-11-05T08:33:14.290 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.290 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.290 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:14.291 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.312642+0000", 2023-11-05T08:33:14.291 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0019297680000000001 2023-11-05T08:33:14.291 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.291 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.291 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:14.292 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.312727+0000", 2023-11-05T08:33:14.292 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.5518000000000004e-05 2023-11-05T08:33:14.292 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.292 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:14.293 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.293 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.293 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.293 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.14s5 56/48 ECSubWrite(tid=32, reqid=client.4691.0:374, at_version=56'19, trim_to=0'0, roll_forward_to=49'18))", 2023-11-05T08:33:14.293 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:09.318983+0000", 2023-11-05T08:33:14.294 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 4.7552762309999999, 2023-11-05T08:33:14.294 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00432556, 2023-11-05T08:33:14.294 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:14.294 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:14.294 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:14.295 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.295 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:14.295 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.318983+0000", 2023-11-05T08:33:14.295 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.295 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.295 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.296 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:14.296 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.318983+0000", 2023-11-05T08:33:14.296 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.296 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.296 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.297 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:14.297 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.318983+0000", 2023-11-05T08:33:14.297 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.297 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.297 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.298 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:14.298 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.318983+0000", 2023-11-05T08:33:14.298 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.298 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.298 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.299 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:14.299 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.319029+0000", 2023-11-05T08:33:14.299 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.6675999999999998e-05 2023-11-05T08:33:14.299 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.299 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.300 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:14.300 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.319054+0000", 2023-11-05T08:33:14.300 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.4669e-05 2023-11-05T08:33:14.300 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.300 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.300 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:14.301 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.319246+0000", 2023-11-05T08:33:14.301 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00019165499999999999 2023-11-05T08:33:14.301 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.301 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.301 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:14.302 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.319278+0000", 2023-11-05T08:33:14.302 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.2744999999999999e-05 2023-11-05T08:33:14.302 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.302 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.302 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:14.303 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.323256+0000", 2023-11-05T08:33:14.303 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0039773589999999998 2023-11-05T08:33:14.303 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.303 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.303 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:14.304 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.323308+0000", 2023-11-05T08:33:14.304 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.2456000000000001e-05 2023-11-05T08:33:14.304 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.304 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:14.304 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.305 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.305 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.305 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es4 57/30 ECSubWrite(tid=60, reqid=client.4691.0:380, at_version=57'45, trim_to=0'0, roll_forward_to=49'43))", 2023-11-05T08:33:14.305 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:10.285514+0000", 2023-11-05T08:33:14.305 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 3.7887453629999999, 2023-11-05T08:33:14.306 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.003439372, 2023-11-05T08:33:14.306 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:14.306 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:14.306 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:14.306 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.307 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:14.307 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285514+0000", 2023-11-05T08:33:14.307 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.307 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.308 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.308 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:14.308 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285514+0000", 2023-11-05T08:33:14.308 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.308 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.309 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.309 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:14.309 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285514+0000", 2023-11-05T08:33:14.309 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.9000000000000002e-07 2023-11-05T08:33:14.309 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.309 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.310 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:14.310 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285558+0000", 2023-11-05T08:33:14.310 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.4229e-05 2023-11-05T08:33:14.310 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.310 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.311 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:14.311 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285584+0000", 2023-11-05T08:33:14.311 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.5706000000000001e-05 2023-11-05T08:33:14.311 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.311 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.312 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:14.312 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285602+0000", 2023-11-05T08:33:14.312 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.7863000000000002e-05 2023-11-05T08:33:14.312 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.312 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.313 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:14.313 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285821+0000", 2023-11-05T08:33:14.313 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000219416 2023-11-05T08:33:14.313 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.313 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.313 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:14.314 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285862+0000", 2023-11-05T08:33:14.314 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.0923000000000001e-05 2023-11-05T08:33:14.314 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.314 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.314 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:14.315 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.288883+0000", 2023-11-05T08:33:14.315 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0030210430000000002 2023-11-05T08:33:14.315 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.315 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.315 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:14.316 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.288953+0000", 2023-11-05T08:33:14.316 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.9801999999999993e-05 2023-11-05T08:33:14.316 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.316 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:14.316 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.317 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.317 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:14.317 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:33:14.318 INFO:tasks.rados.rados.0.smithi090.stdout:163: done (1 left) 2023-11-05T08:33:14.318 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 43 current snap is 15 2023-11-05T08:33:14.318 INFO:tasks.rados.rados.0.smithi090.stdout:164: seq_num 82 ranges {1851392=622592,2473984=475136,2949120=294912} 2023-11-05T08:33:14.320 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2023-11-05T08:33:14.320 INFO:teuthology.orchestra.run.smithi002.stderr:nodeep-scrub is unset 2023-11-05T08:33:14.320 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:33:14.314+0000 7fdc5cd58640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:33:14.355 INFO:tasks.rados.rados.0.smithi090.stdout:164: writing smithi09039252-43 from 1851392 to 2473984 tid 1 2023-11-05T08:33:14.363 INFO:tasks.rados.rados.0.smithi090.stdout:164: writing smithi09039252-43 from 2473984 to 2949120 tid 2 2023-11-05T08:33:14.368 INFO:tasks.rados.rados.0.smithi090.stdout:164: writing smithi09039252-43 from 2949120 to 3244032 tid 3 2023-11-05T08:33:14.368 INFO:tasks.rados.rados.0.smithi090.stdout:165: delete oid 42 current snap is 15 2023-11-05T08:33:14.377 INFO:tasks.rados.rados.0.smithi090.stdout:164: finishing write tid 1 to smithi09039252-43 2023-11-05T08:33:14.382 INFO:tasks.rados.rados.0.smithi090.stdout:164: finishing write tid 2 to smithi09039252-43 2023-11-05T08:33:14.383 INFO:tasks.rados.rados.0.smithi090.stdout:165: done (2 left) 2023-11-05T08:33:14.383 INFO:tasks.rados.rados.0.smithi090.stdout:166: delete oid 50 current snap is 15 2023-11-05T08:33:14.383 INFO:tasks.rados.rados.0.smithi090.stdout:164: finishing write tid 3 to smithi09039252-43 2023-11-05T08:33:14.386 INFO:tasks.rados.rados.0.smithi090.stdout:164: finishing write tid 4 to smithi09039252-43 2023-11-05T08:33:14.389 INFO:tasks.rados.rados.0.smithi090.stdout:166: done (2 left) 2023-11-05T08:33:14.390 INFO:tasks.rados.rados.0.smithi090.stdout:167: rollback oid 35 current snap is 15 2023-11-05T08:33:14.390 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 35 to 4 2023-11-05T08:33:14.390 INFO:tasks.rados.rados.0.smithi090.stdout:168: snap_create 2023-11-05T08:33:14.390 INFO:tasks.rados.rados.0.smithi090.stdout:164: finishing write tid 5 to smithi09039252-43 2023-11-05T08:33:14.390 INFO:tasks.rados.rados.0.smithi090.stdout:164: oid 43 updating version 0 to 26 2023-11-05T08:33:14.391 INFO:tasks.rados.rados.0.smithi090.stdout:164: oid 43 version 26 is already newer than 25 2023-11-05T08:33:14.391 INFO:tasks.rados.rados.0.smithi090.stdout:164: oid 43 version 26 is already newer than 23 2023-11-05T08:33:14.391 INFO:tasks.rados.rados.0.smithi090.stdout:164: oid 43 version 26 is already newer than 24 2023-11-05T08:33:14.391 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 43 v 26 (ObjNum 82 snap 15 seq_num 82) dirty exists 2023-11-05T08:33:14.391 INFO:tasks.rados.rados.0.smithi090.stdout:164: left oid 43 (ObjNum 82 snap 15 seq_num 82) 2023-11-05T08:33:14.400 INFO:tasks.rados.rados.0.smithi090.stdout:167: finishing rollback tid 0 to smithi09039252-35 2023-11-05T08:33:14.401 INFO:tasks.rados.rados.0.smithi090.stdout:167: finishing rollback tid 1 to smithi09039252-35 2023-11-05T08:33:14.402 INFO:tasks.rados.rados.0.smithi090.stdout:167: finishing rollback tid 2 to smithi09039252-35 2023-11-05T08:33:14.402 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 35 v 21 (ObjNum 34 snap 0 seq_num 34) dirty exists 2023-11-05T08:33:14.423 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:33:14.422+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:33:14.507 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:33:14.508 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [], 2023-11-05T08:33:14.508 INFO:teuthology.orchestra.run.smithi002.stdout: "num_ops": 0 2023-11-05T08:33:14.508 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:33:14.520 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:14.517+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:14.522 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2023-11-05T08:33:14.620 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:33:14.617+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:33:14.690 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:33:14.691 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [], 2023-11-05T08:33:14.691 INFO:teuthology.orchestra.run.smithi002.stdout: "complaint_time": 30, 2023-11-05T08:33:14.691 INFO:teuthology.orchestra.run.smithi002.stdout: "num_blocked_ops": 0 2023-11-05T08:33:14.691 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:33:14.705 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2023-11-05T08:33:14.721 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:33:14.718+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:33:14.822 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:33:14.817+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:33:14.835 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:33:14.835 INFO:teuthology.orchestra.run.smithi002.stdout: "size": 20, 2023-11-05T08:33:14.835 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 600, 2023-11-05T08:33:14.835 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [ 2023-11-05T08:33:14.836 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.836 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es6 36/30 ECSubWrite(tid=33, reqid=osd.2.0:33, at_version=36'21, trim_to=0'0, roll_forward_to=34'19))", 2023-11-05T08:33:14.836 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:49.082925+0000", 2023-11-05T08:33:14.836 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 25.749299542999999, 2023-11-05T08:33:14.836 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0014075069999999999, 2023-11-05T08:33:14.837 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:14.837 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:14.837 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:14.837 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.837 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:14.838 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.082925+0000", 2023-11-05T08:33:14.838 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.838 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.838 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.839 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:14.839 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.082925+0000", 2023-11-05T08:33:14.839 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.839 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.839 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.840 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:14.840 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.082925+0000", 2023-11-05T08:33:14.840 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.5400000000000002e-07 2023-11-05T08:33:14.840 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.840 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.841 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:14.841 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.082947+0000", 2023-11-05T08:33:14.841 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.2376999999999999e-05 2023-11-05T08:33:14.841 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.841 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.842 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:14.842 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.082961+0000", 2023-11-05T08:33:14.842 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.3910999999999999e-05 2023-11-05T08:33:14.842 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.843 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.843 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:14.843 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.082972+0000", 2023-11-05T08:33:14.843 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0732e-05 2023-11-05T08:33:14.843 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.844 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.844 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:14.844 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.083069+0000", 2023-11-05T08:33:14.844 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.6788999999999998e-05 2023-11-05T08:33:14.844 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.845 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.845 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:14.845 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.083083+0000", 2023-11-05T08:33:14.845 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4564e-05 2023-11-05T08:33:14.845 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.846 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.846 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:14.846 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.084287+0000", 2023-11-05T08:33:14.846 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001203771 2023-11-05T08:33:14.846 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.847 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.847 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:14.847 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.084332+0000", 2023-11-05T08:33:14.847 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.5108999999999998e-05 2023-11-05T08:33:14.847 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.848 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:14.848 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.848 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.848 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.848 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es6 36/30 ECSubWrite(tid=34, reqid=osd.2.0:33, at_version=0'0, trim_to=0'0, roll_forward_to=36'21))", 2023-11-05T08:33:14.849 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:49.089741+0000", 2023-11-05T08:33:14.849 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 25.742483856, 2023-11-05T08:33:14.849 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0017028290000000001, 2023-11-05T08:33:14.849 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:14.849 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:14.849 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:14.850 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.850 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:14.850 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.089741+0000", 2023-11-05T08:33:14.850 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.850 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.851 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.851 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:14.851 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.089741+0000", 2023-11-05T08:33:14.851 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.851 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.852 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.852 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:14.852 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.089741+0000", 2023-11-05T08:33:14.852 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.6700000000000002e-07 2023-11-05T08:33:14.853 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.853 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.853 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:14.853 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.089772+0000", 2023-11-05T08:33:14.853 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.01e-05 2023-11-05T08:33:14.854 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.854 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.854 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:14.854 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.089792+0000", 2023-11-05T08:33:14.855 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.0593000000000001e-05 2023-11-05T08:33:14.855 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.855 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.855 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:14.855 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.089804+0000", 2023-11-05T08:33:14.856 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.1657999999999999e-05 2023-11-05T08:33:14.856 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.856 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.856 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:14.857 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.089890+0000", 2023-11-05T08:33:14.857 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.6456e-05 2023-11-05T08:33:14.857 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.857 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.857 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:14.858 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.089906+0000", 2023-11-05T08:33:14.858 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.5608999999999999e-05 2023-11-05T08:33:14.858 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.858 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.859 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:14.859 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.091383+0000", 2023-11-05T08:33:14.859 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001477396 2023-11-05T08:33:14.859 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.859 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.860 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:14.860 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.091443+0000", 2023-11-05T08:33:14.860 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.0050000000000003e-05 2023-11-05T08:33:14.860 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.860 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:14.861 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.861 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.861 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.861 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es6 41/30 ECSubWrite(tid=40, reqid=osd.2.0:40, at_version=41'27, trim_to=0'0, roll_forward_to=40'25))", 2023-11-05T08:33:14.861 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:54.120785+0000", 2023-11-05T08:33:14.862 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 20.711439554999998, 2023-11-05T08:33:14.862 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0057023600000000001, 2023-11-05T08:33:14.862 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:14.862 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:14.862 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:14.863 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.863 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:14.863 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120785+0000", 2023-11-05T08:33:14.863 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.864 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.864 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.864 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:14.864 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120785+0000", 2023-11-05T08:33:14.864 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.865 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.865 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.865 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:14.865 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120786+0000", 2023-11-05T08:33:14.865 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.0999999999999998e-07 2023-11-05T08:33:14.866 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.866 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.866 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:14.866 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120850+0000", 2023-11-05T08:33:14.867 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.4727000000000006e-05 2023-11-05T08:33:14.867 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.867 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.867 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:14.867 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120899+0000", 2023-11-05T08:33:14.868 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.8905e-05 2023-11-05T08:33:14.868 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.868 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.868 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:14.868 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120925+0000", 2023-11-05T08:33:14.869 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.5831999999999999e-05 2023-11-05T08:33:14.869 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.869 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.869 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:14.869 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.121107+0000", 2023-11-05T08:33:14.870 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00018210899999999999 2023-11-05T08:33:14.870 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.870 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.870 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:14.871 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.121136+0000", 2023-11-05T08:33:14.871 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.9116e-05 2023-11-05T08:33:14.871 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.871 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.871 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:14.872 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.126383+0000", 2023-11-05T08:33:14.872 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0052464579999999999 2023-11-05T08:33:14.872 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.872 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.872 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:14.873 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.126487+0000", 2023-11-05T08:33:14.873 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000104503 2023-11-05T08:33:14.873 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.873 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:14.873 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.874 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.874 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.874 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es6 41/30 ECSubWrite(tid=41, reqid=osd.2.0:40, at_version=0'0, trim_to=0'0, roll_forward_to=41'27))", 2023-11-05T08:33:14.874 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:54.150296+0000", 2023-11-05T08:33:14.875 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 20.681928219, 2023-11-05T08:33:14.875 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.002204733, 2023-11-05T08:33:14.875 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:14.875 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:14.875 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:14.876 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.876 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:14.876 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.150296+0000", 2023-11-05T08:33:14.876 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.877 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.877 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.877 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:14.877 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.150296+0000", 2023-11-05T08:33:14.877 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.878 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.878 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.878 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:14.878 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.150298+0000", 2023-11-05T08:33:14.878 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.9530000000000002e-06 2023-11-05T08:33:14.879 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.879 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.879 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:14.879 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.150330+0000", 2023-11-05T08:33:14.879 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.1483000000000003e-05 2023-11-05T08:33:14.880 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.880 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.880 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:14.880 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.150368+0000", 2023-11-05T08:33:14.880 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.8194000000000001e-05 2023-11-05T08:33:14.881 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.881 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.881 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:14.881 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.150395+0000", 2023-11-05T08:33:14.881 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.6806000000000001e-05 2023-11-05T08:33:14.881 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.882 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.882 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:14.882 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.150620+0000", 2023-11-05T08:33:14.882 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00022512399999999999 2023-11-05T08:33:14.882 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.883 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.883 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:14.883 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.150652+0000", 2023-11-05T08:33:14.883 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.1912000000000002e-05 2023-11-05T08:33:14.883 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.884 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.884 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:14.884 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.152424+0000", 2023-11-05T08:33:14.884 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0017722339999999999 2023-11-05T08:33:14.884 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.885 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.885 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:14.885 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.152501+0000", 2023-11-05T08:33:14.885 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.7027000000000006e-05 2023-11-05T08:33:14.886 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.886 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:14.886 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.886 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.887 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.887 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es6 42/30 ECSubWrite(tid=43, reqid=osd.2.0:43, at_version=42'29, trim_to=0'0, roll_forward_to=41'27))", 2023-11-05T08:33:14.887 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:55.137493+0000", 2023-11-05T08:33:14.887 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 19.694731632, 2023-11-05T08:33:14.887 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0031512160000000001, 2023-11-05T08:33:14.888 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:14.888 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:14.888 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:14.888 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.888 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:14.889 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.137493+0000", 2023-11-05T08:33:14.889 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.889 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.889 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.890 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:14.890 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.137493+0000", 2023-11-05T08:33:14.890 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.890 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.890 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.891 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:14.891 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.137493+0000", 2023-11-05T08:33:14.891 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.7999999999999995e-07 2023-11-05T08:33:14.891 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.891 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.891 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:14.892 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.137559+0000", 2023-11-05T08:33:14.892 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.6098000000000003e-05 2023-11-05T08:33:14.892 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.892 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.892 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:14.893 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.137594+0000", 2023-11-05T08:33:14.893 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.4082000000000003e-05 2023-11-05T08:33:14.893 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.893 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.893 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:14.894 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.137620+0000", 2023-11-05T08:33:14.894 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.6271999999999999e-05 2023-11-05T08:33:14.894 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.894 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.894 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:14.895 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.137815+0000", 2023-11-05T08:33:14.895 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000195229 2023-11-05T08:33:14.895 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.895 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.895 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:14.896 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.137848+0000", 2023-11-05T08:33:14.896 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.2611999999999999e-05 2023-11-05T08:33:14.896 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.896 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.896 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:14.897 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.140476+0000", 2023-11-05T08:33:14.897 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0026284860000000002 2023-11-05T08:33:14.897 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.897 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.897 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:14.897 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.140644+0000", 2023-11-05T08:33:14.898 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00016785699999999999 2023-11-05T08:33:14.898 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.898 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:14.898 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.898 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.899 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.899 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es6 42/30 ECSubWrite(tid=44, reqid=osd.2.0:43, at_version=0'0, trim_to=0'0, roll_forward_to=42'29))", 2023-11-05T08:33:14.899 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:55.157875+0000", 2023-11-05T08:33:14.899 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 19.674349791000001, 2023-11-05T08:33:14.899 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0016845110000000001, 2023-11-05T08:33:14.900 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:14.900 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:14.900 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:14.900 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.900 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:14.901 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.157875+0000", 2023-11-05T08:33:14.901 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.901 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.901 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.901 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:14.901 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.157875+0000", 2023-11-05T08:33:14.902 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.902 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.902 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.902 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:14.903 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.157877+0000", 2023-11-05T08:33:14.903 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.9769999999999999e-06 2023-11-05T08:33:14.903 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.903 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.903 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:14.903 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.157904+0000", 2023-11-05T08:33:14.904 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.7648999999999999e-05 2023-11-05T08:33:14.904 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.904 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.904 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:14.904 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.157944+0000", 2023-11-05T08:33:14.905 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.0040999999999998e-05 2023-11-05T08:33:14.905 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.905 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.905 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:14.905 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.157968+0000", 2023-11-05T08:33:14.906 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.3903000000000001e-05 2023-11-05T08:33:14.906 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.906 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.906 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:14.906 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.158144+0000", 2023-11-05T08:33:14.907 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00017574600000000001 2023-11-05T08:33:14.907 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.907 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.907 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:14.907 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.158159+0000", 2023-11-05T08:33:14.908 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.5415999999999999e-05 2023-11-05T08:33:14.908 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.908 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.908 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:14.908 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.159453+0000", 2023-11-05T08:33:14.909 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001293244 2023-11-05T08:33:14.909 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.909 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.909 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:14.909 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.159559+0000", 2023-11-05T08:33:14.910 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000106535 2023-11-05T08:33:14.910 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.910 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:14.910 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.911 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.911 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.911 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.18s6 46/45 ECSubWrite(tid=195, reqid=osd.13.0:195, at_version=46'19, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:14.911 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:59.244820+0000", 2023-11-05T08:33:14.911 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 15.587404326, 2023-11-05T08:33:14.912 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0041897130000000003, 2023-11-05T08:33:14.912 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:14.912 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:14.912 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:14.912 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.913 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:14.913 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.244820+0000", 2023-11-05T08:33:14.913 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.913 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.913 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.914 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:14.914 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.244820+0000", 2023-11-05T08:33:14.914 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.914 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.914 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.915 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:14.915 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.244829+0000", 2023-11-05T08:33:14.915 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.8559999999999996e-06 2023-11-05T08:33:14.915 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.915 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.915 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:14.916 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.244875+0000", 2023-11-05T08:33:14.916 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.5837999999999997e-05 2023-11-05T08:33:14.916 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.916 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.916 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:14.917 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.244903+0000", 2023-11-05T08:33:14.917 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.8123000000000002e-05 2023-11-05T08:33:14.917 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.917 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.917 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:14.918 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.244919+0000", 2023-11-05T08:33:14.918 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.5973000000000001e-05 2023-11-05T08:33:14.918 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.918 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.919 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:14.919 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.245039+0000", 2023-11-05T08:33:14.919 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000120289 2023-11-05T08:33:14.919 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.919 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.920 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:14.920 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.245054+0000", 2023-11-05T08:33:14.920 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.491e-05 2023-11-05T08:33:14.920 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.920 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.921 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:14.921 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.248956+0000", 2023-11-05T08:33:14.921 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0039018859999999998 2023-11-05T08:33:14.921 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.921 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.922 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:14.922 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.249010+0000", 2023-11-05T08:33:14.922 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.3838000000000002e-05 2023-11-05T08:33:14.922 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.922 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:14.923 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.923 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.923 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.923 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.18s6 46/45 ECSubWrite(tid=196, reqid=osd.13.0:195, at_version=0'0, trim_to=0'0, roll_forward_to=46'19))", 2023-11-05T08:33:14.923 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:59.251615+0000", 2023-11-05T08:33:14.924 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 15.580609079, 2023-11-05T08:33:14.924 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001167012, 2023-11-05T08:33:14.924 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:14.924 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:14.925 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:14.925 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.925 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:14.925 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.251615+0000", 2023-11-05T08:33:14.925 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.926 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.926 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.926 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:14.926 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.251615+0000", 2023-11-05T08:33:14.926 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.927 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.927 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.927 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:14.927 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.251617+0000", 2023-11-05T08:33:14.927 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.3969999999999999e-06 2023-11-05T08:33:14.928 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.928 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.928 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:14.928 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.251648+0000", 2023-11-05T08:33:14.928 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.1288999999999998e-05 2023-11-05T08:33:14.929 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.929 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.929 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:14.929 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.251676+0000", 2023-11-05T08:33:14.929 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.7761999999999999e-05 2023-11-05T08:33:14.930 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.930 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.930 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:14.930 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.251691+0000", 2023-11-05T08:33:14.930 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.5379e-05 2023-11-05T08:33:14.931 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.931 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.931 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:14.931 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.251774+0000", 2023-11-05T08:33:14.931 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.2782999999999997e-05 2023-11-05T08:33:14.932 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.932 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.932 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:14.932 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.251788+0000", 2023-11-05T08:33:14.932 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4129e-05 2023-11-05T08:33:14.933 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.933 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.933 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:14.933 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.252738+0000", 2023-11-05T08:33:14.933 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00094977199999999999 2023-11-05T08:33:14.934 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.934 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.934 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:14.934 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.252782+0000", 2023-11-05T08:33:14.934 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.4501e-05 2023-11-05T08:33:14.935 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.935 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:14.935 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.935 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.935 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.936 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.18s6 51/45 ECSubWrite(tid=202, reqid=client.4691.0:335, at_version=51'23, trim_to=0'0, roll_forward_to=49'21))", 2023-11-05T08:33:14.936 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:04.246821+0000", 2023-11-05T08:33:14.936 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 10.585403869, 2023-11-05T08:33:14.936 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0016459389999999999, 2023-11-05T08:33:14.936 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:14.937 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:14.937 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:14.937 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.937 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:14.937 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.246821+0000", 2023-11-05T08:33:14.938 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.938 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.938 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.938 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:14.938 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.246821+0000", 2023-11-05T08:33:14.939 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.939 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.939 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.939 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:14.939 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.246822+0000", 2023-11-05T08:33:14.940 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.517e-06 2023-11-05T08:33:14.940 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.940 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.940 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:14.940 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.246864+0000", 2023-11-05T08:33:14.941 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.1675999999999999e-05 2023-11-05T08:33:14.941 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.941 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.941 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:14.941 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.246915+0000", 2023-11-05T08:33:14.941 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.1406000000000002e-05 2023-11-05T08:33:14.942 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.942 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.942 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:14.942 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.246932+0000", 2023-11-05T08:33:14.943 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.7255e-05 2023-11-05T08:33:14.943 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.943 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.943 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:14.943 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.247022+0000", 2023-11-05T08:33:14.944 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.0019999999999995e-05 2023-11-05T08:33:14.944 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.944 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.944 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:14.944 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.247032+0000", 2023-11-05T08:33:14.945 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.7119999999999995e-06 2023-11-05T08:33:14.945 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.945 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.945 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:14.946 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.248426+0000", 2023-11-05T08:33:14.946 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0013938399999999999 2023-11-05T08:33:14.946 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.946 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.946 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:14.947 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.248466+0000", 2023-11-05T08:33:14.947 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.0513000000000003e-05 2023-11-05T08:33:14.947 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.947 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:14.947 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.948 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.948 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.948 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.18s6 51/45 ECSubWrite(tid=203, reqid=client.4691.0:335, at_version=0'0, trim_to=0'0, roll_forward_to=51'23))", 2023-11-05T08:33:14.948 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:04.255817+0000", 2023-11-05T08:33:14.948 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 10.576407168999999, 2023-11-05T08:33:14.949 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0018718389999999999, 2023-11-05T08:33:14.949 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:14.949 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:14.949 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:14.950 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.950 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:14.950 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.255817+0000", 2023-11-05T08:33:14.950 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.950 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.951 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.951 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:14.951 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.255817+0000", 2023-11-05T08:33:14.951 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.951 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.952 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.952 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:14.952 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.255818+0000", 2023-11-05T08:33:14.952 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.8999999999999995e-07 2023-11-05T08:33:14.952 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.953 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.953 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:14.953 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.255834+0000", 2023-11-05T08:33:14.953 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.6240999999999999e-05 2023-11-05T08:33:14.953 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.954 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.954 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:14.954 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.255851+0000", 2023-11-05T08:33:14.954 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.6427999999999999e-05 2023-11-05T08:33:14.954 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.955 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.955 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:14.955 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.255865+0000", 2023-11-05T08:33:14.955 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4290999999999999e-05 2023-11-05T08:33:14.956 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.956 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.956 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:14.956 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.256066+0000", 2023-11-05T08:33:14.956 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00020114199999999999 2023-11-05T08:33:14.957 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.957 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.957 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:14.957 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.256100+0000", 2023-11-05T08:33:14.958 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.4159000000000003e-05 2023-11-05T08:33:14.958 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.958 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.958 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:14.959 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.257640+0000", 2023-11-05T08:33:14.959 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001539913 2023-11-05T08:33:14.959 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.959 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.959 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:14.960 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.257689+0000", 2023-11-05T08:33:14.960 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.8875000000000002e-05 2023-11-05T08:33:14.960 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.960 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:14.960 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.961 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.961 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.961 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpRead(3.bs3 52/37 ECSubRead(tid=82, to_read={3:d0f842ed:::smithi09039252-18:head=0,73728,0}, subchunks={3:d0f842ed:::smithi09039252-18:head=[0,1]}, attrs_to_read=))", 2023-11-05T08:33:14.961 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:05.259738+0000", 2023-11-05T08:33:14.961 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 9.5724866839999994, 2023-11-05T08:33:14.962 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00040284099999999997, 2023-11-05T08:33:14.962 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:14.962 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:14.962 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:14.963 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.963 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:14.963 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.259738+0000", 2023-11-05T08:33:14.963 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.963 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.964 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.964 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:14.964 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.259738+0000", 2023-11-05T08:33:14.964 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.965 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.965 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.965 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:14.965 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.259738+0000", 2023-11-05T08:33:14.965 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4000000000000001e-07 2023-11-05T08:33:14.966 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.966 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.966 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:14.966 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.259744+0000", 2023-11-05T08:33:14.967 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.6969999999999998e-06 2023-11-05T08:33:14.967 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.967 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.967 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:14.967 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.259761+0000", 2023-11-05T08:33:14.968 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.7728e-05 2023-11-05T08:33:14.968 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.968 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.968 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:14.968 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.259776+0000", 2023-11-05T08:33:14.969 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4962e-05 2023-11-05T08:33:14.969 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.969 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.969 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:14.969 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.259863+0000", 2023-11-05T08:33:14.970 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.7213999999999995e-05 2023-11-05T08:33:14.970 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.970 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.970 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:14.971 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.260141+0000", 2023-11-05T08:33:14.971 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00027710000000000001 2023-11-05T08:33:14.971 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.971 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:14.971 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.972 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.972 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.972 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.18s6 52/45 ECSubWrite(tid=205, reqid=client.4691.0:351, at_version=52'24, trim_to=0'0, roll_forward_to=51'23))", 2023-11-05T08:33:14.972 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:05.281319+0000", 2023-11-05T08:33:14.972 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 9.5509048960000005, 2023-11-05T08:33:14.973 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0018246200000000001, 2023-11-05T08:33:14.973 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:14.973 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:14.973 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:14.973 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.974 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:14.974 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.281319+0000", 2023-11-05T08:33:14.974 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.974 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.975 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.975 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:14.975 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.281319+0000", 2023-11-05T08:33:14.975 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.975 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.976 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.976 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:14.976 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.281320+0000", 2023-11-05T08:33:14.976 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.9500000000000002e-07 2023-11-05T08:33:14.976 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.977 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.977 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:14.977 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.281768+0000", 2023-11-05T08:33:14.977 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00044751799999999999 2023-11-05T08:33:14.977 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.978 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.978 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:14.978 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.281785+0000", 2023-11-05T08:33:14.978 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.6993000000000001e-05 2023-11-05T08:33:14.978 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.979 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.979 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:14.979 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.281798+0000", 2023-11-05T08:33:14.979 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.3733999999999999e-05 2023-11-05T08:33:14.980 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.980 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.980 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:14.980 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.281884+0000", 2023-11-05T08:33:14.980 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.5649999999999995e-05 2023-11-05T08:33:14.981 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.981 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.981 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:14.981 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.281899+0000", 2023-11-05T08:33:14.982 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.5258e-05 2023-11-05T08:33:14.982 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.982 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.982 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:14.982 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.283079+0000", 2023-11-05T08:33:14.983 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0011801559999999999 2023-11-05T08:33:14.983 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.983 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.983 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:14.983 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.283144+0000", 2023-11-05T08:33:14.984 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.4616000000000006e-05 2023-11-05T08:33:14.984 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.984 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:14.984 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.984 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.985 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.985 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.18s6 52/45 ECSubWrite(tid=206, reqid=client.4691.0:352, at_version=52'25, trim_to=0'0, roll_forward_to=51'23))", 2023-11-05T08:33:14.985 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:05.287173+0000", 2023-11-05T08:33:14.985 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 9.5450515229999997, 2023-11-05T08:33:14.985 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0019672029999999998, 2023-11-05T08:33:14.986 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:14.986 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:14.986 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:14.986 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.986 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:14.987 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.287173+0000", 2023-11-05T08:33:14.987 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.987 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.987 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.987 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:14.988 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.287173+0000", 2023-11-05T08:33:14.988 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.988 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.988 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.988 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:14.989 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.287174+0000", 2023-11-05T08:33:14.989 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.1200000000000002e-07 2023-11-05T08:33:14.989 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.989 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.989 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:14.990 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.287701+0000", 2023-11-05T08:33:14.990 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00052763599999999995 2023-11-05T08:33:14.990 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.990 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.990 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:14.991 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.287719+0000", 2023-11-05T08:33:14.991 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.7937000000000001e-05 2023-11-05T08:33:14.991 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.991 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.991 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:14.992 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.287732+0000", 2023-11-05T08:33:14.992 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.2289999999999999e-05 2023-11-05T08:33:14.992 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.992 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.992 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:14.993 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.287801+0000", 2023-11-05T08:33:14.993 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.9029000000000006e-05 2023-11-05T08:33:14.993 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.993 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.993 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:14.994 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.287809+0000", 2023-11-05T08:33:14.994 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.4530000000000002e-06 2023-11-05T08:33:14.994 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.994 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.994 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:14.995 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.289058+0000", 2023-11-05T08:33:14.995 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001248678 2023-11-05T08:33:14.995 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.995 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.995 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:14.996 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.289140+0000", 2023-11-05T08:33:14.996 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.2367999999999995e-05 2023-11-05T08:33:14.996 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.996 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:14.996 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:14.997 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.997 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.997 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.18s6 52/45 ECSubWrite(tid=207, reqid=client.4691.0:353, at_version=52'26, trim_to=0'0, roll_forward_to=51'23))", 2023-11-05T08:33:14.997 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:05.293726+0000", 2023-11-05T08:33:14.997 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 9.5384984409999998, 2023-11-05T08:33:14.998 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000777267, 2023-11-05T08:33:14.998 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:14.998 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:14.998 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:14.998 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:14.999 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:14.999 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.293726+0000", 2023-11-05T08:33:14.999 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:14.999 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:14.999 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.000 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:15.000 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.293726+0000", 2023-11-05T08:33:15.000 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:15.000 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.000 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.001 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:15.001 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.293727+0000", 2023-11-05T08:33:15.001 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.24e-06 2023-11-05T08:33:15.001 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.001 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.002 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:15.002 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.293752+0000", 2023-11-05T08:33:15.002 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.4998e-05 2023-11-05T08:33:15.002 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.002 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.002 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:15.003 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.293768+0000", 2023-11-05T08:33:15.003 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.5860999999999999e-05 2023-11-05T08:33:15.003 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.003 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.003 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:15.004 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.293781+0000", 2023-11-05T08:33:15.004 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.3158e-05 2023-11-05T08:33:15.004 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.004 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.004 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:15.005 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.293863+0000", 2023-11-05T08:33:15.005 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.1526999999999994e-05 2023-11-05T08:33:15.005 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.005 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.005 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:15.006 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.293876+0000", 2023-11-05T08:33:15.006 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.3553999999999999e-05 2023-11-05T08:33:15.006 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.006 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.006 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:15.006 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.294471+0000", 2023-11-05T08:33:15.007 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00059427999999999998 2023-11-05T08:33:15.007 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.007 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.007 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:15.007 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.294503+0000", 2023-11-05T08:33:15.008 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.2648999999999999e-05 2023-11-05T08:33:15.008 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:15.008 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:15.008 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:15.008 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.008 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.009 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.18s6 52/45 ECSubWrite(tid=208, reqid=client.4691.0:353, at_version=0'0, trim_to=0'0, roll_forward_to=52'26))", 2023-11-05T08:33:15.009 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:05.299166+0000", 2023-11-05T08:33:15.009 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 9.5330587609999995, 2023-11-05T08:33:15.009 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00076829299999999997, 2023-11-05T08:33:15.009 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:15.009 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:15.010 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:15.010 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.010 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:15.010 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.299166+0000", 2023-11-05T08:33:15.010 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:15.011 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.011 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.011 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:15.011 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.299166+0000", 2023-11-05T08:33:15.011 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:15.011 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.012 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.012 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:15.012 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.299166+0000", 2023-11-05T08:33:15.012 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.9699999999999995e-07 2023-11-05T08:33:15.012 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.013 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.013 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:15.013 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.299181+0000", 2023-11-05T08:33:15.013 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4912999999999999e-05 2023-11-05T08:33:15.014 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.014 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.014 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:15.014 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.299196+0000", 2023-11-05T08:33:15.014 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4559000000000001e-05 2023-11-05T08:33:15.015 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.015 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.015 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:15.015 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.299208+0000", 2023-11-05T08:33:15.015 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.1873e-05 2023-11-05T08:33:15.016 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.016 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.016 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:15.016 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.299268+0000", 2023-11-05T08:33:15.016 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.0457000000000001e-05 2023-11-05T08:33:15.017 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.017 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.017 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:15.017 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.299279+0000", 2023-11-05T08:33:15.017 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0380000000000001e-05 2023-11-05T08:33:15.018 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.018 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.018 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:15.018 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.299891+0000", 2023-11-05T08:33:15.018 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00061195900000000003 2023-11-05T08:33:15.019 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.019 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.019 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:15.019 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.299934+0000", 2023-11-05T08:33:15.019 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.3355000000000001e-05 2023-11-05T08:33:15.020 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:15.020 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:15.020 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:15.020 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.020 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.021 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.18s6 52/45 ECSubWrite(tid=211, reqid=osd.13.0:211, at_version=52'28, trim_to=0'0, roll_forward_to=52'26))", 2023-11-05T08:33:15.021 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:05.304991+0000", 2023-11-05T08:33:15.021 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 9.5272331860000001, 2023-11-05T08:33:15.021 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00092099800000000002, 2023-11-05T08:33:15.021 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:15.022 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:15.022 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:15.022 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.022 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:15.022 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.304991+0000", 2023-11-05T08:33:15.023 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:15.023 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.023 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.023 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:15.023 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.304991+0000", 2023-11-05T08:33:15.024 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:15.024 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.024 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.024 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:15.025 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.304993+0000", 2023-11-05T08:33:15.025 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.5069999999999999e-06 2023-11-05T08:33:15.025 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.025 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.025 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:15.026 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.305028+0000", 2023-11-05T08:33:15.026 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.4935999999999999e-05 2023-11-05T08:33:15.026 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.026 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.026 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:15.027 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.305051+0000", 2023-11-05T08:33:15.027 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.3252e-05 2023-11-05T08:33:15.027 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.027 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.027 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:15.028 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.305070+0000", 2023-11-05T08:33:15.028 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.8757e-05 2023-11-05T08:33:15.028 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.028 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.028 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:15.029 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.305136+0000", 2023-11-05T08:33:15.029 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.6655999999999993e-05 2023-11-05T08:33:15.029 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.029 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.029 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:15.030 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.305143+0000", 2023-11-05T08:33:15.030 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.6730000000000003e-06 2023-11-05T08:33:15.030 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.030 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.030 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:15.031 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.305882+0000", 2023-11-05T08:33:15.031 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00073891500000000004 2023-11-05T08:33:15.031 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.031 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.031 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:15.032 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.305912+0000", 2023-11-05T08:33:15.032 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.0301999999999999e-05 2023-11-05T08:33:15.032 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:15.032 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:15.032 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:15.033 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.033 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.033 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.18s6 52/45 ECSubWrite(tid=212, reqid=osd.13.0:211, at_version=0'0, trim_to=0'0, roll_forward_to=52'28))", 2023-11-05T08:33:15.033 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:05.308503+0000", 2023-11-05T08:33:15.033 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 9.5237215410000005, 2023-11-05T08:33:15.034 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00052752300000000001, 2023-11-05T08:33:15.034 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:15.034 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:15.034 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:15.034 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.035 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:15.035 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.308503+0000", 2023-11-05T08:33:15.035 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:15.035 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.035 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.036 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:15.036 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.308503+0000", 2023-11-05T08:33:15.036 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:15.036 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.036 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.037 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:15.037 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.308503+0000", 2023-11-05T08:33:15.037 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.0500000000000003e-07 2023-11-05T08:33:15.037 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.037 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.038 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:15.038 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.308515+0000", 2023-11-05T08:33:15.038 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.1399999999999999e-05 2023-11-05T08:33:15.038 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.038 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.039 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:15.039 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.308534+0000", 2023-11-05T08:33:15.039 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.9553000000000001e-05 2023-11-05T08:33:15.039 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.039 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.040 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:15.040 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.308548+0000", 2023-11-05T08:33:15.040 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.3132999999999999e-05 2023-11-05T08:33:15.040 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.040 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.040 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:15.041 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.308590+0000", 2023-11-05T08:33:15.041 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.2533000000000001e-05 2023-11-05T08:33:15.041 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.041 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.041 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:15.042 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.308596+0000", 2023-11-05T08:33:15.042 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.7710000000000003e-06 2023-11-05T08:33:15.042 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.042 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.042 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:15.043 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.309003+0000", 2023-11-05T08:33:15.043 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00040676699999999999 2023-11-05T08:33:15.043 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.043 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.043 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:15.044 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.309030+0000", 2023-11-05T08:33:15.044 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.7761000000000001e-05 2023-11-05T08:33:15.044 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:15.044 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:15.044 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:15.044 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.045 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.045 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es6 57/30 ECSubWrite(tid=60, reqid=client.4691.0:380, at_version=57'45, trim_to=0'0, roll_forward_to=49'43))", 2023-11-05T08:33:15.045 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:10.285649+0000", 2023-11-05T08:33:15.045 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 4.5465749689999999, 2023-11-05T08:33:15.045 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0029323890000000001, 2023-11-05T08:33:15.045 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:15.046 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:15.046 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:15.046 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.046 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:15.046 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285649+0000", 2023-11-05T08:33:15.047 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:15.047 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.047 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.047 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:15.047 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285649+0000", 2023-11-05T08:33:15.047 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:15.048 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.048 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.048 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:15.048 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285650+0000", 2023-11-05T08:33:15.048 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.4300000000000003e-07 2023-11-05T08:33:15.048 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.049 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.049 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:15.049 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285727+0000", 2023-11-05T08:33:15.049 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.7088000000000005e-05 2023-11-05T08:33:15.049 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.049 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.050 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:15.050 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285762+0000", 2023-11-05T08:33:15.050 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.4808999999999998e-05 2023-11-05T08:33:15.050 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.050 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.051 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:15.051 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285788+0000", 2023-11-05T08:33:15.051 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.6075999999999999e-05 2023-11-05T08:33:15.051 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.051 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.051 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:15.052 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285958+0000", 2023-11-05T08:33:15.052 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00016993499999999999 2023-11-05T08:33:15.052 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.052 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.052 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:15.052 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285977+0000", 2023-11-05T08:33:15.053 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.8692999999999999e-05 2023-11-05T08:33:15.053 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.053 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.053 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:15.053 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.288484+0000", 2023-11-05T08:33:15.054 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.002507085 2023-11-05T08:33:15.054 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.054 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.054 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:15.054 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.288582+0000", 2023-11-05T08:33:15.055 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.8060000000000006e-05 2023-11-05T08:33:15.055 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:15.055 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:15.055 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:15.056 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.056 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.056 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es6 57/30 ECSubWrite(tid=61, reqid=client.4691.0:380, at_version=0'0, trim_to=0'0, roll_forward_to=57'45))", 2023-11-05T08:33:15.056 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:10.290231+0000", 2023-11-05T08:33:15.056 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 4.541993723, 2023-11-05T08:33:15.057 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0024684110000000002, 2023-11-05T08:33:15.057 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:15.057 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:15.057 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:15.057 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.058 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:15.058 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.290231+0000", 2023-11-05T08:33:15.058 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:15.058 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.058 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.059 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:15.059 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.290231+0000", 2023-11-05T08:33:15.059 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:15.059 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.059 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.060 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:15.060 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.290233+0000", 2023-11-05T08:33:15.060 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.9829999999999999e-06 2023-11-05T08:33:15.060 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.061 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.061 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:15.061 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.290260+0000", 2023-11-05T08:33:15.061 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.6889e-05 2023-11-05T08:33:15.061 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.062 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.062 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:15.062 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.290304+0000", 2023-11-05T08:33:15.062 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.4589000000000003e-05 2023-11-05T08:33:15.063 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.063 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.063 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:15.063 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.290327+0000", 2023-11-05T08:33:15.063 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.3302000000000002e-05 2023-11-05T08:33:15.064 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.064 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.064 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:15.064 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.290608+0000", 2023-11-05T08:33:15.065 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00028071599999999997 2023-11-05T08:33:15.065 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.065 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.065 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:15.065 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.290656+0000", 2023-11-05T08:33:15.066 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.8257999999999998e-05 2023-11-05T08:33:15.066 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.066 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.066 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:15.066 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.292606+0000", 2023-11-05T08:33:15.067 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0019496100000000001 2023-11-05T08:33:15.067 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.067 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.067 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:15.067 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.292699+0000", 2023-11-05T08:33:15.068 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.3064000000000002e-05 2023-11-05T08:33:15.068 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:15.068 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:15.068 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:15.068 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.069 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.069 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDScrubReserve(3.18s6 REQUEST e60)", 2023-11-05T08:33:15.069 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:13.953181+0000", 2023-11-05T08:33:15.069 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 0.87904355700000003, 2023-11-05T08:33:15.070 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0021641680000000002, 2023-11-05T08:33:15.070 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:15.070 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "started", 2023-11-05T08:33:15.070 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:15.071 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.071 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:15.071 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:13.953181+0000", 2023-11-05T08:33:15.071 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:15.071 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.072 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.072 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:15.072 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:13.953181+0000", 2023-11-05T08:33:15.072 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:15.072 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.073 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.073 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:15.073 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:13.953181+0000", 2023-11-05T08:33:15.073 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.5000000000000003e-07 2023-11-05T08:33:15.074 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.074 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.074 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:15.074 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:13.953187+0000", 2023-11-05T08:33:15.075 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.1e-06 2023-11-05T08:33:15.075 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.075 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.075 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:15.075 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:13.953215+0000", 2023-11-05T08:33:15.076 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.7959000000000001e-05 2023-11-05T08:33:15.076 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.076 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.076 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:15.076 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:13.953233+0000", 2023-11-05T08:33:15.077 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.7402999999999999e-05 2023-11-05T08:33:15.077 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.077 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.077 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:15.078 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:13.953326+0000", 2023-11-05T08:33:15.078 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.3616999999999995e-05 2023-11-05T08:33:15.078 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.078 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.078 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "op must wait for map", 2023-11-05T08:33:15.079 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:13.953334+0000", 2023-11-05T08:33:15.079 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.131e-06 2023-11-05T08:33:15.079 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.079 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.079 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:15.080 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:13.955269+0000", 2023-11-05T08:33:15.080 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0019357179999999999 2023-11-05T08:33:15.080 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.080 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.081 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:15.081 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:13.955288+0000", 2023-11-05T08:33:15.081 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.8533e-05 2023-11-05T08:33:15.081 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:15.081 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:15.082 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:15.082 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:13.955345+0000", 2023-11-05T08:33:15.082 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.7157000000000002e-05 2023-11-05T08:33:15.082 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:15.082 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:15.083 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:15.083 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:15.083 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:15.083 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:33:15.085 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-11-05T08:33:15.086 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:33:15.082+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:33:15.119 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [3] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 7, 11, 15] 2023-11-05T08:33:15.119 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-11-05T08:33:15.120 INFO:tasks.thrashosds.thrasher:Reweighting osd 1 to 0.9591620623257814 2023-11-05T08:33:15.120 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 0.9591620623257814 2023-11-05T08:33:15.186 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:33:15.179+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:33:15.221 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:33:15.221 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2023-11-05T08:33:15.222 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 0 2023-11-05T08:33:15.222 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:33:15.236 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-11-05T08:33:15.287 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:15.283+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:15.332 INFO:tasks.rados.rados.0.smithi090.stdout:164: done (3 left) 2023-11-05T08:33:15.332 INFO:tasks.rados.rados.0.smithi090.stdout:167: done (2 left) 2023-11-05T08:33:15.332 INFO:tasks.rados.rados.0.smithi090.stdout:168: done (1 left) 2023-11-05T08:33:15.332 INFO:tasks.rados.rados.0.smithi090.stdout:169: rmattr oid 11 current snap is 16 2023-11-05T08:33:15.333 INFO:tasks.rados.rados.0.smithi090.stdout:169: done (1 left) 2023-11-05T08:33:15.333 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 41 current snap is 16 2023-11-05T08:33:15.333 INFO:tasks.rados.rados.0.smithi090.stdout:170: seq_num 85 ranges {688128=671744,1359872=770048} 2023-11-05T08:33:15.352 INFO:tasks.rados.rados.0.smithi090.stdout:170: writing smithi09039252-41 from 688128 to 1359872 tid 1 2023-11-05T08:33:15.364 INFO:tasks.rados.rados.0.smithi090.stdout:170: writing smithi09039252-41 from 1359872 to 2129920 tid 2 2023-11-05T08:33:15.365 INFO:tasks.rados.rados.0.smithi090.stdout:171: read oid 21 snap -1 2023-11-05T08:33:15.365 INFO:tasks.rados.rados.0.smithi090.stdout:171: expect deleted 2023-11-05T08:33:15.365 INFO:tasks.rados.rados.0.smithi090.stdout:172: rmattr oid 44 current snap is 16 2023-11-05T08:33:15.365 INFO:tasks.rados.rados.0.smithi090.stdout:172: done (3 left) 2023-11-05T08:33:15.365 INFO:tasks.rados.rados.0.smithi090.stdout:173: setattr oid 11 current snap is 16 2023-11-05T08:33:15.367 INFO:tasks.rados.rados.0.smithi090.stdout:171: done (3 left) 2023-11-05T08:33:15.367 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 5 current snap is 16 2023-11-05T08:33:15.368 INFO:tasks.rados.rados.0.smithi090.stdout:174: seq_num 86 ranges {688128=688128,1376256=540672,1916928=229376} 2023-11-05T08:33:15.373 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:33:15.373 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2023-11-05T08:33:15.374 INFO:teuthology.orchestra.run.smithi053.stdout: "complaint_time": 30, 2023-11-05T08:33:15.374 INFO:teuthology.orchestra.run.smithi053.stdout: "num_blocked_ops": 0 2023-11-05T08:33:15.374 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:33:15.387 INFO:tasks.rados.rados.0.smithi090.stdout:174: writing smithi09039252-5 from 688128 to 1376256 tid 1 2023-11-05T08:33:15.387 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:33:15.386+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:33:15.389 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-11-05T08:33:15.393 INFO:tasks.rados.rados.0.smithi090.stdout:174: writing smithi09039252-5 from 1376256 to 1916928 tid 2 2023-11-05T08:33:15.396 INFO:tasks.rados.rados.0.smithi090.stdout:174: writing smithi09039252-5 from 1916928 to 2146304 tid 3 2023-11-05T08:33:15.396 INFO:tasks.rados.rados.0.smithi090.stdout:175: snap_remove snap 15 2023-11-05T08:33:15.488 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:33:15.485+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:33:15.512 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:33:15.512 INFO:teuthology.orchestra.run.smithi053.stdout: "size": 20, 2023-11-05T08:33:15.512 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 600, 2023-11-05T08:33:15.512 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2023-11-05T08:33:15.513 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.513 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.8s3 36/30 ECSubWrite(tid=23, reqid=osd.5.0:23, at_version=36'15, trim_to=0'0, roll_forward_to=34'13))", 2023-11-05T08:33:15.513 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:49.078476+0000", 2023-11-05T08:33:15.513 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 26.430265128999999, 2023-11-05T08:33:15.513 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0057340960000000002, 2023-11-05T08:33:15.514 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:15.514 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:15.514 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:15.514 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.514 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:15.515 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.078476+0000", 2023-11-05T08:33:15.515 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.515 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.515 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.515 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:15.515 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.078476+0000", 2023-11-05T08:33:15.516 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.516 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.516 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.516 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:15.516 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.078476+0000", 2023-11-05T08:33:15.517 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1299999999999998e-07 2023-11-05T08:33:15.517 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.517 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.517 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:15.517 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.078542+0000", 2023-11-05T08:33:15.518 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5121000000000002e-05 2023-11-05T08:33:15.518 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.518 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.518 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:15.518 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.078572+0000", 2023-11-05T08:33:15.518 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0812000000000002e-05 2023-11-05T08:33:15.519 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.519 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.519 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:15.519 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.078599+0000", 2023-11-05T08:33:15.519 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6653e-05 2023-11-05T08:33:15.520 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.520 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.520 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:15.520 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.078773+0000", 2023-11-05T08:33:15.520 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00017369099999999999 2023-11-05T08:33:15.521 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.521 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.521 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:15.521 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.078798+0000", 2023-11-05T08:33:15.521 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5153000000000001e-05 2023-11-05T08:33:15.522 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.522 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.522 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:15.522 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.084110+0000", 2023-11-05T08:33:15.522 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0053123190000000002 2023-11-05T08:33:15.523 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.523 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.523 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:15.523 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:49.084210+0000", 2023-11-05T08:33:15.523 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.9433999999999997e-05 2023-11-05T08:33:15.524 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.524 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:15.524 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.524 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.524 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.524 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es2 41/30 ECSubWrite(tid=40, reqid=osd.2.0:40, at_version=41'27, trim_to=0'0, roll_forward_to=40'25))", 2023-11-05T08:33:15.525 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.127197+0000", 2023-11-05T08:33:15.525 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 21.381543424, 2023-11-05T08:33:15.525 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.019700031, 2023-11-05T08:33:15.525 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:15.525 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:15.526 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:15.526 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.526 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:15.526 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127197+0000", 2023-11-05T08:33:15.526 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.527 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.527 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.527 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:15.527 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127197+0000", 2023-11-05T08:33:15.527 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.528 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.528 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.528 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:15.528 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127198+0000", 2023-11-05T08:33:15.528 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.0999999999999998e-07 2023-11-05T08:33:15.528 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.529 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.529 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:15.529 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127255+0000", 2023-11-05T08:33:15.529 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6762000000000003e-05 2023-11-05T08:33:15.529 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.529 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.530 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:15.530 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127291+0000", 2023-11-05T08:33:15.530 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5948999999999998e-05 2023-11-05T08:33:15.530 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.530 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.531 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:15.531 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127314+0000", 2023-11-05T08:33:15.531 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3686e-05 2023-11-05T08:33:15.531 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.531 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.532 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:15.532 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127534+0000", 2023-11-05T08:33:15.532 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00021922700000000001 2023-11-05T08:33:15.532 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.532 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.533 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:15.533 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127562+0000", 2023-11-05T08:33:15.533 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8099e-05 2023-11-05T08:33:15.533 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.533 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.534 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:15.534 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.146742+0000", 2023-11-05T08:33:15.534 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.019180135000000001 2023-11-05T08:33:15.534 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.534 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.534 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:15.535 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.146897+0000", 2023-11-05T08:33:15.535 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000155463 2023-11-05T08:33:15.535 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.535 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:15.535 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.536 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.536 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.536 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as2 41/33 ECSubWrite(tid=35, reqid=osd.7.0:35, at_version=41'12, trim_to=0'0, roll_forward_to=40'10))", 2023-11-05T08:33:15.536 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.130596+0000", 2023-11-05T08:33:15.536 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 21.378144201000001, 2023-11-05T08:33:15.537 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022182642999999998, 2023-11-05T08:33:15.537 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:15.537 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:15.537 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:15.537 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.538 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:15.538 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130596+0000", 2023-11-05T08:33:15.538 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.538 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.538 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.539 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:15.539 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130596+0000", 2023-11-05T08:33:15.539 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.539 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.539 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.540 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:15.540 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130597+0000", 2023-11-05T08:33:15.540 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.7600000000000006e-07 2023-11-05T08:33:15.540 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.540 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.541 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:15.541 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130660+0000", 2023-11-05T08:33:15.541 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3015000000000006e-05 2023-11-05T08:33:15.541 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.541 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.541 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:15.542 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130725+0000", 2023-11-05T08:33:15.542 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4988999999999997e-05 2023-11-05T08:33:15.542 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.542 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.542 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:15.543 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130750+0000", 2023-11-05T08:33:15.543 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4802000000000001e-05 2023-11-05T08:33:15.543 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.543 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.543 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:15.544 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.131037+0000", 2023-11-05T08:33:15.544 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00028702900000000001 2023-11-05T08:33:15.544 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.544 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.544 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:15.544 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.131064+0000", 2023-11-05T08:33:15.545 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7022e-05 2023-11-05T08:33:15.545 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.545 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.545 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:15.545 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.152706+0000", 2023-11-05T08:33:15.546 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.021641734999999999 2023-11-05T08:33:15.546 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.546 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.546 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:15.546 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.152779+0000", 2023-11-05T08:33:15.547 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.3075000000000001e-05 2023-11-05T08:33:15.547 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.547 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:15.547 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.547 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.548 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.548 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as2 41/33 ECSubWrite(tid=36, reqid=osd.7.0:36, at_version=41'14, trim_to=0'0, roll_forward_to=40'10))", 2023-11-05T08:33:15.548 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.130806+0000", 2023-11-05T08:33:15.548 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 21.377934321000001, 2023-11-05T08:33:15.548 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.035467152000000002, 2023-11-05T08:33:15.548 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:15.549 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:15.549 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:15.549 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.549 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:15.549 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130806+0000", 2023-11-05T08:33:15.550 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.550 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.550 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.550 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:15.550 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130806+0000", 2023-11-05T08:33:15.551 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.551 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.551 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.551 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:15.551 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130807+0000", 2023-11-05T08:33:15.552 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1000000000000001e-06 2023-11-05T08:33:15.552 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.552 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.552 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:15.552 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130852+0000", 2023-11-05T08:33:15.552 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4428999999999998e-05 2023-11-05T08:33:15.553 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.553 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.553 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:15.553 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130876+0000", 2023-11-05T08:33:15.553 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3975999999999999e-05 2023-11-05T08:33:15.553 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.554 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.554 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:15.554 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130896+0000", 2023-11-05T08:33:15.554 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9905999999999999e-05 2023-11-05T08:33:15.554 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.555 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.555 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:15.555 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.143384+0000", 2023-11-05T08:33:15.555 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012488727999999999 2023-11-05T08:33:15.555 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.555 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.556 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:15.556 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.143412+0000", 2023-11-05T08:33:15.556 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7988999999999999e-05 2023-11-05T08:33:15.556 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.556 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.556 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:15.557 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.166186+0000", 2023-11-05T08:33:15.557 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022773787 2023-11-05T08:33:15.557 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.557 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.557 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:15.558 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.166273+0000", 2023-11-05T08:33:15.558 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.7237000000000005e-05 2023-11-05T08:33:15.558 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.558 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:15.558 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.559 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.559 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.559 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es2 41/30 ECSubWrite(tid=41, reqid=osd.2.0:40, at_version=0'0, trim_to=0'0, roll_forward_to=41'27))", 2023-11-05T08:33:15.559 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.148654+0000", 2023-11-05T08:33:15.559 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 21.360086677000002, 2023-11-05T08:33:15.559 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017394370999999999, 2023-11-05T08:33:15.560 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:15.560 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:15.560 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:15.560 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.560 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:15.561 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148654+0000", 2023-11-05T08:33:15.561 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.561 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.561 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.561 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:15.561 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148654+0000", 2023-11-05T08:33:15.562 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.562 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.562 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.562 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:15.562 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148656+0000", 2023-11-05T08:33:15.562 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.677e-06 2023-11-05T08:33:15.563 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.563 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.563 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:15.563 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148683+0000", 2023-11-05T08:33:15.563 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7246e-05 2023-11-05T08:33:15.564 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.564 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.564 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:15.564 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148715+0000", 2023-11-05T08:33:15.564 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2518999999999998e-05 2023-11-05T08:33:15.564 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.565 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.565 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:15.565 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148742+0000", 2023-11-05T08:33:15.565 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6876000000000002e-05 2023-11-05T08:33:15.565 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.566 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.566 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:15.566 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148890+0000", 2023-11-05T08:33:15.566 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014771599999999999 2023-11-05T08:33:15.566 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.567 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.567 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:15.567 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148915+0000", 2023-11-05T08:33:15.567 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5099000000000001e-05 2023-11-05T08:33:15.567 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.568 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.568 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:15.568 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.165914+0000", 2023-11-05T08:33:15.568 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.016999018000000001 2023-11-05T08:33:15.568 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.569 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.569 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:15.569 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.166048+0000", 2023-11-05T08:33:15.569 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013422 2023-11-05T08:33:15.569 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.570 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:15.570 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.570 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.570 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.570 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.2s5 41/40 ECSubWrite(tid=187, reqid=client.4691.0:292, at_version=41'9, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:15.571 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.162642+0000", 2023-11-05T08:33:15.571 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 21.346098277999999, 2023-11-05T08:33:15.571 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010859549, 2023-11-05T08:33:15.571 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:15.571 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:15.571 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:15.572 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.572 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:15.572 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.162642+0000", 2023-11-05T08:33:15.572 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.572 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.573 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.573 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:15.573 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.162642+0000", 2023-11-05T08:33:15.573 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.573 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.574 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.574 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:15.574 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.162644+0000", 2023-11-05T08:33:15.574 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.823e-06 2023-11-05T08:33:15.574 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.575 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.575 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:15.575 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.162700+0000", 2023-11-05T08:33:15.575 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5863999999999999e-05 2023-11-05T08:33:15.575 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.575 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.576 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:15.576 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.162731+0000", 2023-11-05T08:33:15.576 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0731000000000001e-05 2023-11-05T08:33:15.576 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.576 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.577 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:15.577 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.162754+0000", 2023-11-05T08:33:15.577 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3234000000000002e-05 2023-11-05T08:33:15.577 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.577 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.577 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:15.578 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.162988+0000", 2023-11-05T08:33:15.578 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00023393399999999999 2023-11-05T08:33:15.578 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.578 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.578 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:15.579 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.163016+0000", 2023-11-05T08:33:15.579 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8309e-05 2023-11-05T08:33:15.579 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.579 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.579 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:15.579 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.173420+0000", 2023-11-05T08:33:15.580 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010403802 2023-11-05T08:33:15.580 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.580 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.580 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:15.580 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.173502+0000", 2023-11-05T08:33:15.580 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.1852000000000002e-05 2023-11-05T08:33:15.581 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.581 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:15.581 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.581 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.581 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.581 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as2 41/33 ECSubWrite(tid=37, reqid=osd.7.0:36, at_version=0'0, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:33:15.582 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.167511+0000", 2023-11-05T08:33:15.582 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 21.34122984, 2023-11-05T08:33:15.582 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011732691, 2023-11-05T08:33:15.582 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:15.582 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:15.583 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:15.583 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.583 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:15.583 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167511+0000", 2023-11-05T08:33:15.583 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.583 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.584 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.584 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:15.584 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167511+0000", 2023-11-05T08:33:15.584 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.584 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.585 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.585 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:15.585 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167513+0000", 2023-11-05T08:33:15.585 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.0829999999999998e-06 2023-11-05T08:33:15.585 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.585 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.586 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:15.586 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167540+0000", 2023-11-05T08:33:15.586 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7095999999999999e-05 2023-11-05T08:33:15.586 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.586 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.587 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:15.587 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167572+0000", 2023-11-05T08:33:15.587 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2462000000000002e-05 2023-11-05T08:33:15.587 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.587 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.588 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:15.588 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167594+0000", 2023-11-05T08:33:15.588 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1316999999999999e-05 2023-11-05T08:33:15.588 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.588 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.588 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:15.589 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167791+0000", 2023-11-05T08:33:15.589 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00019748199999999999 2023-11-05T08:33:15.589 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.589 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.589 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:15.590 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167815+0000", 2023-11-05T08:33:15.590 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3536999999999998e-05 2023-11-05T08:33:15.590 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.590 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.590 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:15.591 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.179154+0000", 2023-11-05T08:33:15.591 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011339373999999999 2023-11-05T08:33:15.591 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.591 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.591 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:15.592 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.179244+0000", 2023-11-05T08:33:15.592 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9339999999999995e-05 2023-11-05T08:33:15.592 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.592 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:15.592 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.592 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.593 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.593 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.2s5 41/40 ECSubWrite(tid=188, reqid=osd.13.0:188, at_version=41'11, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:15.593 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.171651+0000", 2023-11-05T08:33:15.593 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 21.337089252999998, 2023-11-05T08:33:15.593 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0076644390000000003, 2023-11-05T08:33:15.594 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:15.594 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:15.594 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:15.594 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.594 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:15.595 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171651+0000", 2023-11-05T08:33:15.595 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.595 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.595 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.595 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:15.596 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171651+0000", 2023-11-05T08:33:15.596 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.596 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.596 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.596 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:15.597 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171653+0000", 2023-11-05T08:33:15.597 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5039999999999999e-06 2023-11-05T08:33:15.597 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.597 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.597 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:15.597 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171700+0000", 2023-11-05T08:33:15.598 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.6989e-05 2023-11-05T08:33:15.598 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.598 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.598 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:15.598 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171731+0000", 2023-11-05T08:33:15.599 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1310999999999999e-05 2023-11-05T08:33:15.599 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.599 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.599 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:15.599 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171750+0000", 2023-11-05T08:33:15.600 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9188e-05 2023-11-05T08:33:15.600 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.600 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.600 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:15.600 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171900+0000", 2023-11-05T08:33:15.601 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00015010699999999999 2023-11-05T08:33:15.601 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.601 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.601 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:15.601 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171920+0000", 2023-11-05T08:33:15.601 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9626000000000001e-05 2023-11-05T08:33:15.602 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.602 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.602 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:15.602 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.179250+0000", 2023-11-05T08:33:15.602 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0073298900000000004 2023-11-05T08:33:15.603 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.603 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.603 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:15.603 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.179316+0000", 2023-11-05T08:33:15.603 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5823999999999999e-05 2023-11-05T08:33:15.604 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.604 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:15.604 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.604 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.604 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.605 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.14s3 41/40 ECSubWrite(tid=17, reqid=client.4691.0:278, at_version=41'14, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:15.605 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.184891+0000", 2023-11-05T08:33:15.605 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 21.323849963000001, 2023-11-05T08:33:15.605 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0062390429999999997, 2023-11-05T08:33:15.605 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:15.605 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:15.606 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:15.606 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.606 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:15.606 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.184891+0000", 2023-11-05T08:33:15.607 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.607 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.607 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.607 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:15.607 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.184891+0000", 2023-11-05T08:33:15.607 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.608 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.608 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.608 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:15.608 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.184892+0000", 2023-11-05T08:33:15.608 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.497e-06 2023-11-05T08:33:15.609 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.609 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.609 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:15.609 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.185224+0000", 2023-11-05T08:33:15.610 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00033133899999999999 2023-11-05T08:33:15.610 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.610 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.610 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:15.610 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.185245+0000", 2023-11-05T08:33:15.611 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1185000000000001e-05 2023-11-05T08:33:15.611 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.611 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.611 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:15.611 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.185261+0000", 2023-11-05T08:33:15.611 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6067999999999999e-05 2023-11-05T08:33:15.612 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.612 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.612 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:15.612 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.185463+0000", 2023-11-05T08:33:15.612 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00020208800000000001 2023-11-05T08:33:15.613 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.613 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.613 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:15.613 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.185483+0000", 2023-11-05T08:33:15.613 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.0092e-05 2023-11-05T08:33:15.614 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.614 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.614 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:15.614 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.191053+0000", 2023-11-05T08:33:15.614 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0055700919999999996 2023-11-05T08:33:15.615 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.615 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.615 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:15.615 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.191130+0000", 2023-11-05T08:33:15.615 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.6681999999999995e-05 2023-11-05T08:33:15.616 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.616 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:15.616 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.616 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.616 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.616 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as2 42/33 ECSubWrite(tid=38, reqid=osd.7.0:38, at_version=42'16, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:33:15.617 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:55.121755+0000", 2023-11-05T08:33:15.617 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 20.386985252999999, 2023-11-05T08:33:15.617 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011609566, 2023-11-05T08:33:15.617 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:15.617 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:15.618 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:15.618 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.618 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:15.618 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.121755+0000", 2023-11-05T08:33:15.618 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.619 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.619 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.619 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:15.619 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.121755+0000", 2023-11-05T08:33:15.619 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.620 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.620 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.620 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:15.620 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.121756+0000", 2023-11-05T08:33:15.620 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.0299999999999998e-07 2023-11-05T08:33:15.620 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.621 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.621 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:15.621 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.121810+0000", 2023-11-05T08:33:15.621 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3918000000000001e-05 2023-11-05T08:33:15.621 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.622 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.622 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:15.622 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.121861+0000", 2023-11-05T08:33:15.622 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.0846000000000001e-05 2023-11-05T08:33:15.622 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.623 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.623 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:15.623 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.121889+0000", 2023-11-05T08:33:15.623 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8359000000000001e-05 2023-11-05T08:33:15.623 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.624 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.624 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:15.624 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.122158+0000", 2023-11-05T08:33:15.624 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00026898800000000001 2023-11-05T08:33:15.624 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.625 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.625 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:15.625 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.122192+0000", 2023-11-05T08:33:15.625 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3259000000000001e-05 2023-11-05T08:33:15.625 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.626 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.626 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:15.626 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.133269+0000", 2023-11-05T08:33:15.626 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011076979000000001 2023-11-05T08:33:15.626 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.627 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.627 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:15.627 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.133365+0000", 2023-11-05T08:33:15.627 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6414000000000002e-05 2023-11-05T08:33:15.627 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.627 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:15.628 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.628 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.628 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.628 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as2 42/33 ECSubWrite(tid=39, reqid=osd.7.0:39, at_version=42'18, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:33:15.628 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:55.124648+0000", 2023-11-05T08:33:15.629 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 20.384092393, 2023-11-05T08:33:15.629 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.025426339999999999, 2023-11-05T08:33:15.629 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:15.629 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:15.629 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:15.630 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.630 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:15.630 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.124648+0000", 2023-11-05T08:33:15.630 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.630 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.631 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.631 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:15.631 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.124648+0000", 2023-11-05T08:33:15.631 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.631 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.631 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.632 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:15.632 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.124650+0000", 2023-11-05T08:33:15.632 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.463e-06 2023-11-05T08:33:15.632 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.632 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.633 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:15.633 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.124710+0000", 2023-11-05T08:33:15.633 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0001999999999999e-05 2023-11-05T08:33:15.633 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.633 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.634 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:15.634 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.124739+0000", 2023-11-05T08:33:15.634 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9278999999999999e-05 2023-11-05T08:33:15.634 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.634 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.635 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:15.635 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.124763+0000", 2023-11-05T08:33:15.635 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4036000000000002e-05 2023-11-05T08:33:15.635 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.635 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.636 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:15.636 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.125122+0000", 2023-11-05T08:33:15.636 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00035926900000000001 2023-11-05T08:33:15.636 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.636 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.637 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:15.637 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.125149+0000", 2023-11-05T08:33:15.637 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7192999999999999e-05 2023-11-05T08:33:15.637 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.637 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.638 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:15.638 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.149969+0000", 2023-11-05T08:33:15.638 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.024819074 2023-11-05T08:33:15.638 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.638 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.639 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:15.639 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.150075+0000", 2023-11-05T08:33:15.639 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000106024 2023-11-05T08:33:15.639 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.639 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:15.640 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.640 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.640 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.640 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es2 42/30 ECSubWrite(tid=43, reqid=osd.2.0:43, at_version=42'29, trim_to=0'0, roll_forward_to=41'27))", 2023-11-05T08:33:15.640 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:55.134544+0000", 2023-11-05T08:33:15.640 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 20.374196473000001, 2023-11-05T08:33:15.641 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.02091324, 2023-11-05T08:33:15.641 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:15.641 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:15.641 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:15.641 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.642 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:15.642 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134544+0000", 2023-11-05T08:33:15.642 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.642 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.642 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.642 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:15.643 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134544+0000", 2023-11-05T08:33:15.643 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.643 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.643 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.643 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:15.644 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134545+0000", 2023-11-05T08:33:15.644 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3499999999999996e-07 2023-11-05T08:33:15.644 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.644 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.644 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:15.645 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134593+0000", 2023-11-05T08:33:15.645 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.8538999999999997e-05 2023-11-05T08:33:15.645 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.645 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.645 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:15.646 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134621+0000", 2023-11-05T08:33:15.646 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7939000000000001e-05 2023-11-05T08:33:15.646 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.646 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.646 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:15.647 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134641+0000", 2023-11-05T08:33:15.647 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9352999999999999e-05 2023-11-05T08:33:15.647 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.647 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.647 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:15.648 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134788+0000", 2023-11-05T08:33:15.648 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000147806 2023-11-05T08:33:15.648 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.648 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.648 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:15.649 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134850+0000", 2023-11-05T08:33:15.649 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1614999999999999e-05 2023-11-05T08:33:15.649 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.649 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.649 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:15.650 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.155400+0000", 2023-11-05T08:33:15.650 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.020550023000000001 2023-11-05T08:33:15.650 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.650 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.650 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:15.651 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.155457+0000", 2023-11-05T08:33:15.651 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7330000000000002e-05 2023-11-05T08:33:15.651 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.651 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:15.651 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.652 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.652 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.652 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.2s5 42/40 ECSubWrite(tid=192, reqid=osd.13.0:192, at_version=42'13, trim_to=0'0, roll_forward_to=41'11))", 2023-11-05T08:33:15.652 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:55.150316+0000", 2023-11-05T08:33:15.652 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 20.358424976999999, 2023-11-05T08:33:15.653 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0088926939999999996, 2023-11-05T08:33:15.653 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:15.653 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:15.653 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:15.653 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.654 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:15.654 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.150316+0000", 2023-11-05T08:33:15.654 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.654 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.654 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.655 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:15.655 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.150316+0000", 2023-11-05T08:33:15.655 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.655 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.656 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.656 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:15.656 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.150318+0000", 2023-11-05T08:33:15.656 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.0140000000000001e-06 2023-11-05T08:33:15.656 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.656 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.657 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:15.657 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.150377+0000", 2023-11-05T08:33:15.657 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9231000000000003e-05 2023-11-05T08:33:15.657 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.658 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.658 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:15.658 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.150411+0000", 2023-11-05T08:33:15.658 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4279000000000002e-05 2023-11-05T08:33:15.658 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.659 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.659 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:15.659 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.150435+0000", 2023-11-05T08:33:15.659 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4165999999999999e-05 2023-11-05T08:33:15.659 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.659 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.660 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:15.660 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.152989+0000", 2023-11-05T08:33:15.660 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0025534889999999999 2023-11-05T08:33:15.660 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.660 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.660 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:15.661 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.153035+0000", 2023-11-05T08:33:15.661 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.6078000000000002e-05 2023-11-05T08:33:15.661 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.661 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.661 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:15.662 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.159134+0000", 2023-11-05T08:33:15.662 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0060986069999999998 2023-11-05T08:33:15.662 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.662 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.662 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:15.663 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.159208+0000", 2023-11-05T08:33:15.663 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.483e-05 2023-11-05T08:33:15.663 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.663 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:15.663 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.664 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.664 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.664 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as2 42/33 ECSubWrite(tid=40, reqid=osd.7.0:39, at_version=0'0, trim_to=0'0, roll_forward_to=42'18))", 2023-11-05T08:33:15.664 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:55.151263+0000", 2023-11-05T08:33:15.664 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 20.357477543999998, 2023-11-05T08:33:15.665 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0080059020000000005, 2023-11-05T08:33:15.665 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:15.665 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:15.665 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:15.665 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.666 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:15.666 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.151263+0000", 2023-11-05T08:33:15.666 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.666 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.666 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.666 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:15.667 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.151263+0000", 2023-11-05T08:33:15.667 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.667 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.667 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.667 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:15.668 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.151265+0000", 2023-11-05T08:33:15.668 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9939999999999999e-06 2023-11-05T08:33:15.668 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.668 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.668 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:15.668 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.151295+0000", 2023-11-05T08:33:15.669 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0355e-05 2023-11-05T08:33:15.669 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.669 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.669 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:15.669 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.151328+0000", 2023-11-05T08:33:15.670 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2246000000000003e-05 2023-11-05T08:33:15.670 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.670 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.670 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:15.670 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.151359+0000", 2023-11-05T08:33:15.670 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1458999999999998e-05 2023-11-05T08:33:15.671 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.671 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.671 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:15.671 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.153854+0000", 2023-11-05T08:33:15.671 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002495115 2023-11-05T08:33:15.672 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.672 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.672 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:15.672 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.153879+0000", 2023-11-05T08:33:15.672 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5065000000000001e-05 2023-11-05T08:33:15.672 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.673 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.673 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:15.673 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.159215+0000", 2023-11-05T08:33:15.673 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0053359690000000003 2023-11-05T08:33:15.673 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.674 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.674 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:15.674 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.159269+0000", 2023-11-05T08:33:15.674 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3699000000000002e-05 2023-11-05T08:33:15.674 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.675 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:15.675 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.675 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.675 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.675 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.9s1 52/45 ECSubWrite(tid=75, reqid=client.4691.0:341, at_version=52'24, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:15.676 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:05.257274+0000", 2023-11-05T08:33:15.676 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 10.251466974, 2023-11-05T08:33:15.676 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0067753529999999996, 2023-11-05T08:33:15.676 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:15.676 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:15.677 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:15.677 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.677 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:15.677 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.257274+0000", 2023-11-05T08:33:15.677 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.678 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.678 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.678 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:15.678 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.257274+0000", 2023-11-05T08:33:15.678 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.679 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.679 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.679 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:15.679 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.257275+0000", 2023-11-05T08:33:15.679 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1599999999999999e-06 2023-11-05T08:33:15.680 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.680 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.680 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:15.680 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.257860+0000", 2023-11-05T08:33:15.680 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00058503400000000001 2023-11-05T08:33:15.681 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.681 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.681 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:15.681 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.257898+0000", 2023-11-05T08:33:15.681 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8535000000000003e-05 2023-11-05T08:33:15.682 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.682 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.682 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:15.682 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.257924+0000", 2023-11-05T08:33:15.682 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5439000000000002e-05 2023-11-05T08:33:15.683 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.683 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.683 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:15.683 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258089+0000", 2023-11-05T08:33:15.683 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000165234 2023-11-05T08:33:15.684 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.684 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.684 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:15.684 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258118+0000", 2023-11-05T08:33:15.684 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9335999999999998e-05 2023-11-05T08:33:15.685 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.685 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.685 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:15.685 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.263959+0000", 2023-11-05T08:33:15.685 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0058405269999999999 2023-11-05T08:33:15.686 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.686 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.686 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:15.686 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.264049+0000", 2023-11-05T08:33:15.686 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.0087999999999995e-05 2023-11-05T08:33:15.687 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.687 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:15.687 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.687 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.687 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.688 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as2 52/45 ECSubWrite(tid=45, reqid=client.4691.0:349, at_version=52'21, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:15.688 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:05.262984+0000", 2023-11-05T08:33:15.688 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 10.245756901, 2023-11-05T08:33:15.688 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0076861730000000001, 2023-11-05T08:33:15.688 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:15.689 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:15.689 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:15.689 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.689 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:15.689 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.262984+0000", 2023-11-05T08:33:15.690 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.690 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.690 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.690 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:15.690 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.262984+0000", 2023-11-05T08:33:15.691 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.691 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.691 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.691 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:15.691 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.262985+0000", 2023-11-05T08:33:15.692 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.8400000000000002e-07 2023-11-05T08:33:15.692 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.692 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.692 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:15.692 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.263110+0000", 2023-11-05T08:33:15.692 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012501999999999999 2023-11-05T08:33:15.693 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.693 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.693 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:15.693 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.263149+0000", 2023-11-05T08:33:15.693 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8899000000000001e-05 2023-11-05T08:33:15.694 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.694 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.694 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:15.694 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.263173+0000", 2023-11-05T08:33:15.694 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4243000000000002e-05 2023-11-05T08:33:15.695 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.695 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.695 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:15.695 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.263816+0000", 2023-11-05T08:33:15.695 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00064262900000000003 2023-11-05T08:33:15.695 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.696 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.696 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:15.696 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.263830+0000", 2023-11-05T08:33:15.696 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4676e-05 2023-11-05T08:33:15.696 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.697 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.697 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:15.697 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.270569+0000", 2023-11-05T08:33:15.697 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0067386939999999999 2023-11-05T08:33:15.697 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.697 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.698 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:15.698 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.270670+0000", 2023-11-05T08:33:15.698 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000101028 2023-11-05T08:33:15.698 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.698 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:15.699 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.699 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.699 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.699 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.9s1 52/45 ECSubWrite(tid=77, reqid=client.4691.0:344, at_version=52'26, trim_to=0'0, roll_forward_to=52'24))", 2023-11-05T08:33:15.699 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:05.279896+0000", 2023-11-05T08:33:15.700 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 10.228844508, 2023-11-05T08:33:15.700 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0070183040000000004, 2023-11-05T08:33:15.700 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:15.700 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:15.700 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:15.701 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.701 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:15.701 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.279896+0000", 2023-11-05T08:33:15.701 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.701 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.702 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.702 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:15.702 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.279896+0000", 2023-11-05T08:33:15.702 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.702 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.703 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.703 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:15.703 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.279899+0000", 2023-11-05T08:33:15.703 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4830000000000002e-06 2023-11-05T08:33:15.704 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.704 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.704 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:15.704 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.280323+0000", 2023-11-05T08:33:15.704 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00042454999999999999 2023-11-05T08:33:15.705 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.705 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.705 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:15.705 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.280369+0000", 2023-11-05T08:33:15.706 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5469000000000001e-05 2023-11-05T08:33:15.706 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.706 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.706 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:15.706 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.280395+0000", 2023-11-05T08:33:15.707 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6023000000000001e-05 2023-11-05T08:33:15.707 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.707 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.707 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:15.708 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.280758+0000", 2023-11-05T08:33:15.708 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00036362500000000002 2023-11-05T08:33:15.708 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.708 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.708 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:15.709 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.280788+0000", 2023-11-05T08:33:15.709 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0207000000000001e-05 2023-11-05T08:33:15.709 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.709 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.709 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:15.710 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.286749+0000", 2023-11-05T08:33:15.710 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0059600709999999999 2023-11-05T08:33:15.710 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.710 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.711 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:15.711 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.286914+0000", 2023-11-05T08:33:15.711 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000165876 2023-11-05T08:33:15.711 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.711 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:15.712 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.712 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.712 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.712 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.14s3 56/48 ECSubWrite(tid=32, reqid=client.4691.0:374, at_version=56'19, trim_to=0'0, roll_forward_to=49'18))", 2023-11-05T08:33:15.712 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:09.317760+0000", 2023-11-05T08:33:15.713 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 6.1909804790000003, 2023-11-05T08:33:15.713 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0071132290000000004, 2023-11-05T08:33:15.713 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:15.713 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:15.713 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:15.714 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.714 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:15.714 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:09.317760+0000", 2023-11-05T08:33:15.714 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.715 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.715 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.715 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:15.715 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:09.317760+0000", 2023-11-05T08:33:15.715 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.716 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.716 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.716 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:15.716 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:09.317761+0000", 2023-11-05T08:33:15.716 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3300000000000003e-07 2023-11-05T08:33:15.716 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.717 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.717 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:15.717 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:09.318127+0000", 2023-11-05T08:33:15.717 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00036675999999999999 2023-11-05T08:33:15.717 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.718 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.718 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:15.718 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:09.318169+0000", 2023-11-05T08:33:15.718 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1888999999999998e-05 2023-11-05T08:33:15.718 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.719 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.719 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:15.719 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:09.318194+0000", 2023-11-05T08:33:15.719 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4256e-05 2023-11-05T08:33:15.719 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.720 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.720 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:15.720 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:09.318338+0000", 2023-11-05T08:33:15.720 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000144363 2023-11-05T08:33:15.720 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.720 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.721 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:15.721 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:09.318360+0000", 2023-11-05T08:33:15.721 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.2022999999999999e-05 2023-11-05T08:33:15.721 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.721 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.722 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:15.722 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:09.324756+0000", 2023-11-05T08:33:15.722 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0063961809999999999 2023-11-05T08:33:15.722 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.722 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.723 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:15.723 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:09.324873+0000", 2023-11-05T08:33:15.723 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000117324 2023-11-05T08:33:15.723 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.723 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:15.723 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.724 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.724 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.724 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as2 61/45 ECSubWrite(tid=57, reqid=client.4691.0:405, at_version=61'23, trim_to=0'0, roll_forward_to=52'21))", 2023-11-05T08:33:15.724 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:14.364843+0000", 2023-11-05T08:33:15.724 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 1.1438975949999999, 2023-11-05T08:33:15.725 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0067844369999999999, 2023-11-05T08:33:15.725 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:15.725 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:15.725 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:15.725 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.726 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:15.726 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.364843+0000", 2023-11-05T08:33:15.726 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.726 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.726 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.727 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:15.727 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.364843+0000", 2023-11-05T08:33:15.727 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.727 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.727 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.728 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:15.728 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.364843+0000", 2023-11-05T08:33:15.728 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8700000000000001e-07 2023-11-05T08:33:15.728 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.728 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.729 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:15.729 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.365545+0000", 2023-11-05T08:33:15.729 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00070121599999999995 2023-11-05T08:33:15.729 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.729 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.730 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:15.730 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.365585+0000", 2023-11-05T08:33:15.730 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.0456e-05 2023-11-05T08:33:15.730 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.730 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.731 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:15.731 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.365610+0000", 2023-11-05T08:33:15.731 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4485999999999999e-05 2023-11-05T08:33:15.731 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.731 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.732 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:15.732 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.365782+0000", 2023-11-05T08:33:15.732 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00017226700000000001 2023-11-05T08:33:15.732 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.732 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.733 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:15.733 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.365807+0000", 2023-11-05T08:33:15.733 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4922999999999998e-05 2023-11-05T08:33:15.733 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.733 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.734 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:15.734 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.371523+0000", 2023-11-05T08:33:15.734 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0057164599999999996 2023-11-05T08:33:15.734 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.734 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.735 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:15.735 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.371627+0000", 2023-11-05T08:33:15.735 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000104242 2023-11-05T08:33:15.735 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.735 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:15.736 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.736 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.736 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.736 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as2 61/45 ECSubWrite(tid=58, reqid=client.4691.0:406, at_version=61'24, trim_to=0'0, roll_forward_to=52'21))", 2023-11-05T08:33:15.736 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:14.371804+0000", 2023-11-05T08:33:15.737 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 1.136936561, 2023-11-05T08:33:15.737 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00575319, 2023-11-05T08:33:15.737 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:15.737 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:15.737 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:15.738 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.738 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:15.738 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.371804+0000", 2023-11-05T08:33:15.738 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.738 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.739 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.739 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:15.739 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.371804+0000", 2023-11-05T08:33:15.739 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:15.739 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.740 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.740 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:15.740 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.371806+0000", 2023-11-05T08:33:15.740 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4959999999999999e-06 2023-11-05T08:33:15.740 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.740 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.741 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:15.741 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.372335+0000", 2023-11-05T08:33:15.741 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00052975600000000004 2023-11-05T08:33:15.741 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.741 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.742 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:15.742 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.372370+0000", 2023-11-05T08:33:15.742 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4316000000000001e-05 2023-11-05T08:33:15.742 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.742 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.743 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:15.743 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.372392+0000", 2023-11-05T08:33:15.743 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.2442999999999998e-05 2023-11-05T08:33:15.743 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.743 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.743 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:15.744 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.372564+0000", 2023-11-05T08:33:15.744 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000171977 2023-11-05T08:33:15.744 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.744 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.744 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:15.745 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.372596+0000", 2023-11-05T08:33:15.745 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2082000000000002e-05 2023-11-05T08:33:15.745 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.745 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.745 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:15.745 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.377485+0000", 2023-11-05T08:33:15.746 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0048886909999999997 2023-11-05T08:33:15.746 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:15.746 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:15.746 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:15.746 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.377557+0000", 2023-11-05T08:33:15.747 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.2428999999999994e-05 2023-11-05T08:33:15.747 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.747 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:15.747 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.747 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:15.748 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:15.748 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:33:15.749 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-05T08:33:15.750 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:33:15.746+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:33:15.751 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-11-05T08:33:15.850 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:33:15.849+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:33:15.892 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-11-05T08:33:15.914 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1: {} 2023-11-05T08:33:15.914 INFO:teuthology.orchestra.run.smithi002.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-11-05T08:33:15.928 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:33:15.928 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2023-11-05T08:33:15.928 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 0 2023-11-05T08:33:15.928 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:33:15.934 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2: {} 2023-11-05T08:33:15.935 INFO:teuthology.orchestra.run.smithi002.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-11-05T08:33:15.935 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-11-05T08:33:15.942 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-11-05T08:33:15.951 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:33:15.950+0000 7fdc5cd58640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:33:15.952 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4: {} 2023-11-05T08:33:15.952 INFO:teuthology.orchestra.run.smithi002.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-11-05T08:33:15.972 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5: {} 2023-11-05T08:33:15.973 INFO:teuthology.orchestra.run.smithi002.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-11-05T08:33:15.995 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6: {} 2023-11-05T08:33:15.996 INFO:teuthology.orchestra.run.smithi002.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-11-05T08:33:16.018 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7: {} 2023-11-05T08:33:16.019 INFO:teuthology.orchestra.run.smithi002.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-11-05T08:33:16.039 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8: {} 2023-11-05T08:33:16.040 INFO:teuthology.orchestra.run.smithi002.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-11-05T08:33:16.052 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:16.047+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:16.063 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9: {} 2023-11-05T08:33:16.063 INFO:teuthology.orchestra.run.smithi002.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-11-05T08:33:16.087 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10: {} 2023-11-05T08:33:16.087 INFO:teuthology.orchestra.run.smithi002.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-11-05T08:33:16.112 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11: {} 2023-11-05T08:33:16.112 INFO:teuthology.orchestra.run.smithi002.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-11-05T08:33:16.115 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:16.115 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:16.129 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-11-05T08:33:16.134 INFO:teuthology.orchestra.run.smithi002.stdout:osd.12: {} 2023-11-05T08:33:16.135 INFO:teuthology.orchestra.run.smithi002.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-11-05T08:33:16.152 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:33:16.150+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:33:16.156 INFO:teuthology.orchestra.run.smithi002.stdout:osd.13: {} 2023-11-05T08:33:16.157 INFO:teuthology.orchestra.run.smithi002.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-11-05T08:33:16.181 INFO:teuthology.orchestra.run.smithi002.stdout:osd.14: {} 2023-11-05T08:33:16.181 INFO:teuthology.orchestra.run.smithi002.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-11-05T08:33:16.206 INFO:teuthology.orchestra.run.smithi002.stdout:osd.15: {} 2023-11-05T08:33:16.206 INFO:teuthology.orchestra.run.smithi002.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-11-05T08:33:16.212 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-05T08:33:16.213 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-05T08:33:16.251 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:16.251 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:16.253 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:33:16.250+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:33:16.265 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2023-11-05T08:33:16.334 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-11-05T08:33:16.337 INFO:teuthology.orchestra.run.smithi002.stderr:reweighted osd.1 to 0.959162 (f58b) 2023-11-05T08:33:16.345 INFO:tasks.rados.rados.0.smithi090.stdout:175: done (4 left) 2023-11-05T08:33:16.345 INFO:tasks.rados.rados.0.smithi090.stdout:176: rmattr oid 30 current snap is 16 2023-11-05T08:33:16.345 INFO:tasks.rados.rados.0.smithi090.stdout:170: finishing write tid 1 to smithi09039252-41 2023-11-05T08:33:16.345 INFO:tasks.rados.rados.0.smithi090.stdout:170: finishing write tid 2 to smithi09039252-41 2023-11-05T08:33:16.346 INFO:tasks.rados.rados.0.smithi090.stdout:170: finishing write tid 3 to smithi09039252-41 2023-11-05T08:33:16.346 INFO:tasks.rados.rados.0.smithi090.stdout:170: finishing write tid 4 to smithi09039252-41 2023-11-05T08:33:16.346 INFO:tasks.rados.rados.0.smithi090.stdout:170: oid 41 updating version 0 to 14 2023-11-05T08:33:16.346 INFO:tasks.rados.rados.0.smithi090.stdout:170: oid 41 version 14 is already newer than 13 2023-11-05T08:33:16.347 INFO:tasks.rados.rados.0.smithi090.stdout:170: oid 41 updating version 14 to 15 2023-11-05T08:33:16.347 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 41 v 15 (ObjNum 85 snap 16 seq_num 85) dirty exists 2023-11-05T08:33:16.347 INFO:tasks.rados.rados.0.smithi090.stdout:170: left oid 41 (ObjNum 85 snap 16 seq_num 85) 2023-11-05T08:33:16.348 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 11 v 30 (ObjNum 75 snap 9 seq_num 75) dirty exists 2023-11-05T08:33:16.348 INFO:tasks.rados.rados.0.smithi090.stdout:174: finishing write tid 1 to smithi09039252-5 2023-11-05T08:33:16.348 INFO:tasks.rados.rados.0.smithi090.stdout:174: finishing write tid 2 to smithi09039252-5 2023-11-05T08:33:16.348 INFO:tasks.rados.rados.0.smithi090.stdout:174: finishing write tid 3 to smithi09039252-5 2023-11-05T08:33:16.349 INFO:tasks.rados.rados.0.smithi090.stdout:174: finishing write tid 4 to smithi09039252-5 2023-11-05T08:33:16.349 INFO:tasks.rados.rados.0.smithi090.stdout:174: finishing write tid 5 to smithi09039252-5 2023-11-05T08:33:16.349 INFO:tasks.rados.rados.0.smithi090.stdout:174: oid 5 updating version 0 to 14 2023-11-05T08:33:16.349 INFO:tasks.rados.rados.0.smithi090.stdout:174: oid 5 version 14 is already newer than 11 2023-11-05T08:33:16.350 INFO:tasks.rados.rados.0.smithi090.stdout:174: oid 5 version 14 is already newer than 13 2023-11-05T08:33:16.350 INFO:tasks.rados.rados.0.smithi090.stdout:174: oid 5 version 14 is already newer than 12 2023-11-05T08:33:16.350 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 5 v 14 (ObjNum 86 snap 16 seq_num 86) dirty exists 2023-11-05T08:33:16.350 INFO:tasks.rados.rados.0.smithi090.stdout:174: left oid 5 (ObjNum 86 snap 16 seq_num 86) 2023-11-05T08:33:16.350 INFO:tasks.rados.rados.0.smithi090.stdout:170: done (4 left) 2023-11-05T08:33:16.351 INFO:tasks.rados.rados.0.smithi090.stdout:173: done (3 left) 2023-11-05T08:33:16.351 INFO:tasks.rados.rados.0.smithi090.stdout:174: done (2 left) 2023-11-05T08:33:16.351 INFO:tasks.rados.rados.0.smithi090.stdout:176: done (1 left) 2023-11-05T08:33:16.352 INFO:tasks.rados.rados.0.smithi090.stdout:177: copy_from oid 29 from oid 19 current snap is 16 2023-11-05T08:33:16.352 INFO:tasks.rados.rados.0.smithi090.stdout:178: read oid 39 snap -1 2023-11-05T08:33:16.352 INFO:tasks.rados.rados.0.smithi090.stdout:178: expect (ObjNum 38 snap 0 seq_num 38) 2023-11-05T08:33:16.353 INFO:tasks.rados.rados.0.smithi090.stdout:179: read oid 48 snap -1 2023-11-05T08:33:16.353 INFO:tasks.rados.rados.0.smithi090.stdout:179: expect (ObjNum 47 snap 0 seq_num 47) 2023-11-05T08:33:16.354 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 31 current snap is 16 2023-11-05T08:33:16.354 INFO:tasks.rados.rados.0.smithi090.stdout:180: seq_num 87 ranges {1294336=704512,1998848=688128,2686976=81920} 2023-11-05T08:33:16.355 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:33:16.354+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:33:16.368 INFO:tasks.rados.rados.0.smithi090.stdout:180: writing smithi09039252-31 from 1294336 to 1998848 tid 1 2023-11-05T08:33:16.377 INFO:tasks.rados.rados.0.smithi090.stdout:180: writing smithi09039252-31 from 1998848 to 2686976 tid 2 2023-11-05T08:33:16.379 INFO:tasks.rados.rados.0.smithi090.stdout:180: writing smithi09039252-31 from 2686976 to 2768896 tid 3 2023-11-05T08:33:16.379 INFO:tasks.rados.rados.0.smithi090.stdout:179: expect (ObjNum 47 snap 0 seq_num 47) 2023-11-05T08:33:16.404 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:16.404 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:16.414 INFO:tasks.rados.rados.0.smithi090.stdout:178: expect (ObjNum 38 snap 0 seq_num 38) 2023-11-05T08:33:16.418 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2023-11-05T08:33:16.442 INFO:tasks.rados.rados.0.smithi090.stdout:177: finishing copy_from racing read to smithi09039252-29 2023-11-05T08:33:16.442 INFO:tasks.rados.rados.0.smithi090.stdout:178: done (4 left) 2023-11-05T08:33:16.442 INFO:tasks.rados.rados.0.smithi090.stdout:179: done (3 left) 2023-11-05T08:33:16.443 INFO:tasks.rados.rados.0.smithi090.stdout:181: rmattr oid 22 current snap is 16 2023-11-05T08:33:16.443 INFO:tasks.rados.rados.0.smithi090.stdout:181: done (3 left) 2023-11-05T08:33:16.443 INFO:tasks.rados.rados.0.smithi090.stdout:182: rollback oid 15 current snap is 16 2023-11-05T08:33:16.443 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 15 to 13 2023-11-05T08:33:16.444 INFO:tasks.rados.rados.0.smithi090.stdout:177: finishing copy_from to smithi09039252-29 2023-11-05T08:33:16.444 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 29 v 2 (ObjNum 18 snap 0 seq_num 18) dirty exists 2023-11-05T08:33:16.444 INFO:tasks.rados.rados.0.smithi090.stdout:180: finishing write tid 1 to smithi09039252-31 2023-11-05T08:33:16.444 INFO:tasks.rados.rados.0.smithi090.stdout:180: finishing write tid 2 to smithi09039252-31 2023-11-05T08:33:16.444 INFO:tasks.rados.rados.0.smithi090.stdout:180: finishing write tid 3 to smithi09039252-31 2023-11-05T08:33:16.445 INFO:tasks.rados.rados.0.smithi090.stdout:180: finishing write tid 4 to smithi09039252-31 2023-11-05T08:33:16.445 INFO:tasks.rados.rados.0.smithi090.stdout:180: finishing write tid 5 to smithi09039252-31 2023-11-05T08:33:16.445 INFO:tasks.rados.rados.0.smithi090.stdout:180: oid 31 updating version 0 to 25 2023-11-05T08:33:16.445 INFO:tasks.rados.rados.0.smithi090.stdout:180: oid 31 updating version 25 to 28 2023-11-05T08:33:16.445 INFO:tasks.rados.rados.0.smithi090.stdout:180: oid 31 version 28 is already newer than 26 2023-11-05T08:33:16.446 INFO:tasks.rados.rados.0.smithi090.stdout:180: oid 31 version 28 is already newer than 27 2023-11-05T08:33:16.446 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 31 v 28 (ObjNum 87 snap 16 seq_num 87) dirty exists 2023-11-05T08:33:16.446 INFO:tasks.rados.rados.0.smithi090.stdout:180: left oid 31 (ObjNum 87 snap 16 seq_num 87) 2023-11-05T08:33:16.446 INFO:tasks.rados.rados.0.smithi090.stdout:177: done (3 left) 2023-11-05T08:33:16.446 INFO:tasks.rados.rados.0.smithi090.stdout:180: done (2 left) 2023-11-05T08:33:16.447 INFO:tasks.rados.rados.0.smithi090.stdout:183: delete oid 14 current snap is 16 2023-11-05T08:33:16.449 INFO:tasks.rados.rados.0.smithi090.stdout:183: done (2 left) 2023-11-05T08:33:16.449 INFO:tasks.rados.rados.0.smithi090.stdout:184: snap_remove snap 14 2023-11-05T08:33:16.456 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:33:16.454+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:33:16.542 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:16.542 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:16.556 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2023-11-05T08:33:16.558 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:16.551+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:16.659 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:33:16.658+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:33:16.678 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:16.678 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:16.694 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2023-11-05T08:33:16.760 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:33:16.758+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:33:16.822 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:16.823 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:16.837 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2023-11-05T08:33:16.861 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:33:16.859+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:33:16.960 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:16.960 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:16.962 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:33:16.958+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:33:16.974 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2023-11-05T08:33:17.063 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:17.060+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:17.092 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:17.093 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:17.107 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-11-05T08:33:17.164 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:17.163+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:17.239 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:17.240 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:17.253 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-11-05T08:33:17.265 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:33:17.263+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:33:17.347 INFO:teuthology.orchestra.run.smithi002.stderr:noscrub is set 2023-11-05T08:33:17.353 INFO:tasks.rados.rados.0.smithi090.stdout:184: done (2 left) 2023-11-05T08:33:17.354 INFO:tasks.rados.rados.0.smithi090.stdout:185: read oid 9 snap -1 2023-11-05T08:33:17.354 INFO:tasks.rados.rados.0.smithi090.stdout:185: expect (ObjNum 6 snap 0 seq_num 6) 2023-11-05T08:33:17.354 INFO:tasks.rados.rados.0.smithi090.stdout:186: snap_remove snap 11 2023-11-05T08:33:17.366 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:17.359+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:17.372 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:17.372 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:17.391 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-11-05T08:33:17.467 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:33:17.466+0000 7fdc5cd58640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:33:17.507 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:17.507 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:17.521 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-11-05T08:33:17.568 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:33:17.563+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:33:17.644 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:17.644 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:17.658 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-11-05T08:33:17.668 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~22s 2023-11-05T08:33:17.669 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:33:17.664+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:33:17.770 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:33:17.766+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:33:17.775 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:17.775 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:17.790 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-11-05T08:33:17.870 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:33:17.866+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:33:17.956 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:17.956 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:17.969 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2023-11-05T08:33:17.970 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-05T08:33:18.072 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:18.071+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:18.093 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:18.093 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:18.105 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2023-11-05T08:33:18.173 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:33:18.167+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:33:18.214 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-05T08:33:18.226 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:18.226 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:18.240 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2023-11-05T08:33:18.283 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:33:18.282+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:33:18.364 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-11-05T08:33:18.368 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:18.368 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:18.370 INFO:tasks.rados.rados.0.smithi090.stdout:186: done (3 left) 2023-11-05T08:33:18.370 INFO:tasks.rados.rados.0.smithi090.stdout:187: read oid 16 snap -1 2023-11-05T08:33:18.371 INFO:tasks.rados.rados.0.smithi090.stdout:187: expect (ObjNum 68 snap 6 seq_num 68) 2023-11-05T08:33:18.371 INFO:tasks.rados.rados.0.smithi090.stdout:188: read oid 27 snap -1 2023-11-05T08:33:18.371 INFO:tasks.rados.rados.0.smithi090.stdout:188: expect (ObjNum 26 snap 0 seq_num 26) 2023-11-05T08:33:18.371 INFO:tasks.rados.rados.0.smithi090.stdout:189: read oid 25 snap -1 2023-11-05T08:33:18.372 INFO:tasks.rados.rados.0.smithi090.stdout:189: expect (ObjNum 52 snap 0 seq_num 52) 2023-11-05T08:33:18.372 INFO:tasks.rados.rados.0.smithi090.stdout:190: rollback oid 34 current snap is 16 2023-11-05T08:33:18.372 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 34 to 13 2023-11-05T08:33:18.372 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 26 current snap is 16 2023-11-05T08:33:18.372 INFO:tasks.rados.rados.0.smithi090.stdout:191: seq_num 89 ranges {425984=737280,1163264=573440,1736704=196608} 2023-11-05T08:33:18.374 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-05T08:33:18.390 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1: {} 2023-11-05T08:33:18.390 INFO:teuthology.orchestra.run.smithi002.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-11-05T08:33:18.392 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2023-11-05T08:33:18.392 INFO:tasks.rados.rados.0.smithi090.stdout:191: writing smithi09039252-26 from 425984 to 1163264 tid 1 2023-11-05T08:33:18.403 INFO:tasks.rados.rados.0.smithi090.stdout:191: writing smithi09039252-26 from 1163264 to 1736704 tid 2 2023-11-05T08:33:18.407 INFO:tasks.rados.rados.0.smithi090.stdout:191: writing smithi09039252-26 from 1736704 to 1933312 tid 3 2023-11-05T08:33:18.407 INFO:tasks.rados.rados.0.smithi090.stdout:192: delete oid 29 current snap is 16 2023-11-05T08:33:18.407 INFO:tasks.rados.rados.0.smithi090.stdout:182: finishing rollback tid 0 to smithi09039252-15 2023-11-05T08:33:18.407 INFO:tasks.rados.rados.0.smithi090.stdout:182: finishing rollback tid 1 to smithi09039252-15 2023-11-05T08:33:18.408 INFO:tasks.rados.rados.0.smithi090.stdout:182: finishing rollback tid 2 to smithi09039252-15 2023-11-05T08:33:18.408 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 15 v 30 (ObjNum 17 snap 0 seq_num 17) dirty exists 2023-11-05T08:33:18.408 INFO:tasks.rados.rados.0.smithi090.stdout:185: expect (ObjNum 6 snap 0 seq_num 6) 2023-11-05T08:33:18.410 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2: {} 2023-11-05T08:33:18.410 INFO:teuthology.orchestra.run.smithi002.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-11-05T08:33:18.411 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-11-05T08:33:18.418 INFO:tasks.rados.rados.0.smithi090.stdout:188: expect (ObjNum 26 snap 0 seq_num 26) 2023-11-05T08:33:18.428 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4: {} 2023-11-05T08:33:18.429 INFO:teuthology.orchestra.run.smithi002.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-11-05T08:33:18.447 INFO:tasks.rados.rados.0.smithi090.stdout:187: expect (ObjNum 68 snap 6 seq_num 68) 2023-11-05T08:33:18.448 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5: {} 2023-11-05T08:33:18.449 INFO:teuthology.orchestra.run.smithi002.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-11-05T08:33:18.465 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6: {} 2023-11-05T08:33:18.465 INFO:teuthology.orchestra.run.smithi002.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-11-05T08:33:18.476 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:33:18.471+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:33:18.485 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7: {} 2023-11-05T08:33:18.485 INFO:teuthology.orchestra.run.smithi002.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-11-05T08:33:18.502 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8: {} 2023-11-05T08:33:18.503 INFO:teuthology.orchestra.run.smithi002.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-11-05T08:33:18.506 INFO:tasks.rados.rados.0.smithi090.stdout:189: expect (ObjNum 52 snap 0 seq_num 52) 2023-11-05T08:33:18.523 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9: {} 2023-11-05T08:33:18.523 INFO:teuthology.orchestra.run.smithi002.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-11-05T08:33:18.544 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10: {} 2023-11-05T08:33:18.544 INFO:teuthology.orchestra.run.smithi002.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-11-05T08:33:18.545 INFO:tasks.rados.rados.0.smithi090.stdout:190: finishing rollback tid 0 to smithi09039252-34 2023-11-05T08:33:18.545 INFO:tasks.rados.rados.0.smithi090.stdout:190: finishing rollback tid 1 to smithi09039252-34 2023-11-05T08:33:18.546 INFO:tasks.rados.rados.0.smithi090.stdout:182: done (8 left) 2023-11-05T08:33:18.546 INFO:tasks.rados.rados.0.smithi090.stdout:185: done (7 left) 2023-11-05T08:33:18.546 INFO:tasks.rados.rados.0.smithi090.stdout:187: done (6 left) 2023-11-05T08:33:18.546 INFO:tasks.rados.rados.0.smithi090.stdout:188: done (5 left) 2023-11-05T08:33:18.546 INFO:tasks.rados.rados.0.smithi090.stdout:189: done (4 left) 2023-11-05T08:33:18.547 INFO:tasks.rados.rados.0.smithi090.stdout:192: done (3 left) 2023-11-05T08:33:18.547 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 50 current snap is 16 2023-11-05T08:33:18.547 INFO:tasks.rados.rados.0.smithi090.stdout:190: finishing rollback tid 2 to smithi09039252-34 2023-11-05T08:33:18.547 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 34 v 61 (ObjNum 79 snap 13 seq_num 79) dirty exists 2023-11-05T08:33:18.547 INFO:tasks.rados.rados.0.smithi090.stdout:191: finishing write tid 1 to smithi09039252-26 2023-11-05T08:33:18.548 INFO:tasks.rados.rados.0.smithi090.stdout:191: finishing write tid 2 to smithi09039252-26 2023-11-05T08:33:18.548 INFO:tasks.rados.rados.0.smithi090.stdout:193: seq_num 91 ranges {0=770048,770048=475136,1245184=294912} 2023-11-05T08:33:18.549 INFO:tasks.rados.rados.0.smithi090.stdout:193: writing smithi09039252-50 from 0 to 770048 tid 1 2023-11-05T08:33:18.552 INFO:tasks.rados.rados.0.smithi090.stdout:193: writing smithi09039252-50 from 770048 to 1245184 tid 2 2023-11-05T08:33:18.554 INFO:tasks.rados.rados.0.smithi090.stdout:193: writing smithi09039252-50 from 1245184 to 1540096 tid 3 2023-11-05T08:33:18.555 INFO:tasks.rados.rados.0.smithi090.stdout:190: done (3 left) 2023-11-05T08:33:18.555 INFO:tasks.rados.rados.0.smithi090.stdout:194: read oid 7 snap -1 2023-11-05T08:33:18.555 INFO:tasks.rados.rados.0.smithi090.stdout:194: expect deleted 2023-11-05T08:33:18.555 INFO:tasks.rados.rados.0.smithi090.stdout:195: read oid 13 snap 4 2023-11-05T08:33:18.555 INFO:tasks.rados.rados.0.smithi090.stdout:195: expect (ObjNum 50 snap 0 seq_num 50) 2023-11-05T08:33:18.556 INFO:tasks.rados.rados.0.smithi090.stdout:196: read oid 16 snap 4 2023-11-05T08:33:18.556 INFO:tasks.rados.rados.0.smithi090.stdout:196: expect (ObjNum 15 snap 0 seq_num 15) 2023-11-05T08:33:18.556 INFO:tasks.rados.rados.0.smithi090.stdout:197: delete oid 25 current snap is 16 2023-11-05T08:33:18.556 INFO:tasks.rados.rados.0.smithi090.stdout:191: finishing write tid 3 to smithi09039252-26 2023-11-05T08:33:18.556 INFO:tasks.rados.rados.0.smithi090.stdout:191: finishing write tid 4 to smithi09039252-26 2023-11-05T08:33:18.557 INFO:tasks.rados.rados.0.smithi090.stdout:191: finishing write tid 5 to smithi09039252-26 2023-11-05T08:33:18.557 INFO:tasks.rados.rados.0.smithi090.stdout:191: oid 26 updating version 0 to 33 2023-11-05T08:33:18.557 INFO:tasks.rados.rados.0.smithi090.stdout:191: oid 26 updating version 33 to 34 2023-11-05T08:33:18.557 INFO:tasks.rados.rados.0.smithi090.stdout:191: oid 26 version 34 is already newer than 32 2023-11-05T08:33:18.557 INFO:tasks.rados.rados.0.smithi090.stdout:191: oid 26 version 34 is already newer than 31 2023-11-05T08:33:18.557 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 26 v 34 (ObjNum 89 snap 16 seq_num 89) dirty exists 2023-11-05T08:33:18.558 INFO:tasks.rados.rados.0.smithi090.stdout:191: left oid 26 (ObjNum 89 snap 16 seq_num 89) 2023-11-05T08:33:18.558 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:18.558 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:18.564 INFO:tasks.rados.rados.0.smithi090.stdout:191: done (6 left) 2023-11-05T08:33:18.564 INFO:tasks.rados.rados.0.smithi090.stdout:197: done (5 left) 2023-11-05T08:33:18.564 INFO:tasks.rados.rados.0.smithi090.stdout:198: rollback oid 41 current snap is 16 2023-11-05T08:33:18.564 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 41 to 13 2023-11-05T08:33:18.564 INFO:tasks.rados.rados.0.smithi090.stdout:199: rmattr oid 37 current snap is 16 2023-11-05T08:33:18.565 INFO:tasks.rados.rados.0.smithi090.stdout:199: done (6 left) 2023-11-05T08:33:18.565 INFO:tasks.rados.rados.0.smithi090.stdout:200: rmattr oid 30 current snap is 16 2023-11-05T08:33:18.565 INFO:tasks.rados.rados.0.smithi090.stdout:200: done (6 left) 2023-11-05T08:33:18.565 INFO:tasks.rados.rados.0.smithi090.stdout:201: setattr oid 18 current snap is 16 2023-11-05T08:33:18.565 INFO:tasks.rados.rados.0.smithi090.stdout:196: expect (ObjNum 15 snap 0 seq_num 15) 2023-11-05T08:33:18.569 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11: {} 2023-11-05T08:33:18.569 INFO:teuthology.orchestra.run.smithi002.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-11-05T08:33:18.571 INFO:tasks.rados.rados.0.smithi090.stdout:195: expect (ObjNum 50 snap 0 seq_num 50) 2023-11-05T08:33:18.577 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:18.571+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:18.578 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2023-11-05T08:33:18.588 INFO:teuthology.orchestra.run.smithi002.stdout:osd.12: {} 2023-11-05T08:33:18.588 INFO:teuthology.orchestra.run.smithi002.stderr:osd.12: osd_enable_op_tracker = 'true' 2023-11-05T08:33:18.603 INFO:tasks.rados.rados.0.smithi090.stdout:193: finishing write tid 1 to smithi09039252-50 2023-11-05T08:33:18.603 INFO:tasks.rados.rados.0.smithi090.stdout:198: finishing rollback tid 0 to smithi09039252-41 2023-11-05T08:33:18.603 INFO:tasks.rados.rados.0.smithi090.stdout:198: finishing rollback tid 1 to smithi09039252-41 2023-11-05T08:33:18.603 INFO:tasks.rados.rados.0.smithi090.stdout:198: finishing rollback tid 2 to smithi09039252-41 2023-11-05T08:33:18.603 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 41 v 24 (ObjNum 40 snap 0 seq_num 40) dirty exists 2023-11-05T08:33:18.604 INFO:tasks.rados.rados.0.smithi090.stdout:193: finishing write tid 2 to smithi09039252-50 2023-11-05T08:33:18.604 INFO:tasks.rados.rados.0.smithi090.stdout:193: finishing write tid 3 to smithi09039252-50 2023-11-05T08:33:18.604 INFO:tasks.rados.rados.0.smithi090.stdout:193: finishing write tid 4 to smithi09039252-50 2023-11-05T08:33:18.604 INFO:tasks.rados.rados.0.smithi090.stdout:193: finishing write tid 5 to smithi09039252-50 2023-11-05T08:33:18.604 INFO:tasks.rados.rados.0.smithi090.stdout:193: oid 50 updating version 0 to 20 2023-11-05T08:33:18.604 INFO:tasks.rados.rados.0.smithi090.stdout:193: oid 50 updating version 20 to 21 2023-11-05T08:33:18.605 INFO:tasks.rados.rados.0.smithi090.stdout:193: oid 50 version 21 is already newer than 19 2023-11-05T08:33:18.605 INFO:tasks.rados.rados.0.smithi090.stdout:193: oid 50 version 21 is already newer than 18 2023-11-05T08:33:18.605 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 50 v 21 (ObjNum 91 snap 16 seq_num 91) dirty exists 2023-11-05T08:33:18.605 INFO:tasks.rados.rados.0.smithi090.stdout:193: left oid 50 (ObjNum 91 snap 16 seq_num 91) 2023-11-05T08:33:18.605 INFO:tasks.rados.rados.0.smithi090.stdout:193: done (6 left) 2023-11-05T08:33:18.606 INFO:tasks.rados.rados.0.smithi090.stdout:194: done (5 left) 2023-11-05T08:33:18.606 INFO:tasks.rados.rados.0.smithi090.stdout:195: done (4 left) 2023-11-05T08:33:18.606 INFO:tasks.rados.rados.0.smithi090.stdout:196: done (3 left) 2023-11-05T08:33:18.606 INFO:tasks.rados.rados.0.smithi090.stdout:198: done (2 left) 2023-11-05T08:33:18.606 INFO:tasks.rados.rados.0.smithi090.stdout:202: setattr oid 13 current snap is 16 2023-11-05T08:33:18.607 INFO:tasks.rados.rados.0.smithi090.stdout:203: read oid 39 snap -1 2023-11-05T08:33:18.607 INFO:tasks.rados.rados.0.smithi090.stdout:203: expect (ObjNum 38 snap 0 seq_num 38) 2023-11-05T08:33:18.607 INFO:tasks.rados.rados.0.smithi090.stdout:204: copy_from oid 46 from oid 40 current snap is 16 2023-11-05T08:33:18.608 INFO:tasks.rados.rados.0.smithi090.stdout:205: snap_remove snap 13 2023-11-05T08:33:18.612 INFO:teuthology.orchestra.run.smithi002.stdout:osd.13: {} 2023-11-05T08:33:18.613 INFO:teuthology.orchestra.run.smithi002.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-11-05T08:33:18.631 INFO:teuthology.orchestra.run.smithi002.stdout:osd.14: {} 2023-11-05T08:33:18.632 INFO:teuthology.orchestra.run.smithi002.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-11-05T08:33:18.652 INFO:teuthology.orchestra.run.smithi002.stdout:osd.15: {} 2023-11-05T08:33:18.653 INFO:teuthology.orchestra.run.smithi002.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-11-05T08:33:18.659 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-05T08:33:18.659 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-05T08:33:18.678 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:33:18.671+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:33:18.696 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:33:18.696 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2023-11-05T08:33:18.697 INFO:teuthology.orchestra.run.smithi069.stdout: "complaint_time": 30, 2023-11-05T08:33:18.697 INFO:teuthology.orchestra.run.smithi069.stdout: "num_blocked_ops": 0 2023-11-05T08:33:18.697 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:33:18.710 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2023-11-05T08:33:18.779 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:33:18.775+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:33:18.834 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:33:18.834 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 20, 2023-11-05T08:33:18.834 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 600, 2023-11-05T08:33:18.834 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [ 2023-11-05T08:33:18.835 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.835 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.ds2 36/30 ECSubWrite(tid=23, reqid=osd.7.0:23, at_version=36'14, trim_to=0'0, roll_forward_to=34'8))", 2023-11-05T08:33:18.835 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.078426+0000", 2023-11-05T08:33:18.835 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 29.753036695999999, 2023-11-05T08:33:18.835 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0071789940000000002, 2023-11-05T08:33:18.836 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:18.836 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:18.836 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:18.836 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.836 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:18.837 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078426+0000", 2023-11-05T08:33:18.837 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:18.837 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.837 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.837 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:18.838 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078426+0000", 2023-11-05T08:33:18.838 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:18.838 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.838 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.838 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:18.839 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078427+0000", 2023-11-05T08:33:18.839 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.093e-06 2023-11-05T08:33:18.839 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.839 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.839 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:18.840 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078484+0000", 2023-11-05T08:33:18.840 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.7240000000000001e-05 2023-11-05T08:33:18.840 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.840 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.840 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:18.840 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078508+0000", 2023-11-05T08:33:18.841 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4032e-05 2023-11-05T08:33:18.841 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.841 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.841 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:18.841 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078527+0000", 2023-11-05T08:33:18.841 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8754999999999999e-05 2023-11-05T08:33:18.842 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.842 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.842 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:18.842 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079037+0000", 2023-11-05T08:33:18.842 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000509722 2023-11-05T08:33:18.843 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.843 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.843 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:18.843 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079059+0000", 2023-11-05T08:33:18.843 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2467999999999999e-05 2023-11-05T08:33:18.844 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.844 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.844 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:18.844 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.085512+0000", 2023-11-05T08:33:18.844 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0064529059999999996 2023-11-05T08:33:18.845 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.845 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.845 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:18.845 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.085605+0000", 2023-11-05T08:33:18.845 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.2777999999999998e-05 2023-11-05T08:33:18.845 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:18.846 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:18.846 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:18.846 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.846 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.846 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.4s5 36/30 ECSubWrite(tid=10, reqid=osd.8.0:10, at_version=36'8, trim_to=0'0, roll_forward_to=34'6))", 2023-11-05T08:33:18.847 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.078716+0000", 2023-11-05T08:33:18.847 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 29.752746910999999, 2023-11-05T08:33:18.847 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.006931782, 2023-11-05T08:33:18.847 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:18.847 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:18.848 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:18.848 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.848 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:18.848 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078716+0000", 2023-11-05T08:33:18.848 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:18.848 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.849 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.849 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:18.849 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078716+0000", 2023-11-05T08:33:18.849 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:18.849 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.849 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.850 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:18.850 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078716+0000", 2023-11-05T08:33:18.850 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.0399999999999995e-07 2023-11-05T08:33:18.850 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.850 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.851 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:18.851 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078773+0000", 2023-11-05T08:33:18.851 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.6149000000000001e-05 2023-11-05T08:33:18.851 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.851 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.852 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:18.852 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078804+0000", 2023-11-05T08:33:18.852 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.1677999999999997e-05 2023-11-05T08:33:18.852 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.852 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.853 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:18.853 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078827+0000", 2023-11-05T08:33:18.853 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2469000000000001e-05 2023-11-05T08:33:18.853 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.853 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.853 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:18.854 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079007+0000", 2023-11-05T08:33:18.854 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000180242 2023-11-05T08:33:18.854 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.854 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.854 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:18.855 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079032+0000", 2023-11-05T08:33:18.855 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4915000000000001e-05 2023-11-05T08:33:18.855 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.855 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.855 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:18.856 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.085566+0000", 2023-11-05T08:33:18.856 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0065336140000000001 2023-11-05T08:33:18.856 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.856 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.856 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:18.857 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.085648+0000", 2023-11-05T08:33:18.857 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.2010999999999998e-05 2023-11-05T08:33:18.857 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:18.857 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:18.857 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:18.857 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.858 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.858 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.8s4 36/30 ECSubWrite(tid=23, reqid=osd.5.0:23, at_version=36'15, trim_to=0'0, roll_forward_to=34'13))", 2023-11-05T08:33:18.858 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.079985+0000", 2023-11-05T08:33:18.858 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 29.751477896000001, 2023-11-05T08:33:18.858 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.007626023, 2023-11-05T08:33:18.859 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:18.859 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:18.859 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:18.859 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.859 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:18.860 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079985+0000", 2023-11-05T08:33:18.860 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:18.860 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.860 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.860 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:18.861 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079985+0000", 2023-11-05T08:33:18.861 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:18.861 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.861 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.861 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:18.861 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079985+0000", 2023-11-05T08:33:18.862 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.6300000000000005e-07 2023-11-05T08:33:18.862 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.862 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.862 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:18.862 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.080040+0000", 2023-11-05T08:33:18.863 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.4796999999999997e-05 2023-11-05T08:33:18.863 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.863 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.863 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:18.863 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.080073+0000", 2023-11-05T08:33:18.864 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.3034999999999998e-05 2023-11-05T08:33:18.864 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.864 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.864 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:18.864 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.080095+0000", 2023-11-05T08:33:18.865 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1841999999999999e-05 2023-11-05T08:33:18.865 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.865 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.865 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:18.865 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.080255+0000", 2023-11-05T08:33:18.866 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00016012700000000001 2023-11-05T08:33:18.866 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.866 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.866 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:18.866 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.080280+0000", 2023-11-05T08:33:18.867 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4953999999999999e-05 2023-11-05T08:33:18.867 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.867 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.867 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:18.867 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.087517+0000", 2023-11-05T08:33:18.868 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0072370799999999999 2023-11-05T08:33:18.868 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.868 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.868 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:18.868 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.087611+0000", 2023-11-05T08:33:18.869 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.3524999999999996e-05 2023-11-05T08:33:18.869 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:18.869 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:18.869 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:18.869 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.870 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.870 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.ds2 36/30 ECSubWrite(tid=24, reqid=osd.7.0:23, at_version=0'0, trim_to=0'0, roll_forward_to=36'14))", 2023-11-05T08:33:18.870 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.089710+0000", 2023-11-05T08:33:18.870 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 29.741752565999999, 2023-11-05T08:33:18.870 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0067357069999999996, 2023-11-05T08:33:18.871 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:18.871 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:18.871 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:18.871 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.871 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:18.872 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089710+0000", 2023-11-05T08:33:18.872 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:18.872 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.872 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.872 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:18.872 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089710+0000", 2023-11-05T08:33:18.873 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:18.873 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.873 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.873 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:18.873 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089711+0000", 2023-11-05T08:33:18.874 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1429999999999999e-06 2023-11-05T08:33:18.874 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.874 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.874 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:18.874 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089733+0000", 2023-11-05T08:33:18.875 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1559000000000001e-05 2023-11-05T08:33:18.875 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.875 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.875 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:18.875 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089757+0000", 2023-11-05T08:33:18.876 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4515000000000002e-05 2023-11-05T08:33:18.876 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.876 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.876 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:18.876 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089780+0000", 2023-11-05T08:33:18.876 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2789000000000002e-05 2023-11-05T08:33:18.877 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.877 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.877 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:18.877 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.090049+0000", 2023-11-05T08:33:18.877 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000269363 2023-11-05T08:33:18.878 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.878 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.878 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:18.878 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.090074+0000", 2023-11-05T08:33:18.878 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4071999999999999e-05 2023-11-05T08:33:18.879 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.879 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.879 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:18.879 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.096355+0000", 2023-11-05T08:33:18.879 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0062814489999999997 2023-11-05T08:33:18.879 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.880 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.880 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:18.880 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.096446+0000", 2023-11-05T08:33:18.880 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.0816999999999995e-05 2023-11-05T08:33:18.880 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:18.881 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:18.881 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:18.881 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.881 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.881 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.8s4 36/30 ECSubWrite(tid=24, reqid=osd.5.0:23, at_version=0'0, trim_to=0'0, roll_forward_to=36'15))", 2023-11-05T08:33:18.881 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.089889+0000", 2023-11-05T08:33:18.882 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 29.741574156999999, 2023-11-05T08:33:18.882 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0065355869999999998, 2023-11-05T08:33:18.882 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:18.882 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:18.882 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:18.882 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.883 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:18.883 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089889+0000", 2023-11-05T08:33:18.883 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:18.883 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.883 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.884 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:18.884 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089889+0000", 2023-11-05T08:33:18.884 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:18.884 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.884 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.885 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:18.885 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089890+0000", 2023-11-05T08:33:18.885 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1599999999999999e-06 2023-11-05T08:33:18.885 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.885 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.885 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:18.886 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089908+0000", 2023-11-05T08:33:18.886 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8448999999999999e-05 2023-11-05T08:33:18.886 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.886 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.886 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:18.887 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089929+0000", 2023-11-05T08:33:18.887 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0534999999999999e-05 2023-11-05T08:33:18.887 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.887 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.887 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:18.888 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089943+0000", 2023-11-05T08:33:18.888 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4469e-05 2023-11-05T08:33:18.888 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.888 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.888 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:18.888 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.090098+0000", 2023-11-05T08:33:18.889 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00015513699999999999 2023-11-05T08:33:18.889 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.889 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.889 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:18.889 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.090114+0000", 2023-11-05T08:33:18.890 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5962999999999999e-05 2023-11-05T08:33:18.890 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.890 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.890 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:18.890 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.096334+0000", 2023-11-05T08:33:18.891 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0062193320000000002 2023-11-05T08:33:18.891 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.891 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.891 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:18.891 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.096424+0000", 2023-11-05T08:33:18.892 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.0542000000000001e-05 2023-11-05T08:33:18.892 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:18.892 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:18.892 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:18.892 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.892 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.893 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.as4 41/33 ECSubWrite(tid=35, reqid=osd.7.0:35, at_version=41'12, trim_to=0'0, roll_forward_to=40'10))", 2023-11-05T08:33:18.893 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.128988+0000", 2023-11-05T08:33:18.893 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 24.702474554999998, 2023-11-05T08:33:18.893 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0097032620000000007, 2023-11-05T08:33:18.893 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:18.894 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:18.894 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:18.894 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.894 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:18.894 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.128988+0000", 2023-11-05T08:33:18.895 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:18.895 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.895 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.895 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:18.895 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.128988+0000", 2023-11-05T08:33:18.895 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:18.896 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.896 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.896 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:18.896 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.128989+0000", 2023-11-05T08:33:18.896 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.3600000000000003e-07 2023-11-05T08:33:18.897 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.897 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.897 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:18.897 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.129043+0000", 2023-11-05T08:33:18.897 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.4163999999999998e-05 2023-11-05T08:33:18.898 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.898 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.898 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:18.898 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.129080+0000", 2023-11-05T08:33:18.898 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.7144999999999998e-05 2023-11-05T08:33:18.899 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.899 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.899 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:18.899 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.129104+0000", 2023-11-05T08:33:18.899 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3538999999999999e-05 2023-11-05T08:33:18.900 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.900 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.900 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:18.900 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.129248+0000", 2023-11-05T08:33:18.900 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00014404500000000001 2023-11-05T08:33:18.901 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.901 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.901 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:18.901 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.129273+0000", 2023-11-05T08:33:18.901 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5814999999999999e-05 2023-11-05T08:33:18.902 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.902 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.902 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:18.902 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.138576+0000", 2023-11-05T08:33:18.902 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0093020840000000004 2023-11-05T08:33:18.902 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.903 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.903 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:18.903 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.138691+0000", 2023-11-05T08:33:18.903 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000115834 2023-11-05T08:33:18.903 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:18.904 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:18.904 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:18.904 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.904 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.904 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.as4 41/33 ECSubWrite(tid=36, reqid=osd.7.0:36, at_version=41'14, trim_to=0'0, roll_forward_to=40'10))", 2023-11-05T08:33:18.905 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.131320+0000", 2023-11-05T08:33:18.905 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 24.700142673999999, 2023-11-05T08:33:18.905 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0086819870000000004, 2023-11-05T08:33:18.905 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:18.905 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:18.905 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:18.906 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.906 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:18.906 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131320+0000", 2023-11-05T08:33:18.906 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:18.906 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.907 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.907 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:18.907 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131320+0000", 2023-11-05T08:33:18.907 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:18.907 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.908 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.908 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:18.908 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131322+0000", 2023-11-05T08:33:18.908 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2500000000000001e-06 2023-11-05T08:33:18.909 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.909 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.909 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:18.909 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131375+0000", 2023-11-05T08:33:18.909 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.3041000000000003e-05 2023-11-05T08:33:18.910 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.910 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.910 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:18.910 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131422+0000", 2023-11-05T08:33:18.910 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.6526999999999997e-05 2023-11-05T08:33:18.911 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.911 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.911 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:18.911 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131445+0000", 2023-11-05T08:33:18.911 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3672e-05 2023-11-05T08:33:18.911 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.912 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.912 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:18.912 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131641+0000", 2023-11-05T08:33:18.912 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00019531299999999999 2023-11-05T08:33:18.912 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.913 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.913 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:18.913 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131666+0000", 2023-11-05T08:33:18.913 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5122000000000001e-05 2023-11-05T08:33:18.913 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.913 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.914 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:18.914 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.139945+0000", 2023-11-05T08:33:18.914 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0082790889999999999 2023-11-05T08:33:18.914 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.914 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.915 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:18.915 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.140002+0000", 2023-11-05T08:33:18.915 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.6972999999999998e-05 2023-11-05T08:33:18.915 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:18.915 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:18.915 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:18.916 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.916 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.916 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpReadReply(3.19s0 41/40 ECSubReadReply(tid=67, attrs_read=0))", 2023-11-05T08:33:18.916 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.176501+0000", 2023-11-05T08:33:18.916 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 24.654962157, 2023-11-05T08:33:18.917 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.006463318, 2023-11-05T08:33:18.917 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:18.917 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:18.917 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:18.917 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.918 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:18.918 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.176501+0000", 2023-11-05T08:33:18.918 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:18.918 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.918 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.919 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:18.919 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.176501+0000", 2023-11-05T08:33:18.919 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:18.919 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.919 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.920 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:18.920 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.176501+0000", 2023-11-05T08:33:18.920 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.5199999999999995e-07 2023-11-05T08:33:18.920 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.920 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.921 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:18.921 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.176802+0000", 2023-11-05T08:33:18.921 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00030095700000000003 2023-11-05T08:33:18.921 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.921 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.921 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:18.922 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.176822+0000", 2023-11-05T08:33:18.922 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9179000000000001e-05 2023-11-05T08:33:18.922 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.922 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.922 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:18.923 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.176834+0000", 2023-11-05T08:33:18.923 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2152e-05 2023-11-05T08:33:18.923 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.923 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.923 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:18.924 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.181163+0000", 2023-11-05T08:33:18.924 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0043295790000000001 2023-11-05T08:33:18.924 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.924 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.924 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:18.925 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.182964+0000", 2023-11-05T08:33:18.925 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0018004989999999999 2023-11-05T08:33:18.925 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:18.925 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:18.925 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:18.925 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.926 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.926 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpRead(3.19s0 41/40 ECSubRead(tid=68, to_read={3:9ce6fdf0:::smithi09039252-45:head=0,188416,0}, subchunks={3:9ce6fdf0:::smithi09039252-45:head=[0,1]}, attrs_to_read=))", 2023-11-05T08:33:18.926 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.177591+0000", 2023-11-05T08:33:18.926 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 24.653871329000001, 2023-11-05T08:33:18.927 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.006160787, 2023-11-05T08:33:18.927 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:18.927 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:18.927 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:18.927 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.927 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:18.928 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.177591+0000", 2023-11-05T08:33:18.928 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:18.928 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.928 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.928 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:18.929 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.177591+0000", 2023-11-05T08:33:18.929 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:18.929 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.929 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.929 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:18.930 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.177591+0000", 2023-11-05T08:33:18.930 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:18.930 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.930 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.930 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:18.931 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.177591+0000", 2023-11-05T08:33:18.931 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:18.931 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.931 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.931 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:18.932 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.177636+0000", 2023-11-05T08:33:18.932 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.4920999999999999e-05 2023-11-05T08:33:18.932 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.932 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.932 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:18.932 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.177658+0000", 2023-11-05T08:33:18.933 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1999000000000001e-05 2023-11-05T08:33:18.933 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.933 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.933 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:18.933 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.183143+0000", 2023-11-05T08:33:18.934 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0054844409999999996 2023-11-05T08:33:18.934 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.934 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.934 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:18.934 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.183752+0000", 2023-11-05T08:33:18.934 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00060942600000000004 2023-11-05T08:33:18.935 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:18.935 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:18.935 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:18.935 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.935 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.935 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:293 3.9s0 3:96f693b2:::smithi09039252-13:head [read 0~1064960,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2023-11-05T08:33:18.936 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.184012+0000", 2023-11-05T08:33:18.936 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 24.647450833000001, 2023-11-05T08:33:18.936 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0082043189999999998, 2023-11-05T08:33:18.936 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:18.936 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2023-11-05T08:33:18.936 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:33:18.937 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:33:18.937 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:18.937 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 293 2023-11-05T08:33:18.937 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.937 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:18.938 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.938 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:18.938 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184012+0000", 2023-11-05T08:33:18.938 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:18.938 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.938 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.939 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:18.939 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184012+0000", 2023-11-05T08:33:18.939 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:18.939 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.939 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.940 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:18.940 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184014+0000", 2023-11-05T08:33:18.940 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4930000000000001e-06 2023-11-05T08:33:18.940 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.940 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.940 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:18.941 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184024+0000", 2023-11-05T08:33:18.941 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0114e-05 2023-11-05T08:33:18.941 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.941 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.941 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:18.941 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184047+0000", 2023-11-05T08:33:18.942 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2463000000000002e-05 2023-11-05T08:33:18.942 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.942 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.942 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:18.942 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184063+0000", 2023-11-05T08:33:18.942 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6212e-05 2023-11-05T08:33:18.943 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.943 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.943 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:18.943 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184172+0000", 2023-11-05T08:33:18.943 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000108501 2023-11-05T08:33:18.944 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.944 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.944 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:33:18.944 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184445+0000", 2023-11-05T08:33:18.944 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00027307699999999999 2023-11-05T08:33:18.944 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.945 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.945 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:18.945 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.192216+0000", 2023-11-05T08:33:18.945 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0077714589999999997 2023-11-05T08:33:18.945 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:18.946 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:18.946 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:18.946 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.946 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.946 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:294 3.9s0 3:96f693b2:::smithi09039252-13:head [read 0~1064960,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2023-11-05T08:33:18.947 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.184100+0000", 2023-11-05T08:33:18.947 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 24.647362518000001, 2023-11-05T08:33:18.947 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0099915090000000008, 2023-11-05T08:33:18.947 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:18.947 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2023-11-05T08:33:18.947 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:33:18.948 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:33:18.948 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:18.948 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 294 2023-11-05T08:33:18.948 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.948 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:18.949 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.949 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:18.949 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184100+0000", 2023-11-05T08:33:18.949 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:18.949 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.949 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.950 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:18.950 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184100+0000", 2023-11-05T08:33:18.950 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:18.950 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.950 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.951 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:18.951 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184101+0000", 2023-11-05T08:33:18.951 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2100000000000001e-06 2023-11-05T08:33:18.951 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.951 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.952 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:18.952 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184109+0000", 2023-11-05T08:33:18.952 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.5630000000000003e-06 2023-11-05T08:33:18.952 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.952 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.952 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:18.953 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184126+0000", 2023-11-05T08:33:18.953 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6674e-05 2023-11-05T08:33:18.953 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.953 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.953 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:18.954 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184138+0000", 2023-11-05T08:33:18.954 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2306e-05 2023-11-05T08:33:18.954 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.954 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.954 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:18.955 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.189352+0000", 2023-11-05T08:33:18.955 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0052140629999999997 2023-11-05T08:33:18.955 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.955 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.955 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:33:18.956 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.189596+0000", 2023-11-05T08:33:18.956 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00024365899999999999 2023-11-05T08:33:18.956 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.956 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.956 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:18.956 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.194092+0000", 2023-11-05T08:33:18.957 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0044960340000000003 2023-11-05T08:33:18.957 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:18.957 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:18.957 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:18.957 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.958 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.958 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:295 3.9s0 3:96f693b2:::smithi09039252-13:head [read 0~1064960,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2023-11-05T08:33:18.958 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.184155+0000", 2023-11-05T08:33:18.958 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 24.647307713, 2023-11-05T08:33:18.958 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.011309207999999999, 2023-11-05T08:33:18.959 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:18.959 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2023-11-05T08:33:18.959 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:33:18.959 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:33:18.959 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:18.959 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 295 2023-11-05T08:33:18.960 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.960 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:18.960 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.960 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:18.960 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184155+0000", 2023-11-05T08:33:18.961 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:18.961 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.961 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.961 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:18.961 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184155+0000", 2023-11-05T08:33:18.961 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:18.962 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.962 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.962 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:18.962 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184156+0000", 2023-11-05T08:33:18.962 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0139999999999999e-06 2023-11-05T08:33:18.963 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.963 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.963 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:18.963 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184162+0000", 2023-11-05T08:33:18.963 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.5330000000000002e-06 2023-11-05T08:33:18.963 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.964 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.964 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:18.964 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184176+0000", 2023-11-05T08:33:18.964 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.486e-05 2023-11-05T08:33:18.964 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.964 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.965 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:18.965 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184188+0000", 2023-11-05T08:33:18.965 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1246e-05 2023-11-05T08:33:18.965 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.965 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.965 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:18.966 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.190012+0000", 2023-11-05T08:33:18.966 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0058240920000000003 2023-11-05T08:33:18.966 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.966 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.966 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:33:18.967 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.190143+0000", 2023-11-05T08:33:18.967 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000130984 2023-11-05T08:33:18.967 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.967 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.967 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:18.967 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.195464+0000", 2023-11-05T08:33:18.968 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0053214789999999996 2023-11-05T08:33:18.968 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:18.968 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:18.968 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:18.968 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.968 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.969 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.14s5 41/40 ECSubWrite(tid=17, reqid=client.4691.0:278, at_version=41'14, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:18.969 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.185798+0000", 2023-11-05T08:33:18.969 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 24.645664436000001, 2023-11-05T08:33:18.969 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0063439719999999998, 2023-11-05T08:33:18.969 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:18.969 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:18.970 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:18.970 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.970 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:18.970 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.185798+0000", 2023-11-05T08:33:18.970 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:18.971 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.971 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.971 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:18.971 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.185798+0000", 2023-11-05T08:33:18.971 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:18.971 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.972 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.972 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:18.972 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.185800+0000", 2023-11-05T08:33:18.972 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.689e-06 2023-11-05T08:33:18.972 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.973 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.973 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:18.973 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.186569+0000", 2023-11-05T08:33:18.973 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00076927199999999999 2023-11-05T08:33:18.973 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.973 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.974 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:18.974 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.186598+0000", 2023-11-05T08:33:18.974 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8776999999999999e-05 2023-11-05T08:33:18.974 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.974 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.975 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:18.975 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.186615+0000", 2023-11-05T08:33:18.975 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6759e-05 2023-11-05T08:33:18.975 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.975 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.975 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:18.976 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.186763+0000", 2023-11-05T08:33:18.976 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000148646 2023-11-05T08:33:18.976 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.976 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.976 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:18.977 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.186784+0000", 2023-11-05T08:33:18.977 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0709000000000001e-05 2023-11-05T08:33:18.977 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.977 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.977 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:18.977 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.192059+0000", 2023-11-05T08:33:18.978 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0052751400000000002 2023-11-05T08:33:18.978 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.978 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.978 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:18.978 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.192142+0000", 2023-11-05T08:33:18.979 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.2979999999999995e-05 2023-11-05T08:33:18.979 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:18.979 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:18.979 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:18.979 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.979 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.980 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.as4 42/33 ECSubWrite(tid=38, reqid=osd.7.0:38, at_version=42'16, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:33:18.980 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:55.127593+0000", 2023-11-05T08:33:18.980 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 23.703869494999999, 2023-11-05T08:33:18.980 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.014145875, 2023-11-05T08:33:18.980 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:18.981 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:18.981 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:18.981 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.981 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:18.981 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127593+0000", 2023-11-05T08:33:18.981 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:18.982 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.982 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.982 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:18.982 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127593+0000", 2023-11-05T08:33:18.982 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:18.983 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.983 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.983 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:18.983 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127594+0000", 2023-11-05T08:33:18.983 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.1399999999999996e-07 2023-11-05T08:33:18.984 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.984 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.984 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:18.984 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127658+0000", 2023-11-05T08:33:18.984 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.3590000000000006e-05 2023-11-05T08:33:18.984 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.985 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.985 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:18.985 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127690+0000", 2023-11-05T08:33:18.985 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2447999999999998e-05 2023-11-05T08:33:18.985 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.986 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.986 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:18.986 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127716+0000", 2023-11-05T08:33:18.986 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5931999999999998e-05 2023-11-05T08:33:18.986 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.986 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.987 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:18.987 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127955+0000", 2023-11-05T08:33:18.987 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00023882100000000001 2023-11-05T08:33:18.987 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.987 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.988 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:18.988 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127983+0000", 2023-11-05T08:33:18.988 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7928000000000001e-05 2023-11-05T08:33:18.988 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.988 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.988 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:18.989 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.141638+0000", 2023-11-05T08:33:18.989 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.013655244 2023-11-05T08:33:18.989 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.989 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.989 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:18.989 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.141739+0000", 2023-11-05T08:33:18.990 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000101098 2023-11-05T08:33:18.990 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:18.990 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:18.990 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:18.990 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.991 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.991 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.as4 42/33 ECSubWrite(tid=39, reqid=osd.7.0:39, at_version=42'18, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:33:18.991 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:55.127804+0000", 2023-11-05T08:33:18.991 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 23.703658672, 2023-11-05T08:33:18.991 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.017037818, 2023-11-05T08:33:18.991 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:18.992 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:18.992 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:18.992 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.992 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:18.992 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127804+0000", 2023-11-05T08:33:18.992 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:18.993 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.993 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.993 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:18.993 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127804+0000", 2023-11-05T08:33:18.993 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:18.994 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.994 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.994 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:18.994 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127805+0000", 2023-11-05T08:33:18.994 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.026e-06 2023-11-05T08:33:18.994 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.995 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.995 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:18.995 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127848+0000", 2023-11-05T08:33:18.995 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.3195000000000002e-05 2023-11-05T08:33:18.995 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.996 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.996 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:18.996 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.133513+0000", 2023-11-05T08:33:18.996 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.005664741 2023-11-05T08:33:18.996 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.997 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.997 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:18.997 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.133539+0000", 2023-11-05T08:33:18.997 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6389000000000001e-05 2023-11-05T08:33:18.997 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.997 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.998 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:18.998 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.135275+0000", 2023-11-05T08:33:18.998 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001735349 2023-11-05T08:33:18.998 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.998 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:18.999 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:18.999 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.135312+0000", 2023-11-05T08:33:18.999 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.6965000000000003e-05 2023-11-05T08:33:18.999 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:18.999 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.000 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:19.000 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.144746+0000", 2023-11-05T08:33:19.000 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0094338319999999996 2023-11-05T08:33:19.000 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.000 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.001 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:19.001 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.144842+0000", 2023-11-05T08:33:19.001 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.6321000000000002e-05 2023-11-05T08:33:19.001 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:19.001 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:19.001 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:19.002 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.002 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.002 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:341 3.9s0 3:96f693b2:::smithi09039252-13:head [create,setxattr ]ScjNOsReWmcrsXtQckiNfVsraic]liXUgaZfrPb (12280) in=40b,setxattr RP^fql]PXPbbV]`\\UXhZUcc\\s\\eVVh]On^oUhNgZ (593) in=40b,setxattr ejajVrghj]sk_[VhasYoUSNhlO_PNopbXh\\iYad\\ (1963) in=40b,setxattr ocopbYhpUShgXjpVTtXhgrZ^fhTrWUZnka_sOlQ] (1506) in=40b,setxattr drSfWQp[TZZaai[S^\\ecpepRjetPTZ[Qn[slVhsn (1437) in=40b,setxattr qfWkbgV_hlqXWl`QOe]pOPUPZspbtU__NVPUUltj (877) in=40b,setxattr ]Vrng_\\anafncgktlQh_okb\\i]chR]P]jeoaR\\h] (1858) in=40b,setxattr ]W_abhUq]ocdkOt[ZkpjgPX^drZi]OcUWNRonRa\\ (455) in=40b,setxattr WiZeSWPfsqgYXk`d\\gOeSecbpakdsPTXZfZhn_k^ (875) in=40b,setxattr kqn^akbbamUTRXRhQS^i`lQYomhV_P`j\\ahTj]dk (1894) in=40b,setxattr TUcrapOXk_aZjTSdOT`rYcU_YYstnVgNTojk`lXZ (805) in=40b,setxattr a`RfekoeadQgiYossXre`rV^qNeUNiNf\\lSXhohq (15648) in=40b,setxattr VigYdf[rkVcbX`_aXcZrljeilZZV\\O[gpOhof]\\P (1473) in=40b,setxattr Qrfl^hc[_sp[p`bcN[Ne\\tg^]rgrOlRqfQ_VWNt\\ (1261) in=40b,setxattr iQVqR\\bssbQRQf^a`UrVROfgpPZlnstjPUaUnWWt (665) in=40b,setxattr aO^mjdTNk[qYOgYsgTigbbP`]mhe`WmaNUO[^ele (979) in=40b,setxattr cfQeZUernkePqgPcSrokkcnPqZ^Osc^^ng[ilga_ (16840) in=40b,setxattr fdbVa_]fdVq_`kcpaWlQSgY^VgakSqVhj`qqdj[m (1858) in=40b,setxattr tXieTi_TgP_dVipjmQcU_QVRqNdboYhRWgNrpVhm (1858) in=40b,setxattr ]YTcidkg\\][ZtajYPqmOji_UPVPZ`j^]tnNRefcd (2876) in=40b] snapc 10=[7] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:19.002 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:05.255132+0000", 2023-11-05T08:33:19.002 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 13.576330249, 2023-11-05T08:33:19.003 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.01486664, 2023-11-05T08:33:19.003 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:19.003 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:19.003 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:33:19.003 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:33:19.004 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:19.004 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 341 2023-11-05T08:33:19.004 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.004 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:19.004 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.005 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:19.005 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255132+0000", 2023-11-05T08:33:19.005 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:19.005 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.005 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.006 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:19.006 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255132+0000", 2023-11-05T08:33:19.006 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:19.006 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.006 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.006 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:19.007 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255136+0000", 2023-11-05T08:33:19.007 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.5159999999999999e-06 2023-11-05T08:33:19.007 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.007 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.007 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:19.008 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255423+0000", 2023-11-05T08:33:19.008 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00028735799999999998 2023-11-05T08:33:19.008 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.008 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.008 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:19.009 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255462+0000", 2023-11-05T08:33:19.009 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.8512e-05 2023-11-05T08:33:19.009 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.009 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.009 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:19.009 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255499+0000", 2023-11-05T08:33:19.010 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.7531999999999999e-05 2023-11-05T08:33:19.010 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.010 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.010 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:19.010 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255651+0000", 2023-11-05T08:33:19.011 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00015121700000000001 2023-11-05T08:33:19.011 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.011 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.011 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:33:19.011 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.256051+0000", 2023-11-05T08:33:19.011 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00040081699999999998 2023-11-05T08:33:19.012 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.012 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.012 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:19.012 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.257968+0000", 2023-11-05T08:33:19.012 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001917052 2023-11-05T08:33:19.013 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.013 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.013 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:19.013 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.265314+0000", 2023-11-05T08:33:19.013 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0073454469999999997 2023-11-05T08:33:19.013 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.014 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.014 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:33:19.014 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.269385+0000", 2023-11-05T08:33:19.014 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0040709839999999997 2023-11-05T08:33:19.014 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.015 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.015 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:19.015 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.269999+0000", 2023-11-05T08:33:19.015 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00061420500000000005 2023-11-05T08:33:19.015 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:19.016 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:19.016 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:19.016 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.016 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.016 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:344 3.9s0 3:936041dc:::smithi09039252-3:head [copy-from ver 4] snapc 10=[7] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:19.016 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:05.255538+0000", 2023-11-05T08:33:19.017 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 13.575924239000001, 2023-11-05T08:33:19.017 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.032906886000000003, 2023-11-05T08:33:19.017 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:19.017 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:19.017 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:33:19.018 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:33:19.018 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:19.018 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 344 2023-11-05T08:33:19.018 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.018 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:19.019 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.019 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:19.019 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255538+0000", 2023-11-05T08:33:19.019 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:19.019 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.020 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.020 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:19.020 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255538+0000", 2023-11-05T08:33:19.020 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:19.020 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.020 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.021 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:19.021 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255540+0000", 2023-11-05T08:33:19.021 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5060000000000001e-06 2023-11-05T08:33:19.021 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.021 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.022 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:19.022 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255550+0000", 2023-11-05T08:33:19.022 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0063e-05 2023-11-05T08:33:19.022 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.022 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.022 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:19.023 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255589+0000", 2023-11-05T08:33:19.023 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.9245000000000001e-05 2023-11-05T08:33:19.023 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.023 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.023 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:19.024 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255606+0000", 2023-11-05T08:33:19.024 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6307000000000002e-05 2023-11-05T08:33:19.024 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.024 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.024 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:19.024 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.259641+0000", 2023-11-05T08:33:19.025 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.004035421 2023-11-05T08:33:19.025 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.025 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.025 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:33:19.025 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.260112+0000", 2023-11-05T08:33:19.026 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00047097799999999999 2023-11-05T08:33:19.026 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.026 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.026 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:19.026 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.280466+0000", 2023-11-05T08:33:19.027 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.020353850999999999 2023-11-05T08:33:19.027 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.027 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.027 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:19.027 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.286378+0000", 2023-11-05T08:33:19.028 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0059124490000000002 2023-11-05T08:33:19.028 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.028 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.028 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:33:19.028 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.288092+0000", 2023-11-05T08:33:19.029 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0017133300000000001 2023-11-05T08:33:19.029 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.029 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.029 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:19.029 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.288445+0000", 2023-11-05T08:33:19.030 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000353736 2023-11-05T08:33:19.030 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:19.030 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:19.030 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:19.030 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.031 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.031 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:345 3.9s0 3:936041dc:::smithi09039252-3:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:19.031 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:05.255630+0000", 2023-11-05T08:33:19.031 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 13.575832779000001, 2023-11-05T08:33:19.031 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.029547922000000001, 2023-11-05T08:33:19.031 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:19.032 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2023-11-05T08:33:19.032 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:33:19.032 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:33:19.032 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:19.032 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 345 2023-11-05T08:33:19.033 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.033 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:19.033 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.033 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:19.033 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255630+0000", 2023-11-05T08:33:19.033 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:19.034 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.034 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.034 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:19.034 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255630+0000", 2023-11-05T08:33:19.034 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:19.035 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.035 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.035 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:19.035 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255631+0000", 2023-11-05T08:33:19.035 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.519e-06 2023-11-05T08:33:19.035 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.036 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.036 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:19.036 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255640+0000", 2023-11-05T08:33:19.036 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.9360000000000005e-06 2023-11-05T08:33:19.036 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.037 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.037 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:19.037 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255659+0000", 2023-11-05T08:33:19.037 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9089999999999998e-05 2023-11-05T08:33:19.037 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.038 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.038 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:19.038 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255675+0000", 2023-11-05T08:33:19.038 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5133000000000001e-05 2023-11-05T08:33:19.038 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.038 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.039 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:19.039 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.260602+0000", 2023-11-05T08:33:19.039 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0049270449999999997 2023-11-05T08:33:19.039 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.039 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.040 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for blocked object", 2023-11-05T08:33:19.040 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.260717+0000", 2023-11-05T08:33:19.040 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000115182 2023-11-05T08:33:19.040 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.040 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.041 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:19.041 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.284014+0000", 2023-11-05T08:33:19.041 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.023297577999999999 2023-11-05T08:33:19.041 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.041 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.041 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:33:19.042 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.284370+0000", 2023-11-05T08:33:19.042 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000355306 2023-11-05T08:33:19.042 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.042 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.042 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:19.043 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.285178+0000", 2023-11-05T08:33:19.043 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00080813300000000005 2023-11-05T08:33:19.043 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:19.043 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:19.043 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:19.043 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.044 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.044 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.as4 52/45 ECSubWrite(tid=45, reqid=client.4691.0:349, at_version=52'21, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:19.044 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:05.264746+0000", 2023-11-05T08:33:19.044 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 13.566716718, 2023-11-05T08:33:19.044 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0072026099999999999, 2023-11-05T08:33:19.045 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:19.045 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:19.045 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:19.045 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.045 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:19.045 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.264746+0000", 2023-11-05T08:33:19.046 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:19.046 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.046 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.046 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:19.046 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.264746+0000", 2023-11-05T08:33:19.047 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:19.047 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.047 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.047 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:19.047 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.264748+0000", 2023-11-05T08:33:19.048 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.677e-06 2023-11-05T08:33:19.048 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.048 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.048 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:19.048 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.264893+0000", 2023-11-05T08:33:19.049 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000145067 2023-11-05T08:33:19.049 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.049 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.049 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:19.049 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.264921+0000", 2023-11-05T08:33:19.049 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8328999999999999e-05 2023-11-05T08:33:19.050 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.050 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.050 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:19.050 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.264942+0000", 2023-11-05T08:33:19.050 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1046000000000001e-05 2023-11-05T08:33:19.050 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.051 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.051 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:19.051 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.265123+0000", 2023-11-05T08:33:19.051 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00018091299999999999 2023-11-05T08:33:19.051 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.052 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.052 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:19.052 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.265150+0000", 2023-11-05T08:33:19.052 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6645e-05 2023-11-05T08:33:19.052 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.052 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.053 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:19.053 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.271806+0000", 2023-11-05T08:33:19.053 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0066566919999999996 2023-11-05T08:33:19.053 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.053 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.054 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:19.054 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.271949+0000", 2023-11-05T08:33:19.054 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00014224100000000001 2023-11-05T08:33:19.054 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:19.054 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:19.055 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:19.055 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.055 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.055 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.as4 61/45 ECSubWrite(tid=57, reqid=client.4691.0:405, at_version=61'23, trim_to=0'0, roll_forward_to=52'21))", 2023-11-05T08:33:19.055 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:14.366205+0000", 2023-11-05T08:33:19.055 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 4.465257298, 2023-11-05T08:33:19.056 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0087065779999999995, 2023-11-05T08:33:19.056 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:19.056 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:19.056 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:19.056 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.056 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:19.057 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.366205+0000", 2023-11-05T08:33:19.057 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:19.057 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.057 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.057 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:19.058 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.366205+0000", 2023-11-05T08:33:19.058 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:19.058 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.058 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.058 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:19.059 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.366206+0000", 2023-11-05T08:33:19.059 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.3000000000000001e-07 2023-11-05T08:33:19.059 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.059 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.059 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:19.059 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.366951+0000", 2023-11-05T08:33:19.060 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00074537699999999998 2023-11-05T08:33:19.060 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.060 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.060 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:19.060 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.366978+0000", 2023-11-05T08:33:19.061 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6721999999999999e-05 2023-11-05T08:33:19.061 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.061 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.061 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:19.061 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.366994+0000", 2023-11-05T08:33:19.061 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6189999999999999e-05 2023-11-05T08:33:19.062 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.062 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.062 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:19.062 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.367186+0000", 2023-11-05T08:33:19.062 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000191387 2023-11-05T08:33:19.063 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.063 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.063 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:19.063 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.367213+0000", 2023-11-05T08:33:19.063 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7169e-05 2023-11-05T08:33:19.064 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.064 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.064 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:19.064 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.374835+0000", 2023-11-05T08:33:19.064 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0076217589999999997 2023-11-05T08:33:19.064 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:19.065 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:19.065 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:19.065 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.374912+0000", 2023-11-05T08:33:19.065 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.7443999999999999e-05 2023-11-05T08:33:19.065 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:19.066 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:19.066 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:19.066 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:19.066 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:19.066 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:33:19.068 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-11-05T08:33:19.068 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:33:19.063+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:33:19.169 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:33:19.164+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:33:19.195 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:33:19.196 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-11-05T08:33:19.196 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-11-05T08:33:19.196 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:33:19.210 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-11-05T08:33:19.269 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:33:19.263+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:33:19.331 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:33:19.331 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-11-05T08:33:19.332 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-11-05T08:33:19.332 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-11-05T08:33:19.332 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:33:19.345 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-11-05T08:33:19.372 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-11-05T08:33:19.377 INFO:tasks.rados.rados.0.smithi090.stdout:205: done (5 left) 2023-11-05T08:33:19.377 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 37 current snap is 16 2023-11-05T08:33:19.377 INFO:tasks.rados.rados.0.smithi090.stdout:206: seq_num 93 ranges {606208=802816,1409024=638976,2048000=131072} 2023-11-05T08:33:19.378 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:33:19.371+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:33:19.387 INFO:tasks.rados.rados.0.smithi090.stdout:206: writing smithi09039252-37 from 606208 to 1409024 tid 1 2023-11-05T08:33:19.392 INFO:tasks.rados.rados.0.smithi090.stdout:206: writing smithi09039252-37 from 1409024 to 2048000 tid 2 2023-11-05T08:33:19.393 INFO:tasks.rados.rados.0.smithi090.stdout:206: writing smithi09039252-37 from 2048000 to 2179072 tid 3 2023-11-05T08:33:19.394 INFO:tasks.rados.rados.0.smithi090.stdout:207: read oid 26 snap 4 2023-11-05T08:33:19.394 INFO:tasks.rados.rados.0.smithi090.stdout:207: expect (ObjNum 25 snap 0 seq_num 25) 2023-11-05T08:33:19.394 INFO:tasks.rados.rados.0.smithi090.stdout:208: rmattr oid 27 current snap is 16 2023-11-05T08:33:19.394 INFO:tasks.rados.rados.0.smithi090.stdout:208: done (7 left) 2023-11-05T08:33:19.394 INFO:tasks.rados.rados.0.smithi090.stdout:209: snap_create 2023-11-05T08:33:19.395 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 18 v 21 (ObjNum 17 snap 0 seq_num 17) dirty exists 2023-11-05T08:33:19.395 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 13 v 28 (ObjNum 50 snap 0 seq_num 50) dirty exists 2023-11-05T08:33:19.395 INFO:tasks.rados.rados.0.smithi090.stdout:203: expect (ObjNum 38 snap 0 seq_num 38) 2023-11-05T08:33:19.425 INFO:tasks.rados.rados.0.smithi090.stdout:204: finishing copy_from racing read to smithi09039252-46 2023-11-05T08:33:19.425 INFO:tasks.rados.rados.0.smithi090.stdout:204: finishing copy_from to smithi09039252-46 2023-11-05T08:33:19.426 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 46 v 2 (ObjNum 17 snap 0 seq_num 17) dirty exists 2023-11-05T08:33:19.426 INFO:tasks.rados.rados.0.smithi090.stdout:207: expect (ObjNum 25 snap 0 seq_num 25) 2023-11-05T08:33:19.437 INFO:tasks.rados.rados.0.smithi090.stdout:206: finishing write tid 1 to smithi09039252-37 2023-11-05T08:33:19.438 INFO:tasks.rados.rados.0.smithi090.stdout:206: finishing write tid 2 to smithi09039252-37 2023-11-05T08:33:19.438 INFO:tasks.rados.rados.0.smithi090.stdout:206: finishing write tid 3 to smithi09039252-37 2023-11-05T08:33:19.438 INFO:tasks.rados.rados.0.smithi090.stdout:206: finishing write tid 4 to smithi09039252-37 2023-11-05T08:33:19.438 INFO:tasks.rados.rados.0.smithi090.stdout:206: finishing write tid 5 to smithi09039252-37 2023-11-05T08:33:19.438 INFO:tasks.rados.rados.0.smithi090.stdout:206: oid 37 updating version 0 to 24 2023-11-05T08:33:19.439 INFO:tasks.rados.rados.0.smithi090.stdout:206: oid 37 updating version 24 to 25 2023-11-05T08:33:19.439 INFO:tasks.rados.rados.0.smithi090.stdout:206: oid 37 updating version 25 to 26 2023-11-05T08:33:19.439 INFO:tasks.rados.rados.0.smithi090.stdout:206: oid 37 version 26 is already newer than 23 2023-11-05T08:33:19.439 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 37 v 26 (ObjNum 93 snap 16 seq_num 93) dirty exists 2023-11-05T08:33:19.439 INFO:tasks.rados.rados.0.smithi090.stdout:206: left oid 37 (ObjNum 93 snap 16 seq_num 93) 2023-11-05T08:33:19.470 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-05T08:33:19.512 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:33:19.512 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-11-05T08:33:19.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-11-05T08:33:19.512 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-11-05T08:33:19.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.513 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:233 3.ds0 3:bf7e600d:::smithi09039252-23:head [append 0~0] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2023-11-05T08:33:19.513 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.059123+0000", 2023-11-05T08:33:19.513 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.449295329999998, 2023-11-05T08:33:19.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019416857999999999, 2023-11-05T08:33:19.513 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:19.514 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:19.514 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:19.514 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:19.514 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:19.514 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 233 2023-11-05T08:33:19.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.515 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:19.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:19.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059123+0000", 2023-11-05T08:33:19.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:19.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059123+0000", 2023-11-05T08:33:19.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:19.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059126+0000", 2023-11-05T08:33:19.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8370000000000001e-06 2023-11-05T08:33:19.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:19.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059140+0000", 2023-11-05T08:33:19.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3669000000000001e-05 2023-11-05T08:33:19.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:19.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059176+0000", 2023-11-05T08:33:19.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6548000000000003e-05 2023-11-05T08:33:19.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:19.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059198+0000", 2023-11-05T08:33:19.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1489e-05 2023-11-05T08:33:19.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:19.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059401+0000", 2023-11-05T08:33:19.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020313300000000001 2023-11-05T08:33:19.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-11-05T08:33:19.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059422+0000", 2023-11-05T08:33:19.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1212e-05 2023-11-05T08:33:19.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:19.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.063981+0000", 2023-11-05T08:33:19.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004558604 2023-11-05T08:33:19.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:19.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.064128+0000", 2023-11-05T08:33:19.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014734899999999999 2023-11-05T08:33:19.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:19.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.066042+0000", 2023-11-05T08:33:19.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019138079999999999 2023-11-05T08:33:19.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:19.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.077186+0000", 2023-11-05T08:33:19.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011144536 2023-11-05T08:33:19.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:19.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.078357+0000", 2023-11-05T08:33:19.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011709089999999999 2023-11-05T08:33:19.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:19.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.078540+0000", 2023-11-05T08:33:19.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000182764 2023-11-05T08:33:19.528 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.528 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:19.528 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.529 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:234 3.ds0 3:bf7e600d:::smithi09039252-23:head [rollback 3] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2023-11-05T08:33:19.529 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.059247+0000", 2023-11-05T08:33:19.529 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.449171733, 2023-11-05T08:33:19.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019739059, 2023-11-05T08:33:19.530 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:19.530 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:19.530 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:19.530 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:19.530 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:19.530 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 234 2023-11-05T08:33:19.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.531 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:19.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:19.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059247+0000", 2023-11-05T08:33:19.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:19.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059247+0000", 2023-11-05T08:33:19.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:19.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059248+0000", 2023-11-05T08:33:19.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.077e-06 2023-11-05T08:33:19.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:19.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059256+0000", 2023-11-05T08:33:19.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1359999999999997e-06 2023-11-05T08:33:19.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:19.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059283+0000", 2023-11-05T08:33:19.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6855e-05 2023-11-05T08:33:19.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:19.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059298+0000", 2023-11-05T08:33:19.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5356e-05 2023-11-05T08:33:19.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:19.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059582+0000", 2023-11-05T08:33:19.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028341299999999999 2023-11-05T08:33:19.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:33:19.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059592+0000", 2023-11-05T08:33:19.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0103e-05 2023-11-05T08:33:19.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:19.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.067711+0000", 2023-11-05T08:33:19.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0081197660000000005 2023-11-05T08:33:19.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:19.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.067965+0000", 2023-11-05T08:33:19.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025397000000000001 2023-11-05T08:33:19.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:19.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.069296+0000", 2023-11-05T08:33:19.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013308910000000001 2023-11-05T08:33:19.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:19.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.077189+0000", 2023-11-05T08:33:19.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0078931079999999994 2023-11-05T08:33:19.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:19.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.078829+0000", 2023-11-05T08:33:19.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016394459999999999 2023-11-05T08:33:19.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:19.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.078986+0000", 2023-11-05T08:33:19.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015693799999999999 2023-11-05T08:33:19.544 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.544 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:19.544 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.545 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:235 3.ds0 3:bf7e600d:::smithi09039252-23:head [append 0~0] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2023-11-05T08:33:19.545 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.059370+0000", 2023-11-05T08:33:19.545 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.449048519000002, 2023-11-05T08:33:19.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025190444999999999, 2023-11-05T08:33:19.546 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:19.546 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:19.546 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:19.546 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:19.546 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:19.546 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 235 2023-11-05T08:33:19.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.547 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:19.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:19.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059370+0000", 2023-11-05T08:33:19.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:19.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059370+0000", 2023-11-05T08:33:19.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:19.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059371+0000", 2023-11-05T08:33:19.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2899999999999999e-06 2023-11-05T08:33:19.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:19.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059378+0000", 2023-11-05T08:33:19.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0729999999999999e-06 2023-11-05T08:33:19.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:19.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059408+0000", 2023-11-05T08:33:19.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9904999999999999e-05 2023-11-05T08:33:19.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:19.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059418+0000", 2023-11-05T08:33:19.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7659999999999993e-06 2023-11-05T08:33:19.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:19.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059648+0000", 2023-11-05T08:33:19.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022963200000000001 2023-11-05T08:33:19.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:33:19.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059657+0000", 2023-11-05T08:33:19.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9700000000000005e-06 2023-11-05T08:33:19.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:19.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.070350+0000", 2023-11-05T08:33:19.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010693342 2023-11-05T08:33:19.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:19.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.070601+0000", 2023-11-05T08:33:19.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025082000000000001 2023-11-05T08:33:19.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:19.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.071898+0000", 2023-11-05T08:33:19.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012976190000000001 2023-11-05T08:33:19.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:19.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.077192+0000", 2023-11-05T08:33:19.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052931840000000003 2023-11-05T08:33:19.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:19.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.084398+0000", 2023-11-05T08:33:19.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0072062619999999997 2023-11-05T08:33:19.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:19.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.084560+0000", 2023-11-05T08:33:19.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016258200000000001 2023-11-05T08:33:19.560 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.560 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:19.560 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.561 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:283 3.as1 3:5396e455:::smithi09039252-43:head [append 0~753664] snapc 8=[8,7,6,5,3] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:33:19.561 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:53.164549+0000", 2023-11-05T08:33:19.561 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 26.343869008999999, 2023-11-05T08:33:19.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012425772, 2023-11-05T08:33:19.561 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:19.562 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:19.562 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:19.562 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:19.562 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:19.562 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 283 2023-11-05T08:33:19.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.563 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:19.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:19.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.164549+0000", 2023-11-05T08:33:19.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:19.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.164549+0000", 2023-11-05T08:33:19.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:19.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.164552+0000", 2023-11-05T08:33:19.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9069999999999999e-06 2023-11-05T08:33:19.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:19.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.165387+0000", 2023-11-05T08:33:19.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00083439899999999999 2023-11-05T08:33:19.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:19.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.165421+0000", 2023-11-05T08:33:19.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4344999999999997e-05 2023-11-05T08:33:19.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:19.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.165439+0000", 2023-11-05T08:33:19.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7793000000000001e-05 2023-11-05T08:33:19.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:19.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.165639+0000", 2023-11-05T08:33:19.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019997400000000001 2023-11-05T08:33:19.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:19.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.165813+0000", 2023-11-05T08:33:19.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017405599999999999 2023-11-05T08:33:19.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:19.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.167863+0000", 2023-11-05T08:33:19.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002050315 2023-11-05T08:33:19.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:19.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.176139+0000", 2023-11-05T08:33:19.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0082754379999999995 2023-11-05T08:33:19.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:19.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.176765+0000", 2023-11-05T08:33:19.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00062584200000000004 2023-11-05T08:33:19.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:19.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.176975+0000", 2023-11-05T08:33:19.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021070299999999999 2023-11-05T08:33:19.574 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.574 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:19.574 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.575 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:285 3.as1 3:5396e455:::smithi09039252-43:head [setxattr _header (56) in=7b] snapc 8=[8,7,6,5,3] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:33:19.575 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:53.169940+0000", 2023-11-05T08:33:19.575 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 26.338478832, 2023-11-05T08:33:19.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011368382, 2023-11-05T08:33:19.575 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:19.575 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:19.576 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:19.576 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:19.576 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:19.576 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 285 2023-11-05T08:33:19.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.577 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:19.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:19.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.169940+0000", 2023-11-05T08:33:19.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:19.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.169940+0000", 2023-11-05T08:33:19.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:19.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.169941+0000", 2023-11-05T08:33:19.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7889999999999999e-06 2023-11-05T08:33:19.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:19.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.169951+0000", 2023-11-05T08:33:19.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6530000000000006e-06 2023-11-05T08:33:19.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:19.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.169973+0000", 2023-11-05T08:33:19.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2028999999999999e-05 2023-11-05T08:33:19.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:19.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.169990+0000", 2023-11-05T08:33:19.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7082999999999999e-05 2023-11-05T08:33:19.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:19.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.177259+0000", 2023-11-05T08:33:19.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.007269045 2023-11-05T08:33:19.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:19.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.177423+0000", 2023-11-05T08:33:19.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016353599999999999 2023-11-05T08:33:19.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:19.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.178314+0000", 2023-11-05T08:33:19.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089105399999999996 2023-11-05T08:33:19.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:19.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.180575+0000", 2023-11-05T08:33:19.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022613249999999998 2023-11-05T08:33:19.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:19.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.181127+0000", 2023-11-05T08:33:19.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00055179600000000001 2023-11-05T08:33:19.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:19.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.181308+0000", 2023-11-05T08:33:19.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000181072 2023-11-05T08:33:19.588 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.588 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:19.588 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.588 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:286 3.as1 3:5396e455:::smithi09039252-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:33:19.589 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:53.170019+0000", 2023-11-05T08:33:19.589 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 26.338399093, 2023-11-05T08:33:19.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014799489000000001, 2023-11-05T08:33:19.589 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:19.589 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:19.590 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:19.590 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:19.590 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:19.590 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 286 2023-11-05T08:33:19.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.590 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:19.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:19.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.170019+0000", 2023-11-05T08:33:19.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:19.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.170019+0000", 2023-11-05T08:33:19.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:19.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.170021+0000", 2023-11-05T08:33:19.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.877e-06 2023-11-05T08:33:19.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:19.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.170029+0000", 2023-11-05T08:33:19.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6960000000000005e-06 2023-11-05T08:33:19.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:19.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.170056+0000", 2023-11-05T08:33:19.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6928999999999999e-05 2023-11-05T08:33:19.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:19.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.170072+0000", 2023-11-05T08:33:19.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6515999999999999e-05 2023-11-05T08:33:19.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:19.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.180292+0000", 2023-11-05T08:33:19.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010219293000000001 2023-11-05T08:33:19.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-11-05T08:33:19.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.180558+0000", 2023-11-05T08:33:19.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026664499999999998 2023-11-05T08:33:19.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:19.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.182159+0000", 2023-11-05T08:33:19.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016003650000000001 2023-11-05T08:33:19.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:19.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.182378+0000", 2023-11-05T08:33:19.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021884699999999999 2023-11-05T08:33:19.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:19.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.184819+0000", 2023-11-05T08:33:19.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024413210000000002 2023-11-05T08:33:19.601 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.601 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:19.601 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.602 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:349 3.as1 3:56f12b5c:::smithi09039252-15:head [copy-from ver 2] snapc 10=[7] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:19.602 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:05.254950+0000", 2023-11-05T08:33:19.602 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 14.253468521, 2023-11-05T08:33:19.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018320570000000001, 2023-11-05T08:33:19.602 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:19.603 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:19.603 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:19.603 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:19.603 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:19.603 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 349 2023-11-05T08:33:19.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.604 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:19.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:19.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.254950+0000", 2023-11-05T08:33:19.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:19.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.254950+0000", 2023-11-05T08:33:19.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:19.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.254953+0000", 2023-11-05T08:33:19.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6699999999999998e-06 2023-11-05T08:33:19.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:19.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.254966+0000", 2023-11-05T08:33:19.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.344e-05 2023-11-05T08:33:19.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:19.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255101+0000", 2023-11-05T08:33:19.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013478200000000001 2023-11-05T08:33:19.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:19.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255123+0000", 2023-11-05T08:33:19.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2541999999999999e-05 2023-11-05T08:33:19.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:19.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255409+0000", 2023-11-05T08:33:19.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028556799999999999 2023-11-05T08:33:19.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:19.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255826+0000", 2023-11-05T08:33:19.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041679299999999999 2023-11-05T08:33:19.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:19.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.262849+0000", 2023-11-05T08:33:19.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0070237030000000001 2023-11-05T08:33:19.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:19.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.272188+0000", 2023-11-05T08:33:19.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0093385970000000006 2023-11-05T08:33:19.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:19.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.273116+0000", 2023-11-05T08:33:19.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00092777199999999999 2023-11-05T08:33:19.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:19.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.273271+0000", 2023-11-05T08:33:19.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015470300000000001 2023-11-05T08:33:19.614 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.615 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:19.615 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.615 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:356 3.as1 3:5396e455:::smithi09039252-43:head [read 0~1851392,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:19.616 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:05.279486+0000", 2023-11-05T08:33:19.616 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 14.228932016, 2023-11-05T08:33:19.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011841981, 2023-11-05T08:33:19.616 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:19.616 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:19.616 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:19.617 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:19.617 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:19.617 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 356 2023-11-05T08:33:19.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.617 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:19.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:19.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.279486+0000", 2023-11-05T08:33:19.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:19.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.279486+0000", 2023-11-05T08:33:19.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:19.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.279487+0000", 2023-11-05T08:33:19.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5300000000000003e-07 2023-11-05T08:33:19.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:19.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.279494+0000", 2023-11-05T08:33:19.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7700000000000004e-06 2023-11-05T08:33:19.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:19.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.279506+0000", 2023-11-05T08:33:19.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1489e-05 2023-11-05T08:33:19.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:19.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.279512+0000", 2023-11-05T08:33:19.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3740000000000003e-06 2023-11-05T08:33:19.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:19.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.280889+0000", 2023-11-05T08:33:19.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013769889999999999 2023-11-05T08:33:19.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:19.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.280994+0000", 2023-11-05T08:33:19.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000105374 2023-11-05T08:33:19.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:19.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.291328+0000", 2023-11-05T08:33:19.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010334131999999999 2023-11-05T08:33:19.626 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.626 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:19.626 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.627 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:357 3.as1 3:5396e455:::smithi09039252-43:head [read 0~1851392,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:19.627 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:05.279538+0000", 2023-11-05T08:33:19.627 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 14.228880547999999, 2023-11-05T08:33:19.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015991607000000001, 2023-11-05T08:33:19.627 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:19.628 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:19.628 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:19.628 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:19.628 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:19.628 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 357 2023-11-05T08:33:19.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.629 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:19.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:19.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.279538+0000", 2023-11-05T08:33:19.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:19.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.279538+0000", 2023-11-05T08:33:19.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:19.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.279538+0000", 2023-11-05T08:33:19.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0100000000000005e-07 2023-11-05T08:33:19.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:19.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.279542+0000", 2023-11-05T08:33:19.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8519999999999997e-06 2023-11-05T08:33:19.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:19.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.279551+0000", 2023-11-05T08:33:19.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2409999999999995e-06 2023-11-05T08:33:19.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:19.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.279557+0000", 2023-11-05T08:33:19.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4219999999999997e-06 2023-11-05T08:33:19.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:19.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.281448+0000", 2023-11-05T08:33:19.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001891104 2023-11-05T08:33:19.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:19.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.281739+0000", 2023-11-05T08:33:19.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029091399999999997 2023-11-05T08:33:19.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:19.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.295530+0000", 2023-11-05T08:33:19.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013790573 2023-11-05T08:33:19.637 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.637 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:19.637 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.638 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:362 3.ds0 3:bf537852:::smithi09039252-10:head [read 0~163840,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2023-11-05T08:33:19.638 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:08.268553+0000", 2023-11-05T08:33:19.638 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.239865147, 2023-11-05T08:33:19.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011319786, 2023-11-05T08:33:19.639 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:19.639 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:19.639 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:19.639 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:19.639 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:19.640 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 362 2023-11-05T08:33:19.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.640 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:19.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:19.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.268553+0000", 2023-11-05T08:33:19.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:19.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.268553+0000", 2023-11-05T08:33:19.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:19.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.268557+0000", 2023-11-05T08:33:19.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2760000000000001e-06 2023-11-05T08:33:19.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:19.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.268574+0000", 2023-11-05T08:33:19.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7193e-05 2023-11-05T08:33:19.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:19.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.268620+0000", 2023-11-05T08:33:19.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5772000000000002e-05 2023-11-05T08:33:19.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:19.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.268643+0000", 2023-11-05T08:33:19.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3099e-05 2023-11-05T08:33:19.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:19.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.268836+0000", 2023-11-05T08:33:19.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019351399999999999 2023-11-05T08:33:19.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-11-05T08:33:19.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.268863+0000", 2023-11-05T08:33:19.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6359999999999998e-05 2023-11-05T08:33:19.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:19.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.276111+0000", 2023-11-05T08:33:19.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0072480909999999999 2023-11-05T08:33:19.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:19.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.276370+0000", 2023-11-05T08:33:19.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025974499999999997 2023-11-05T08:33:19.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:19.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.279873+0000", 2023-11-05T08:33:19.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0035027359999999998 2023-11-05T08:33:19.650 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.651 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:19.651 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.651 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:363 3.ds0 3:bf537852:::smithi09039252-10:head [read 0~163840,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2023-11-05T08:33:19.652 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:08.268702+0000", 2023-11-05T08:33:19.652 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.239716099000001, 2023-11-05T08:33:19.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012581584, 2023-11-05T08:33:19.652 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:19.652 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:19.652 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:19.653 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:19.653 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:19.653 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 363 2023-11-05T08:33:19.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.653 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:19.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:19.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.268702+0000", 2023-11-05T08:33:19.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:19.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.268702+0000", 2023-11-05T08:33:19.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:19.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.268705+0000", 2023-11-05T08:33:19.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2139999999999999e-06 2023-11-05T08:33:19.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:19.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.268715+0000", 2023-11-05T08:33:19.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0066e-05 2023-11-05T08:33:19.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:19.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.268737+0000", 2023-11-05T08:33:19.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2073e-05 2023-11-05T08:33:19.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:19.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.268753+0000", 2023-11-05T08:33:19.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6503000000000001e-05 2023-11-05T08:33:19.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:19.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.269068+0000", 2023-11-05T08:33:19.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031519600000000001 2023-11-05T08:33:19.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:33:19.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.269097+0000", 2023-11-05T08:33:19.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8670000000000002e-05 2023-11-05T08:33:19.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:19.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.276952+0000", 2023-11-05T08:33:19.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0078548060000000006 2023-11-05T08:33:19.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:19.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.277120+0000", 2023-11-05T08:33:19.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016855400000000001 2023-11-05T08:33:19.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:19.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.281284+0000", 2023-11-05T08:33:19.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0041635020000000003 2023-11-05T08:33:19.664 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.664 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:19.664 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.665 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:364 3.ds0 3:bf537852:::smithi09039252-10:head [read 0~163840,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2023-11-05T08:33:19.665 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:08.268782+0000", 2023-11-05T08:33:19.665 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.239636844, 2023-11-05T08:33:19.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013418439000000001, 2023-11-05T08:33:19.665 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:19.666 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:19.666 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:19.666 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:19.666 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:19.666 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 364 2023-11-05T08:33:19.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.667 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:19.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:19.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.268782+0000", 2023-11-05T08:33:19.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:19.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.268782+0000", 2023-11-05T08:33:19.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:19.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.268783+0000", 2023-11-05T08:33:19.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2729999999999999e-06 2023-11-05T08:33:19.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:19.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.268791+0000", 2023-11-05T08:33:19.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9000000000000006e-06 2023-11-05T08:33:19.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:19.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.268810+0000", 2023-11-05T08:33:19.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9556e-05 2023-11-05T08:33:19.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:19.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.268825+0000", 2023-11-05T08:33:19.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4392999999999999e-05 2023-11-05T08:33:19.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:19.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.269166+0000", 2023-11-05T08:33:19.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034102200000000002 2023-11-05T08:33:19.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:33:19.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.269178+0000", 2023-11-05T08:33:19.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2289999999999999e-05 2023-11-05T08:33:19.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:19.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.279976+0000", 2023-11-05T08:33:19.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010797543 2023-11-05T08:33:19.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:19.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.280117+0000", 2023-11-05T08:33:19.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000141755 2023-11-05T08:33:19.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:19.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.282200+0000", 2023-11-05T08:33:19.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002082707 2023-11-05T08:33:19.677 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.677 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:19.677 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.678 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(osd.8.22:2 3.as1 3:56f12b5c:::smithi09039252-15:head [assert-version v2,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2023-11-05T08:33:19.678 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:09.303201+0000", 2023-11-05T08:33:19.678 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 10.205217824, 2023-11-05T08:33:19.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011829628, 2023-11-05T08:33:19.678 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:19.678 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:19.679 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:19.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:19.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.303201+0000", 2023-11-05T08:33:19.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:19.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.303201+0000", 2023-11-05T08:33:19.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:19.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.303203+0000", 2023-11-05T08:33:19.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.526e-06 2023-11-05T08:33:19.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:19.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.303216+0000", 2023-11-05T08:33:19.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2743e-05 2023-11-05T08:33:19.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:19.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.303239+0000", 2023-11-05T08:33:19.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3538999999999999e-05 2023-11-05T08:33:19.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:19.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.303255+0000", 2023-11-05T08:33:19.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5849999999999999e-05 2023-11-05T08:33:19.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:19.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.303449+0000", 2023-11-05T08:33:19.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019373400000000001 2023-11-05T08:33:19.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-11-05T08:33:19.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.303481+0000", 2023-11-05T08:33:19.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1492999999999998e-05 2023-11-05T08:33:19.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:19.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.311128+0000", 2023-11-05T08:33:19.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0076475900000000001 2023-11-05T08:33:19.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:19.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.311382+0000", 2023-11-05T08:33:19.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025368599999999998 2023-11-05T08:33:19.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:19.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.315030+0000", 2023-11-05T08:33:19.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036484669999999999 2023-11-05T08:33:19.689 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.689 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:19.689 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.690 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:378 3.ds0 3:bf537852:::smithi09039252-10:head [read 0~163840,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2023-11-05T08:33:19.690 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:10.278236+0000", 2023-11-05T08:33:19.690 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 9.2301820509999999, 2023-11-05T08:33:19.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011951207, 2023-11-05T08:33:19.691 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:19.691 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:19.691 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:19.691 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:19.691 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:19.692 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 378 2023-11-05T08:33:19.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.692 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:19.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:19.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:10.278236+0000", 2023-11-05T08:33:19.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:19.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:10.278236+0000", 2023-11-05T08:33:19.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:19.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:10.278237+0000", 2023-11-05T08:33:19.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1799999999999995e-07 2023-11-05T08:33:19.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:19.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:10.278242+0000", 2023-11-05T08:33:19.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9660000000000004e-06 2023-11-05T08:33:19.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:19.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:10.278250+0000", 2023-11-05T08:33:19.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8739999999999995e-06 2023-11-05T08:33:19.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:19.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:10.278256+0000", 2023-11-05T08:33:19.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0279999999999997e-06 2023-11-05T08:33:19.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:19.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:10.278560+0000", 2023-11-05T08:33:19.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030377900000000001 2023-11-05T08:33:19.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:33:19.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:10.278573+0000", 2023-11-05T08:33:19.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3706000000000001e-05 2023-11-05T08:33:19.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:19.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:10.284863+0000", 2023-11-05T08:33:19.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.006289963 2023-11-05T08:33:19.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:19.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:10.285046+0000", 2023-11-05T08:33:19.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000182781 2023-11-05T08:33:19.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:19.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:10.290188+0000", 2023-11-05T08:33:19.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0051414920000000001 2023-11-05T08:33:19.702 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.702 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:19.702 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.703 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:379 3.ds0 3:bf537852:::smithi09039252-10:head [read 0~163840,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2023-11-05T08:33:19.703 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:10.278322+0000", 2023-11-05T08:33:19.703 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 9.2300966710000001, 2023-11-05T08:33:19.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01338255, 2023-11-05T08:33:19.704 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:19.704 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:19.704 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:19.704 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:19.705 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:19.705 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 379 2023-11-05T08:33:19.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.705 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:19.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:19.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:10.278322+0000", 2023-11-05T08:33:19.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:19.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:10.278322+0000", 2023-11-05T08:33:19.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:19.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:10.278322+0000", 2023-11-05T08:33:19.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3000000000000001e-07 2023-11-05T08:33:19.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:19.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:10.278326+0000", 2023-11-05T08:33:19.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4139999999999998e-06 2023-11-05T08:33:19.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:19.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:10.278335+0000", 2023-11-05T08:33:19.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8030000000000008e-06 2023-11-05T08:33:19.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:19.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:10.278341+0000", 2023-11-05T08:33:19.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6559999999999997e-06 2023-11-05T08:33:19.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:19.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:10.278660+0000", 2023-11-05T08:33:19.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031871500000000001 2023-11-05T08:33:19.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:33:19.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:10.278687+0000", 2023-11-05T08:33:19.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6829e-05 2023-11-05T08:33:19.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:19.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:10.285854+0000", 2023-11-05T08:33:19.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0071673539999999999 2023-11-05T08:33:19.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:19.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:10.286007+0000", 2023-11-05T08:33:19.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000152583 2023-11-05T08:33:19.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:19.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:10.291704+0000", 2023-11-05T08:33:19.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0056977659999999999 2023-11-05T08:33:19.715 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.716 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:19.716 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.716 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:405 3.as1 3:5396e455:::smithi09039252-43:head [append 0~622592] snapc 1b=[1b,1a,17,7] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2023-11-05T08:33:19.717 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:14.355514+0000", 2023-11-05T08:33:19.717 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.1529040689999999, 2023-11-05T08:33:19.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022581047, 2023-11-05T08:33:19.717 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:19.717 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:19.717 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:19.718 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:19.718 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:19.718 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 405 2023-11-05T08:33:19.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.718 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:19.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:19.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.355514+0000", 2023-11-05T08:33:19.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:19.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.355514+0000", 2023-11-05T08:33:19.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:19.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.355517+0000", 2023-11-05T08:33:19.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7930000000000002e-06 2023-11-05T08:33:19.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:19.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.356220+0000", 2023-11-05T08:33:19.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00070250200000000003 2023-11-05T08:33:19.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:19.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.356253+0000", 2023-11-05T08:33:19.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3719e-05 2023-11-05T08:33:19.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:19.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.356268+0000", 2023-11-05T08:33:19.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5033e-05 2023-11-05T08:33:19.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:19.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.356444+0000", 2023-11-05T08:33:19.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000175549 2023-11-05T08:33:19.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-11-05T08:33:19.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.356471+0000", 2023-11-05T08:33:19.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6755999999999999e-05 2023-11-05T08:33:19.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:19.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.361462+0000", 2023-11-05T08:33:19.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049916350000000003 2023-11-05T08:33:19.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:19.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.361764+0000", 2023-11-05T08:33:19.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030161199999999998 2023-11-05T08:33:19.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:19.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.365071+0000", 2023-11-05T08:33:19.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033064940000000001 2023-11-05T08:33:19.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:19.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.377076+0000", 2023-11-05T08:33:19.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012005969999999999 2023-11-05T08:33:19.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:19.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.377756+0000", 2023-11-05T08:33:19.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067910900000000001 2023-11-05T08:33:19.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:19.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.378095+0000", 2023-11-05T08:33:19.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033987500000000001 2023-11-05T08:33:19.732 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.732 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:19.732 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.733 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:406 3.as1 3:5396e455:::smithi09039252-43:head [append 0~475136] snapc 1b=[1b,1a,17,7] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2023-11-05T08:33:19.733 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:14.363708+0000", 2023-11-05T08:33:19.733 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.1447104499999998, 2023-11-05T08:33:19.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018912735999999999, 2023-11-05T08:33:19.733 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:19.734 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:19.734 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:19.734 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:19.734 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:19.734 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 406 2023-11-05T08:33:19.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.735 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:19.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:19.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.363708+0000", 2023-11-05T08:33:19.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:19.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.363708+0000", 2023-11-05T08:33:19.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:19.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.363710+0000", 2023-11-05T08:33:19.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3700000000000002e-06 2023-11-05T08:33:19.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:19.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.364282+0000", 2023-11-05T08:33:19.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057111800000000002 2023-11-05T08:33:19.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:19.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.364315+0000", 2023-11-05T08:33:19.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3868999999999997e-05 2023-11-05T08:33:19.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:19.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.364331+0000", 2023-11-05T08:33:19.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5130000000000001e-05 2023-11-05T08:33:19.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:19.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.369348+0000", 2023-11-05T08:33:19.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050178250000000001 2023-11-05T08:33:19.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:19.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.369681+0000", 2023-11-05T08:33:19.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033315800000000001 2023-11-05T08:33:19.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:19.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.371729+0000", 2023-11-05T08:33:19.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020475340000000002 2023-11-05T08:33:19.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:19.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.377079+0000", 2023-11-05T08:33:19.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053501529999999999 2023-11-05T08:33:19.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:19.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.382367+0000", 2023-11-05T08:33:19.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052874100000000002 2023-11-05T08:33:19.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:19.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.382621+0000", 2023-11-05T08:33:19.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000254169 2023-11-05T08:33:19.746 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.746 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:19.746 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.747 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:407 3.as1 3:5396e455:::smithi09039252-43:head [append 0~294912] snapc 1b=[1b,1a,17,7] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2023-11-05T08:33:19.747 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:14.369667+0000", 2023-11-05T08:33:19.747 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.1387512600000003, 2023-11-05T08:33:19.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013804643, 2023-11-05T08:33:19.748 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:19.748 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:19.748 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:19.748 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:19.748 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:19.749 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 407 2023-11-05T08:33:19.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.749 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:19.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:19.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.369667+0000", 2023-11-05T08:33:19.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:19.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.369667+0000", 2023-11-05T08:33:19.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:19.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.369670+0000", 2023-11-05T08:33:19.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.044e-06 2023-11-05T08:33:19.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:19.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.369905+0000", 2023-11-05T08:33:19.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023446400000000001 2023-11-05T08:33:19.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:19.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.369940+0000", 2023-11-05T08:33:19.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5159e-05 2023-11-05T08:33:19.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:19.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.369963+0000", 2023-11-05T08:33:19.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3575999999999999e-05 2023-11-05T08:33:19.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:19.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.373576+0000", 2023-11-05T08:33:19.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036127709999999999 2023-11-05T08:33:19.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:19.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.373808+0000", 2023-11-05T08:33:19.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023190999999999999 2023-11-05T08:33:19.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:19.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.375278+0000", 2023-11-05T08:33:19.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014693989999999999 2023-11-05T08:33:19.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.758 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:19.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.378341+0000", 2023-11-05T08:33:19.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030631880000000001 2023-11-05T08:33:19.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:19.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.383024+0000", 2023-11-05T08:33:19.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0046831959999999997 2023-11-05T08:33:19.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:19.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.383472+0000", 2023-11-05T08:33:19.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044793600000000002 2023-11-05T08:33:19.760 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.761 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:19.761 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.761 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:408 3.as1 3:5396e455:::smithi09039252-43:head [setxattr _header (57) in=7b] snapc 1b=[1b,1a,17,7] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2023-11-05T08:33:19.762 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:14.370011+0000", 2023-11-05T08:33:19.762 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.1384070719999997, 2023-11-05T08:33:19.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016422719999999998, 2023-11-05T08:33:19.762 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:19.762 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:19.763 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:19.763 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:19.763 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:19.763 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 408 2023-11-05T08:33:19.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.764 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:19.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:19.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.370011+0000", 2023-11-05T08:33:19.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:19.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.370011+0000", 2023-11-05T08:33:19.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:19.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.370013+0000", 2023-11-05T08:33:19.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7370000000000001e-06 2023-11-05T08:33:19.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:19.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.370024+0000", 2023-11-05T08:33:19.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0976e-05 2023-11-05T08:33:19.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:19.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.370046+0000", 2023-11-05T08:33:19.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2065999999999998e-05 2023-11-05T08:33:19.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.769 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:19.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.370062+0000", 2023-11-05T08:33:19.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5407e-05 2023-11-05T08:33:19.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:19.770 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.378961+0000", 2023-11-05T08:33:19.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0088990579999999996 2023-11-05T08:33:19.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:19.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.379295+0000", 2023-11-05T08:33:19.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033471100000000001 2023-11-05T08:33:19.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:19.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.380741+0000", 2023-11-05T08:33:19.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014457129999999999 2023-11-05T08:33:19.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:19.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.383921+0000", 2023-11-05T08:33:19.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031802660000000002 2023-11-05T08:33:19.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:19.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.386122+0000", 2023-11-05T08:33:19.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022002219999999999 2023-11-05T08:33:19.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:19.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.386434+0000", 2023-11-05T08:33:19.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031256399999999998 2023-11-05T08:33:19.775 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.775 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:19.775 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.776 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:409 3.as1 3:5396e455:::smithi09039252-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e61)", 2023-11-05T08:33:19.776 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:14.370085+0000", 2023-11-05T08:33:19.776 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.1383331200000004, 2023-11-05T08:33:19.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019674614999999999, 2023-11-05T08:33:19.777 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:19.777 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:19.777 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:19.777 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:19.777 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:19.778 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 409 2023-11-05T08:33:19.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.778 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:19.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:19.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.370085+0000", 2023-11-05T08:33:19.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:19.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.370085+0000", 2023-11-05T08:33:19.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:19.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:19.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.370087+0000", 2023-11-05T08:33:19.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2729999999999999e-06 2023-11-05T08:33:19.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:19.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.370093+0000", 2023-11-05T08:33:19.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0700000000000003e-06 2023-11-05T08:33:19.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:19.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.370113+0000", 2023-11-05T08:33:19.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0276e-05 2023-11-05T08:33:19.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:19.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.370132+0000", 2023-11-05T08:33:19.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8626000000000001e-05 2023-11-05T08:33:19.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:19.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.383576+0000", 2023-11-05T08:33:19.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013444717 2023-11-05T08:33:19.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-11-05T08:33:19.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.383807+0000", 2023-11-05T08:33:19.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023070100000000001 2023-11-05T08:33:19.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:19.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.387071+0000", 2023-11-05T08:33:19.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032641710000000002 2023-11-05T08:33:19.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:19.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.387425+0000", 2023-11-05T08:33:19.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035352400000000003 2023-11-05T08:33:19.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:19.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:19.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:19.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.389760+0000", 2023-11-05T08:33:19.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023352569999999999 2023-11-05T08:33:19.788 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.788 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:19.789 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.789 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:19.789 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:19.789 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:33:19.791 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2023-11-05T08:33:19.792 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:33:19.786+0000 7fdc5cd58640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:33:19.891 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:33:19.887+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:33:19.967 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:33:19.968 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-11-05T08:33:19.968 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-11-05T08:33:19.968 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:33:19.981 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2023-11-05T08:33:19.993 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:33:19.990+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:33:20.094 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:33:20.090+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:33:20.150 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:33:20.150 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-11-05T08:33:20.150 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-11-05T08:33:20.150 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-11-05T08:33:20.151 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:33:20.163 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2023-11-05T08:33:20.211 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:20.207+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:20.290 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:33:20.291 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-11-05T08:33:20.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-11-05T08:33:20.291 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-11-05T08:33:20.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.292 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.es3 41/30 ECSubWrite(tid=40, reqid=osd.2.0:40, at_version=41'27, trim_to=0'0, roll_forward_to=40'25))", 2023-11-05T08:33:20.292 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:54.120247+0000", 2023-11-05T08:33:20.292 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 26.166741762000001, 2023-11-05T08:33:20.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0072038550000000003, 2023-11-05T08:33:20.293 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:20.293 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:20.293 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:20.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:20.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.120247+0000", 2023-11-05T08:33:20.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:20.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.120247+0000", 2023-11-05T08:33:20.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:20.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.120247+0000", 2023-11-05T08:33:20.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7000000000000005e-07 2023-11-05T08:33:20.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:20.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.120307+0000", 2023-11-05T08:33:20.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9858000000000002e-05 2023-11-05T08:33:20.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:20.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.120340+0000", 2023-11-05T08:33:20.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3000999999999998e-05 2023-11-05T08:33:20.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:20.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.120365+0000", 2023-11-05T08:33:20.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5232000000000002e-05 2023-11-05T08:33:20.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:20.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.120516+0000", 2023-11-05T08:33:20.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015087400000000001 2023-11-05T08:33:20.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:20.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.120542+0000", 2023-11-05T08:33:20.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5562e-05 2023-11-05T08:33:20.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:20.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.127420+0000", 2023-11-05T08:33:20.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.006878009 2023-11-05T08:33:20.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:20.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.127450+0000", 2023-11-05T08:33:20.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0749e-05 2023-11-05T08:33:20.304 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.304 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:20.304 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.305 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.as6 41/33 ECSubWrite(tid=35, reqid=osd.7.0:35, at_version=41'12, trim_to=0'0, roll_forward_to=40'10))", 2023-11-05T08:33:20.305 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:54.126816+0000", 2023-11-05T08:33:20.305 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 26.160172081999999, 2023-11-05T08:33:20.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0080159169999999991, 2023-11-05T08:33:20.306 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:20.306 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:20.306 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:20.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:20.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.126816+0000", 2023-11-05T08:33:20.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:20.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.126816+0000", 2023-11-05T08:33:20.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:20.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.126817+0000", 2023-11-05T08:33:20.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0999999999999998e-07 2023-11-05T08:33:20.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:20.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.126872+0000", 2023-11-05T08:33:20.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4970999999999999e-05 2023-11-05T08:33:20.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:20.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.126902+0000", 2023-11-05T08:33:20.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0045000000000001e-05 2023-11-05T08:33:20.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:20.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.126927+0000", 2023-11-05T08:33:20.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4586000000000001e-05 2023-11-05T08:33:20.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:20.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.127271+0000", 2023-11-05T08:33:20.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034440499999999999 2023-11-05T08:33:20.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:20.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.127298+0000", 2023-11-05T08:33:20.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6724999999999999e-05 2023-11-05T08:33:20.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:20.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.134738+0000", 2023-11-05T08:33:20.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0074405959999999998 2023-11-05T08:33:20.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:20.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.134832+0000", 2023-11-05T08:33:20.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3979000000000003e-05 2023-11-05T08:33:20.317 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.317 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:20.317 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.318 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.as6 42/33 ECSubWrite(tid=38, reqid=osd.7.0:38, at_version=42'16, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:33:20.318 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:55.122849+0000", 2023-11-05T08:33:20.319 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 25.164139526, 2023-11-05T08:33:20.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015960385000000001, 2023-11-05T08:33:20.319 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:20.319 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:20.319 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:20.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:20.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.122849+0000", 2023-11-05T08:33:20.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:20.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.122849+0000", 2023-11-05T08:33:20.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:20.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.122850+0000", 2023-11-05T08:33:20.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1699999999999997e-07 2023-11-05T08:33:20.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:20.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.122895+0000", 2023-11-05T08:33:20.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5717999999999998e-05 2023-11-05T08:33:20.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:20.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.122919+0000", 2023-11-05T08:33:20.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3932000000000001e-05 2023-11-05T08:33:20.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:20.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.122941+0000", 2023-11-05T08:33:20.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1708999999999998e-05 2023-11-05T08:33:20.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:20.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.125137+0000", 2023-11-05T08:33:20.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021962219999999998 2023-11-05T08:33:20.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:20.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.125161+0000", 2023-11-05T08:33:20.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3436000000000001e-05 2023-11-05T08:33:20.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:20.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.138719+0000", 2023-11-05T08:33:20.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013558115000000001 2023-11-05T08:33:20.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:20.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.138809+0000", 2023-11-05T08:33:20.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0536000000000002e-05 2023-11-05T08:33:20.330 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.331 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:20.331 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.332 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.as6 42/33 ECSubWrite(tid=39, reqid=osd.7.0:39, at_version=42'18, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:33:20.332 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:55.125666+0000", 2023-11-05T08:33:20.332 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 25.161322367, 2023-11-05T08:33:20.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013940509, 2023-11-05T08:33:20.332 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:20.333 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:20.333 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:20.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:20.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.125666+0000", 2023-11-05T08:33:20.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:20.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.125666+0000", 2023-11-05T08:33:20.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:20.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.125667+0000", 2023-11-05T08:33:20.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.47e-07 2023-11-05T08:33:20.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:20.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.125713+0000", 2023-11-05T08:33:20.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5748000000000003e-05 2023-11-05T08:33:20.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:20.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.125737+0000", 2023-11-05T08:33:20.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4369e-05 2023-11-05T08:33:20.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:20.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.125758+0000", 2023-11-05T08:33:20.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0608999999999999e-05 2023-11-05T08:33:20.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:20.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.126227+0000", 2023-11-05T08:33:20.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000468944 2023-11-05T08:33:20.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:20.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.126251+0000", 2023-11-05T08:33:20.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4788999999999999e-05 2023-11-05T08:33:20.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:20.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.139523+0000", 2023-11-05T08:33:20.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013271577 2023-11-05T08:33:20.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:20.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.139607+0000", 2023-11-05T08:33:20.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3526e-05 2023-11-05T08:33:20.344 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.344 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:20.344 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.345 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.es3 42/30 ECSubWrite(tid=43, reqid=osd.2.0:43, at_version=42'29, trim_to=0'0, roll_forward_to=41'27))", 2023-11-05T08:33:20.345 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:55.134690+0000", 2023-11-05T08:33:20.345 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 25.152298163000001, 2023-11-05T08:33:20.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.007527003, 2023-11-05T08:33:20.346 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:20.346 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:20.346 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:20.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:20.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.134690+0000", 2023-11-05T08:33:20.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:20.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.134690+0000", 2023-11-05T08:33:20.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:20.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.134692+0000", 2023-11-05T08:33:20.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4699999999999999e-06 2023-11-05T08:33:20.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:20.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.134749+0000", 2023-11-05T08:33:20.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6941000000000003e-05 2023-11-05T08:33:20.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:20.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.134788+0000", 2023-11-05T08:33:20.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8952000000000002e-05 2023-11-05T08:33:20.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.352 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:20.352 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.134810+0000", 2023-11-05T08:33:20.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2785e-05 2023-11-05T08:33:20.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.353 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:20.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.135041+0000", 2023-11-05T08:33:20.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000230604 2023-11-05T08:33:20.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:20.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.135067+0000", 2023-11-05T08:33:20.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6012000000000001e-05 2023-11-05T08:33:20.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:20.356 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.142127+0000", 2023-11-05T08:33:20.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0070602629999999998 2023-11-05T08:33:20.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:20.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.142217+0000", 2023-11-05T08:33:20.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9975999999999994e-05 2023-11-05T08:33:20.357 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.358 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:20.358 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.359 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.11s1 46/45 ECSubWrite(tid=0, reqid=osd.10.0:0, at_version=46'17, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:20.359 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:59.221890+0000", 2023-11-05T08:33:20.359 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 21.065098873, 2023-11-05T08:33:20.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0086406989999999999, 2023-11-05T08:33:20.359 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:20.360 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:20.360 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:20.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:20.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.221890+0000", 2023-11-05T08:33:20.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:20.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.221890+0000", 2023-11-05T08:33:20.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:20.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.221891+0000", 2023-11-05T08:33:20.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.536e-06 2023-11-05T08:33:20.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:20.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.221950+0000", 2023-11-05T08:33:20.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8615e-05 2023-11-05T08:33:20.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:20.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.221983+0000", 2023-11-05T08:33:20.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3225000000000001e-05 2023-11-05T08:33:20.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:20.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222006+0000", 2023-11-05T08:33:20.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3445999999999999e-05 2023-11-05T08:33:20.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:20.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222169+0000", 2023-11-05T08:33:20.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016258700000000001 2023-11-05T08:33:20.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:20.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222195+0000", 2023-11-05T08:33:20.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6041999999999999e-05 2023-11-05T08:33:20.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:20.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.230485+0000", 2023-11-05T08:33:20.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0082901520000000003 2023-11-05T08:33:20.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:20.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.230530+0000", 2023-11-05T08:33:20.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5096000000000003e-05 2023-11-05T08:33:20.371 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.371 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:20.371 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.372 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:346 3.fs0 3:f2348bf8:::smithi09039252-39:head [read 0~638976,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:20.372 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:05.254913+0000", 2023-11-05T08:33:20.372 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.032075150000001, 2023-11-05T08:33:20.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019954645999999999, 2023-11-05T08:33:20.373 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:20.373 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:20.373 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:20.374 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:20.374 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:20.374 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 346 2023-11-05T08:33:20.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.374 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:20.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:20.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.254913+0000", 2023-11-05T08:33:20.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:20.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.254913+0000", 2023-11-05T08:33:20.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:20.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.254917+0000", 2023-11-05T08:33:20.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1330000000000001e-06 2023-11-05T08:33:20.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:20.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.254934+0000", 2023-11-05T08:33:20.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6897000000000001e-05 2023-11-05T08:33:20.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:20.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.254965+0000", 2023-11-05T08:33:20.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0769000000000003e-05 2023-11-05T08:33:20.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:20.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.254981+0000", 2023-11-05T08:33:20.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6015000000000002e-05 2023-11-05T08:33:20.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:20.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255140+0000", 2023-11-05T08:33:20.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015884499999999999 2023-11-05T08:33:20.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-11-05T08:33:20.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255161+0000", 2023-11-05T08:33:20.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1033e-05 2023-11-05T08:33:20.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:20.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.261568+0000", 2023-11-05T08:33:20.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0064071639999999999 2023-11-05T08:33:20.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:20.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.261728+0000", 2023-11-05T08:33:20.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015980500000000001 2023-11-05T08:33:20.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:20.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.274868+0000", 2023-11-05T08:33:20.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013139985 2023-11-05T08:33:20.386 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.387 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:20.387 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.388 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:347 3.fs0 3:f2348bf8:::smithi09039252-39:head [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:20.388 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:05.255019+0000", 2023-11-05T08:33:20.388 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.031968933, 2023-11-05T08:33:20.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021017214999999999, 2023-11-05T08:33:20.388 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:20.389 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:20.389 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:20.389 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:20.389 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:20.389 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 347 2023-11-05T08:33:20.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.390 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:20.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:20.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255019+0000", 2023-11-05T08:33:20.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:20.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255019+0000", 2023-11-05T08:33:20.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:20.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255021+0000", 2023-11-05T08:33:20.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1260000000000001e-06 2023-11-05T08:33:20.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:20.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255026+0000", 2023-11-05T08:33:20.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8640000000000001e-06 2023-11-05T08:33:20.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:20.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255039+0000", 2023-11-05T08:33:20.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2238999999999999e-05 2023-11-05T08:33:20.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:20.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255047+0000", 2023-11-05T08:33:20.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7159999999999992e-06 2023-11-05T08:33:20.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:20.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255354+0000", 2023-11-05T08:33:20.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030627999999999997 2023-11-05T08:33:20.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:33:20.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255382+0000", 2023-11-05T08:33:20.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8439e-05 2023-11-05T08:33:20.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:20.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.266020+0000", 2023-11-05T08:33:20.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010637807000000001 2023-11-05T08:33:20.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:20.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.266295+0000", 2023-11-05T08:33:20.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027543400000000002 2023-11-05T08:33:20.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:20.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.276037+0000", 2023-11-05T08:33:20.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0097413099999999996 2023-11-05T08:33:20.401 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.402 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:20.402 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.403 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:348 3.fs0 3:f2348bf8:::smithi09039252-39:head [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:20.403 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:05.255103+0000", 2023-11-05T08:33:20.403 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.031884976000001, 2023-11-05T08:33:20.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022537980999999999, 2023-11-05T08:33:20.403 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:20.404 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:20.404 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:20.404 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:20.404 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:20.404 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 348 2023-11-05T08:33:20.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.405 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:20.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:20.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255103+0000", 2023-11-05T08:33:20.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:20.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255103+0000", 2023-11-05T08:33:20.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.407 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:20.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255105+0000", 2023-11-05T08:33:20.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.24e-06 2023-11-05T08:33:20.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:20.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255136+0000", 2023-11-05T08:33:20.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0979000000000003e-05 2023-11-05T08:33:20.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:20.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255164+0000", 2023-11-05T08:33:20.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8662000000000001e-05 2023-11-05T08:33:20.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:20.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255181+0000", 2023-11-05T08:33:20.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6935999999999998e-05 2023-11-05T08:33:20.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:20.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255509+0000", 2023-11-05T08:33:20.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032754300000000002 2023-11-05T08:33:20.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:33:20.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255536+0000", 2023-11-05T08:33:20.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6889e-05 2023-11-05T08:33:20.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:20.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.267115+0000", 2023-11-05T08:33:20.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011579104 2023-11-05T08:33:20.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:20.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.267222+0000", 2023-11-05T08:33:20.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106864 2023-11-05T08:33:20.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:20.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.277641+0000", 2023-11-05T08:33:20.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010419764 2023-11-05T08:33:20.416 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.417 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:20.417 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.418 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:369 3.16s0 3:6df625fb:::smithi09039252-31:head [append 0~524288] snapc 15=[7] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2023-11-05T08:33:20.418 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:09.281473+0000", 2023-11-05T08:33:20.418 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.005515249, 2023-11-05T08:33:20.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020569061999999999, 2023-11-05T08:33:20.419 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:20.419 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:20.419 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:20.419 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:20.419 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:20.420 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 369 2023-11-05T08:33:20.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.420 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:20.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:20.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.281473+0000", 2023-11-05T08:33:20.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:20.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.281473+0000", 2023-11-05T08:33:20.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:20.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.281477+0000", 2023-11-05T08:33:20.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.98e-06 2023-11-05T08:33:20.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:20.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.282265+0000", 2023-11-05T08:33:20.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00078817999999999998 2023-11-05T08:33:20.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:20.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.282308+0000", 2023-11-05T08:33:20.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2775999999999998e-05 2023-11-05T08:33:20.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:20.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.282331+0000", 2023-11-05T08:33:20.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3332999999999998e-05 2023-11-05T08:33:20.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:20.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.282525+0000", 2023-11-05T08:33:20.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000193671 2023-11-05T08:33:20.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-11-05T08:33:20.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.282555+0000", 2023-11-05T08:33:20.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9669e-05 2023-11-05T08:33:20.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:20.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.287913+0000", 2023-11-05T08:33:20.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053581360000000003 2023-11-05T08:33:20.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:20.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.288217+0000", 2023-11-05T08:33:20.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030429900000000001 2023-11-05T08:33:20.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:20.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.291625+0000", 2023-11-05T08:33:20.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034076779999999999 2023-11-05T08:33:20.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:20.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.301060+0000", 2023-11-05T08:33:20.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0094351580000000008 2023-11-05T08:33:20.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:20.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.301904+0000", 2023-11-05T08:33:20.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00084351399999999996 2023-11-05T08:33:20.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:20.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.302042+0000", 2023-11-05T08:33:20.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013866799999999999 2023-11-05T08:33:20.436 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.436 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:20.436 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.437 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:370 3.16s0 3:6df625fb:::smithi09039252-31:head [append 0~524288] snapc 15=[7] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2023-11-05T08:33:20.437 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:09.290980+0000", 2023-11-05T08:33:20.437 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 10.996007930999999, 2023-11-05T08:33:20.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019531185999999999, 2023-11-05T08:33:20.438 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:20.438 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:20.438 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:20.438 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:20.439 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:20.439 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 370 2023-11-05T08:33:20.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.439 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:20.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:20.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.290980+0000", 2023-11-05T08:33:20.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:20.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.290980+0000", 2023-11-05T08:33:20.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:20.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.290983+0000", 2023-11-05T08:33:20.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9560000000000002e-06 2023-11-05T08:33:20.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:20.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.291841+0000", 2023-11-05T08:33:20.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00085721599999999995 2023-11-05T08:33:20.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:20.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.291879+0000", 2023-11-05T08:33:20.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7882000000000001e-05 2023-11-05T08:33:20.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:20.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.291904+0000", 2023-11-05T08:33:20.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5658999999999999e-05 2023-11-05T08:33:20.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:20.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.294143+0000", 2023-11-05T08:33:20.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022388830000000001 2023-11-05T08:33:20.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:20.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.294499+0000", 2023-11-05T08:33:20.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035557600000000003 2023-11-05T08:33:20.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:20.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.296802+0000", 2023-11-05T08:33:20.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023036509999999999 2023-11-05T08:33:20.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:20.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.302059+0000", 2023-11-05T08:33:20.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052565379999999998 2023-11-05T08:33:20.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:20.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.310264+0000", 2023-11-05T08:33:20.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0082055919999999994 2023-11-05T08:33:20.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:20.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.310512+0000", 2023-11-05T08:33:20.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024723300000000002 2023-11-05T08:33:20.452 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.452 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:20.452 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.453 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:371 3.16s0 3:6df625fb:::smithi09039252-31:head [append 0~245760] snapc 15=[7] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2023-11-05T08:33:20.453 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:09.296540+0000", 2023-11-05T08:33:20.454 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 10.990448021000001, 2023-11-05T08:33:20.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015979499000000001, 2023-11-05T08:33:20.454 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:20.454 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:20.454 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:20.455 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:20.455 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:20.455 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 371 2023-11-05T08:33:20.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.455 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:20.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:20.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.296540+0000", 2023-11-05T08:33:20.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:20.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.296540+0000", 2023-11-05T08:33:20.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:20.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.296543+0000", 2023-11-05T08:33:20.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8329999999999998e-06 2023-11-05T08:33:20.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:20.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.296864+0000", 2023-11-05T08:33:20.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032048100000000001 2023-11-05T08:33:20.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:20.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.296899+0000", 2023-11-05T08:33:20.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5735999999999998e-05 2023-11-05T08:33:20.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:20.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.296920+0000", 2023-11-05T08:33:20.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0268999999999999e-05 2023-11-05T08:33:20.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.462 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:20.462 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.302604+0000", 2023-11-05T08:33:20.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0056841879999999997 2023-11-05T08:33:20.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:20.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.302966+0000", 2023-11-05T08:33:20.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036163200000000002 2023-11-05T08:33:20.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:20.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.304745+0000", 2023-11-05T08:33:20.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001779596 2023-11-05T08:33:20.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:20.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.309738+0000", 2023-11-05T08:33:20.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049929500000000003 2023-11-05T08:33:20.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:20.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.312278+0000", 2023-11-05T08:33:20.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002540231 2023-11-05T08:33:20.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:20.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.312520+0000", 2023-11-05T08:33:20.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024158299999999999 2023-11-05T08:33:20.468 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.469 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:20.469 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.469 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:372 3.16s0 3:6df625fb:::smithi09039252-31:head [setxattr _header (57) in=7b] snapc 15=[7] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2023-11-05T08:33:20.470 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:09.296991+0000", 2023-11-05T08:33:20.470 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 10.989997817000001, 2023-11-05T08:33:20.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016671208, 2023-11-05T08:33:20.471 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:20.471 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:20.471 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:20.471 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:20.471 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:20.472 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 372 2023-11-05T08:33:20.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.472 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:20.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:20.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.296991+0000", 2023-11-05T08:33:20.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:20.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.296991+0000", 2023-11-05T08:33:20.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:20.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.296992+0000", 2023-11-05T08:33:20.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5230000000000001e-06 2023-11-05T08:33:20.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:20.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297001+0000", 2023-11-05T08:33:20.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1570000000000006e-06 2023-11-05T08:33:20.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:20.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297023+0000", 2023-11-05T08:33:20.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1965999999999999e-05 2023-11-05T08:33:20.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:20.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297074+0000", 2023-11-05T08:33:20.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0875000000000003e-05 2023-11-05T08:33:20.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:20.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.306255+0000", 2023-11-05T08:33:20.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0091812700000000001 2023-11-05T08:33:20.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:20.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.306557+0000", 2023-11-05T08:33:20.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030117799999999998 2023-11-05T08:33:20.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:20.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.308901+0000", 2023-11-05T08:33:20.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023442990000000002 2023-11-05T08:33:20.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:20.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.312840+0000", 2023-11-05T08:33:20.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039388670000000004 2023-11-05T08:33:20.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:20.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.313428+0000", 2023-11-05T08:33:20.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058814000000000002 2023-11-05T08:33:20.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:20.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.313662+0000", 2023-11-05T08:33:20.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000233933 2023-11-05T08:33:20.485 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.485 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:20.486 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.486 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:373 3.16s0 3:6df625fb:::smithi09039252-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e56)", 2023-11-05T08:33:20.487 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:09.297100+0000", 2023-11-05T08:33:20.487 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 10.989888174000001, 2023-11-05T08:33:20.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020900037999999999, 2023-11-05T08:33:20.487 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:20.487 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:20.488 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:20.488 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:20.488 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:20.488 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 373 2023-11-05T08:33:20.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.489 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:20.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:20.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297100+0000", 2023-11-05T08:33:20.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:20.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297100+0000", 2023-11-05T08:33:20.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:20.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297102+0000", 2023-11-05T08:33:20.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.454e-06 2023-11-05T08:33:20.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:20.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297108+0000", 2023-11-05T08:33:20.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8260000000000004e-06 2023-11-05T08:33:20.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:20.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297131+0000", 2023-11-05T08:33:20.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2042999999999999e-05 2023-11-05T08:33:20.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:20.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297145+0000", 2023-11-05T08:33:20.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4783000000000001e-05 2023-11-05T08:33:20.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:20.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.314858+0000", 2023-11-05T08:33:20.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017712525 2023-11-05T08:33:20.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:20.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.315292+0000", 2023-11-05T08:33:20.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043406100000000002 2023-11-05T08:33:20.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:20.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.318000+0000", 2023-11-05T08:33:20.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0027083459999999999 2023-11-05T08:33:20.498 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.498 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:20.499 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.499 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.as6 61/45 ECSubWrite(tid=57, reqid=client.4691.0:405, at_version=61'23, trim_to=0'0, roll_forward_to=52'21))", 2023-11-05T08:33:20.499 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:14.365081+0000", 2023-11-05T08:33:20.500 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.9219075720000003, 2023-11-05T08:33:20.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0071177870000000004, 2023-11-05T08:33:20.500 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:20.500 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:20.500 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:20.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:20.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.365081+0000", 2023-11-05T08:33:20.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:20.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.365081+0000", 2023-11-05T08:33:20.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:20.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.365081+0000", 2023-11-05T08:33:20.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6000000000000003e-07 2023-11-05T08:33:20.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:20.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.365280+0000", 2023-11-05T08:33:20.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019899100000000001 2023-11-05T08:33:20.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:20.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.365316+0000", 2023-11-05T08:33:20.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5043000000000003e-05 2023-11-05T08:33:20.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:20.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.365494+0000", 2023-11-05T08:33:20.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017802200000000001 2023-11-05T08:33:20.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:20.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.365821+0000", 2023-11-05T08:33:20.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032697800000000001 2023-11-05T08:33:20.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:20.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.365850+0000", 2023-11-05T08:33:20.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9076000000000001e-05 2023-11-05T08:33:20.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:20.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.372106+0000", 2023-11-05T08:33:20.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0062559470000000004 2023-11-05T08:33:20.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:20.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.372199+0000", 2023-11-05T08:33:20.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3070000000000002e-05 2023-11-05T08:33:20.511 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.511 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:20.512 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.512 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:493 3.fs0 3:f9a1f673:::smithi09039252-46:head [copy-from ver 2] snapc 1c=[1a,7] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2023-11-05T08:33:20.513 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:18.605410+0000", 2023-11-05T08:33:20.513 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.6815779850000001, 2023-11-05T08:33:20.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015750235000000001, 2023-11-05T08:33:20.513 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:20.513 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:20.514 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:20.514 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:20.514 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:20.514 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 493 2023-11-05T08:33:20.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.515 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:20.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:20.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.605410+0000", 2023-11-05T08:33:20.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:20.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.605410+0000", 2023-11-05T08:33:20.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:20.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.605411+0000", 2023-11-05T08:33:20.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0100000000000005e-07 2023-11-05T08:33:20.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:20.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.605414+0000", 2023-11-05T08:33:20.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5039999999999998e-06 2023-11-05T08:33:20.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:20.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.605423+0000", 2023-11-05T08:33:20.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6759999999999996e-06 2023-11-05T08:33:20.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:20.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.605429+0000", 2023-11-05T08:33:20.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.93e-06 2023-11-05T08:33:20.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:20.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.609936+0000", 2023-11-05T08:33:20.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0045067650000000002 2023-11-05T08:33:20.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:20.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.610130+0000", 2023-11-05T08:33:20.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019458499999999999 2023-11-05T08:33:20.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:20.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.616944+0000", 2023-11-05T08:33:20.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0068134689999999999 2023-11-05T08:33:20.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:20.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.619996+0000", 2023-11-05T08:33:20.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030521609999999998 2023-11-05T08:33:20.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:20.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.621010+0000", 2023-11-05T08:33:20.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001014048 2023-11-05T08:33:20.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:20.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.621161+0000", 2023-11-05T08:33:20.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015059599999999999 2023-11-05T08:33:20.528 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.528 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:20.528 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.529 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:494 3.fs0 3:f9a1f673:::smithi09039252-46:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2023-11-05T08:33:20.529 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:18.605437+0000", 2023-11-05T08:33:20.529 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.6815509180000001, 2023-11-05T08:33:20.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014062043, 2023-11-05T08:33:20.530 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:20.530 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:20.530 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:20.530 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:20.531 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:20.531 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 494 2023-11-05T08:33:20.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.531 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:20.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:20.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.605437+0000", 2023-11-05T08:33:20.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:20.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.605437+0000", 2023-11-05T08:33:20.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:20.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.605438+0000", 2023-11-05T08:33:20.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2300000000000001e-07 2023-11-05T08:33:20.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:20.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.605441+0000", 2023-11-05T08:33:20.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6180000000000002e-06 2023-11-05T08:33:20.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:20.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.605455+0000", 2023-11-05T08:33:20.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4464000000000001e-05 2023-11-05T08:33:20.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:20.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.605461+0000", 2023-11-05T08:33:20.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6149999999999996e-06 2023-11-05T08:33:20.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:20.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.611206+0000", 2023-11-05T08:33:20.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0057451589999999997 2023-11-05T08:33:20.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-11-05T08:33:20.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.611315+0000", 2023-11-05T08:33:20.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108844 2023-11-05T08:33:20.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:20.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.619080+0000", 2023-11-05T08:33:20.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0077654639999999997 2023-11-05T08:33:20.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:20.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.619201+0000", 2023-11-05T08:33:20.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121274 2023-11-05T08:33:20.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:20.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.619500+0000", 2023-11-05T08:33:20.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029808200000000002 2023-11-05T08:33:20.543 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.543 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:20.544 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.544 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:501 3.fs0 3:f91b0618:::smithi09039252-26:7 [read 0~425984,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2023-11-05T08:33:20.544 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:19.393991+0000", 2023-11-05T08:33:20.545 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.89299742500000001, 2023-11-05T08:33:20.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016310758000000002, 2023-11-05T08:33:20.545 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:20.545 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:20.545 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:20.546 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:20.546 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:20.546 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 501 2023-11-05T08:33:20.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.547 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:20.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:20.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.393991+0000", 2023-11-05T08:33:20.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:20.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.393991+0000", 2023-11-05T08:33:20.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:20.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.393995+0000", 2023-11-05T08:33:20.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6210000000000001e-06 2023-11-05T08:33:20.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:20.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394010+0000", 2023-11-05T08:33:20.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5065000000000001e-05 2023-11-05T08:33:20.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:20.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394045+0000", 2023-11-05T08:33:20.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5095000000000002e-05 2023-11-05T08:33:20.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:20.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394063+0000", 2023-11-05T08:33:20.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8091e-05 2023-11-05T08:33:20.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:20.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394163+0000", 2023-11-05T08:33:20.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100367 2023-11-05T08:33:20.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-11-05T08:33:20.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394195+0000", 2023-11-05T08:33:20.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1349000000000001e-05 2023-11-05T08:33:20.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:20.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.398429+0000", 2023-11-05T08:33:20.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0042346190000000002 2023-11-05T08:33:20.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:20.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.398635+0000", 2023-11-05T08:33:20.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020568600000000001 2023-11-05T08:33:20.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:20.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.410302+0000", 2023-11-05T08:33:20.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011666865 2023-11-05T08:33:20.559 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.559 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:20.559 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.560 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:502 3.fs0 3:f91b0618:::smithi09039252-26:7 [read 0~425984,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2023-11-05T08:33:20.560 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:19.394105+0000", 2023-11-05T08:33:20.560 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.89288362399999999, 2023-11-05T08:33:20.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017035277000000001, 2023-11-05T08:33:20.561 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:20.561 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:20.561 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:20.561 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:20.562 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:20.562 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 502 2023-11-05T08:33:20.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.562 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:20.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:20.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394105+0000", 2023-11-05T08:33:20.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:20.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394105+0000", 2023-11-05T08:33:20.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:20.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394106+0000", 2023-11-05T08:33:20.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9999999999999996e-07 2023-11-05T08:33:20.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:20.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394111+0000", 2023-11-05T08:33:20.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9620000000000001e-06 2023-11-05T08:33:20.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:20.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394122+0000", 2023-11-05T08:33:20.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1837e-05 2023-11-05T08:33:20.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:20.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394135+0000", 2023-11-05T08:33:20.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3103e-05 2023-11-05T08:33:20.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:20.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394300+0000", 2023-11-05T08:33:20.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016444800000000001 2023-11-05T08:33:20.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:33:20.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394334+0000", 2023-11-05T08:33:20.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4518999999999999e-05 2023-11-05T08:33:20.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:20.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.399381+0000", 2023-11-05T08:33:20.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.005046315 2023-11-05T08:33:20.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:20.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.399560+0000", 2023-11-05T08:33:20.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179457 2023-11-05T08:33:20.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:20.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.411140+0000", 2023-11-05T08:33:20.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011579836 2023-11-05T08:33:20.574 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.574 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:20.574 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.575 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:503 3.fs0 3:f91b0618:::smithi09039252-26:7 [read 0~425984,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2023-11-05T08:33:20.575 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:19.394176+0000", 2023-11-05T08:33:20.576 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.89281221499999996, 2023-11-05T08:33:20.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018069346999999999, 2023-11-05T08:33:20.576 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:20.576 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:20.577 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:20.577 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:20.577 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:20.577 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 503 2023-11-05T08:33:20.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.578 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:20.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:20.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394176+0000", 2023-11-05T08:33:20.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:20.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394176+0000", 2023-11-05T08:33:20.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:20.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:20.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394177+0000", 2023-11-05T08:33:20.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9800000000000003e-07 2023-11-05T08:33:20.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:20.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394183+0000", 2023-11-05T08:33:20.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4489999999999996e-06 2023-11-05T08:33:20.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:20.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394195+0000", 2023-11-05T08:33:20.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1572e-05 2023-11-05T08:33:20.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:20.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394203+0000", 2023-11-05T08:33:20.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0399999999999993e-06 2023-11-05T08:33:20.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:20.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394375+0000", 2023-11-05T08:33:20.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017232799999999999 2023-11-05T08:33:20.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:33:20.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394382+0000", 2023-11-05T08:33:20.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5180000000000001e-06 2023-11-05T08:33:20.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:20.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.401433+0000", 2023-11-05T08:33:20.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0070509659999999997 2023-11-05T08:33:20.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:20.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.401608+0000", 2023-11-05T08:33:20.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017502199999999999 2023-11-05T08:33:20.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:20.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:20.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:20.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.412246+0000", 2023-11-05T08:33:20.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010637654 2023-11-05T08:33:20.590 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.590 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:20.590 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.590 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:20.590 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:20.591 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:33:20.593 INFO:teuthology.orchestra.run.smithi002.stderr:nodeep-scrub is set 2023-11-05T08:33:20.594 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2023-11-05T08:33:20.595 INFO:tasks.rados.rados.0.smithi090.stdout:201: done (7 left) 2023-11-05T08:33:20.595 INFO:tasks.rados.rados.0.smithi090.stdout:202: done (6 left) 2023-11-05T08:33:20.595 INFO:tasks.rados.rados.0.smithi090.stdout:203: done (5 left) 2023-11-05T08:33:20.595 INFO:tasks.rados.rados.0.smithi090.stdout:204: done (4 left) 2023-11-05T08:33:20.596 INFO:tasks.rados.rados.0.smithi090.stdout:206: done (3 left) 2023-11-05T08:33:20.596 INFO:tasks.rados.rados.0.smithi090.stdout:207: done (2 left) 2023-11-05T08:33:20.596 INFO:tasks.rados.rados.0.smithi090.stdout:209: done (1 left) 2023-11-05T08:33:20.596 INFO:tasks.rados.rados.0.smithi090.stdout:210: delete oid 12 current snap is 17 2023-11-05T08:33:20.596 INFO:tasks.rados.rados.0.smithi090.stdout:210: done (1 left) 2023-11-05T08:33:20.597 INFO:tasks.rados.rados.0.smithi090.stdout:211: snap_remove snap 16 2023-11-05T08:33:20.597 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:33:20.590+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:33:20.660 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-05T08:33:20.694 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:33:20.692+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:33:20.766 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:33:20.766 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-11-05T08:33:20.767 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-11-05T08:33:20.767 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:33:20.780 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2023-11-05T08:33:20.794 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:33:20.790+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:33:20.806 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-11-05T08:33:20.828 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1: {} 2023-11-05T08:33:20.828 INFO:teuthology.orchestra.run.smithi002.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-11-05T08:33:20.848 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2: {} 2023-11-05T08:33:20.849 INFO:teuthology.orchestra.run.smithi002.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-11-05T08:33:20.849 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-11-05T08:33:20.867 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4: {} 2023-11-05T08:33:20.868 INFO:teuthology.orchestra.run.smithi002.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-11-05T08:33:20.888 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5: {} 2023-11-05T08:33:20.889 INFO:teuthology.orchestra.run.smithi002.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-11-05T08:33:20.895 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:33:20.894+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:33:20.897 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:33:20.898 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-11-05T08:33:20.898 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-11-05T08:33:20.898 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-11-05T08:33:20.898 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:33:20.909 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6: {} 2023-11-05T08:33:20.909 INFO:teuthology.orchestra.run.smithi002.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-11-05T08:33:20.913 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2023-11-05T08:33:20.928 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7: {} 2023-11-05T08:33:20.928 INFO:teuthology.orchestra.run.smithi002.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-11-05T08:33:20.948 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8: {} 2023-11-05T08:33:20.949 INFO:teuthology.orchestra.run.smithi002.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-11-05T08:33:20.971 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9: {} 2023-11-05T08:33:20.972 INFO:teuthology.orchestra.run.smithi002.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-11-05T08:33:20.994 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10: {} 2023-11-05T08:33:20.995 INFO:teuthology.orchestra.run.smithi002.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-11-05T08:33:20.996 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:20.992+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:21.016 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11: {} 2023-11-05T08:33:21.017 INFO:teuthology.orchestra.run.smithi002.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-11-05T08:33:21.032 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:33:21.033 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-11-05T08:33:21.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-11-05T08:33:21.033 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-11-05T08:33:21.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.034 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.3s3 36/30 ECSubWrite(tid=183, reqid=osd.13.0:183, at_version=36'12, trim_to=0'0, roll_forward_to=35'10))", 2023-11-05T08:33:21.034 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.072455+0000", 2023-11-05T08:33:21.034 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 31.957005299999999, 2023-11-05T08:33:21.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0063400499999999999, 2023-11-05T08:33:21.034 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:21.035 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:21.035 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:21.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:21.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072455+0000", 2023-11-05T08:33:21.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:21.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072455+0000", 2023-11-05T08:33:21.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:21.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072456+0000", 2023-11-05T08:33:21.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9999999999999998e-07 2023-11-05T08:33:21.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:21.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072512+0000", 2023-11-05T08:33:21.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5667000000000001e-05 2023-11-05T08:33:21.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:21.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072537+0000", 2023-11-05T08:33:21.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5833000000000002e-05 2023-11-05T08:33:21.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:21.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072581+0000", 2023-11-05T08:33:21.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3470999999999998e-05 2023-11-05T08:33:21.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:21.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072752+0000", 2023-11-05T08:33:21.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017130100000000001 2023-11-05T08:33:21.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:21.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072775+0000", 2023-11-05T08:33:21.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3379000000000001e-05 2023-11-05T08:33:21.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:21.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.078646+0000", 2023-11-05T08:33:21.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0058709419999999997 2023-11-05T08:33:21.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:21.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.078795+0000", 2023-11-05T08:33:21.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014895699999999999 2023-11-05T08:33:21.044 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.045 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:21.045 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.045 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.8s5 36/30 ECSubWrite(tid=23, reqid=osd.5.0:23, at_version=36'15, trim_to=0'0, roll_forward_to=34'13))", 2023-11-05T08:33:21.046 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.079609+0000", 2023-11-05T08:33:21.046 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 31.949851597999999, 2023-11-05T08:33:21.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049579109999999997, 2023-11-05T08:33:21.046 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:21.046 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:21.047 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:21.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:21.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079609+0000", 2023-11-05T08:33:21.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:21.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079609+0000", 2023-11-05T08:33:21.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:21.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079609+0000", 2023-11-05T08:33:21.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.27e-07 2023-11-05T08:33:21.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:21.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079645+0000", 2023-11-05T08:33:21.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5522000000000003e-05 2023-11-05T08:33:21.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:21.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079672+0000", 2023-11-05T08:33:21.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6648e-05 2023-11-05T08:33:21.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:21.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079693+0000", 2023-11-05T08:33:21.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1206e-05 2023-11-05T08:33:21.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:21.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079849+0000", 2023-11-05T08:33:21.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156112 2023-11-05T08:33:21.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:21.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079873+0000", 2023-11-05T08:33:21.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3589000000000001e-05 2023-11-05T08:33:21.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:21.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.084477+0000", 2023-11-05T08:33:21.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0046040969999999997 2023-11-05T08:33:21.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:21.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.084567+0000", 2023-11-05T08:33:21.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0309999999999994e-05 2023-11-05T08:33:21.057 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.057 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:21.057 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.058 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.8s5 36/30 ECSubWrite(tid=24, reqid=osd.5.0:23, at_version=0'0, trim_to=0'0, roll_forward_to=36'15))", 2023-11-05T08:33:21.058 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.088806+0000", 2023-11-05T08:33:21.058 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 31.940654328000001, 2023-11-05T08:33:21.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055977739999999998, 2023-11-05T08:33:21.058 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:21.059 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:21.059 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:21.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:21.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.088806+0000", 2023-11-05T08:33:21.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:21.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.088806+0000", 2023-11-05T08:33:21.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:21.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.092263+0000", 2023-11-05T08:33:21.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034565210000000002 2023-11-05T08:33:21.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:21.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.092287+0000", 2023-11-05T08:33:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4505999999999999e-05 2023-11-05T08:33:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.092314+0000", 2023-11-05T08:33:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6335000000000001e-05 2023-11-05T08:33:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.092329+0000", 2023-11-05T08:33:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5486e-05 2023-11-05T08:33:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.092519+0000", 2023-11-05T08:33:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019026700000000001 2023-11-05T08:33:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.092545+0000", 2023-11-05T08:33:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5644999999999999e-05 2023-11-05T08:33:21.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:21.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.094327+0000", 2023-11-05T08:33:21.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001781543 2023-11-05T08:33:21.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:21.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.094404+0000", 2023-11-05T08:33:21.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7471000000000004e-05 2023-11-05T08:33:21.069 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.069 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:21.069 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.070 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.2s4 37/30 ECSubWrite(tid=185, reqid=client.4691.0:240, at_version=37'7, trim_to=0'0, roll_forward_to=33'5))", 2023-11-05T08:33:21.070 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:50.088409+0000", 2023-11-05T08:33:21.070 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.941052125999999, 2023-11-05T08:33:21.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036490530000000002, 2023-11-05T08:33:21.070 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:21.071 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:21.071 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:21.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:21.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.088409+0000", 2023-11-05T08:33:21.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:21.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.088409+0000", 2023-11-05T08:33:21.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:21.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.088409+0000", 2023-11-05T08:33:21.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.569999999999999e-07 2023-11-05T08:33:21.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:21.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.088461+0000", 2023-11-05T08:33:21.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1665000000000001e-05 2023-11-05T08:33:21.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:21.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.088486+0000", 2023-11-05T08:33:21.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4757999999999999e-05 2023-11-05T08:33:21.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:21.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.088505+0000", 2023-11-05T08:33:21.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8885999999999998e-05 2023-11-05T08:33:21.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:21.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.088651+0000", 2023-11-05T08:33:21.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014625900000000001 2023-11-05T08:33:21.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:21.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.088678+0000", 2023-11-05T08:33:21.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7229e-05 2023-11-05T08:33:21.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:21.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.091969+0000", 2023-11-05T08:33:21.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032904900000000001 2023-11-05T08:33:21.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:21.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.092058+0000", 2023-11-05T08:33:21.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8808999999999996e-05 2023-11-05T08:33:21.081 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.081 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:21.081 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.082 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.2s4 41/40 ECSubWrite(tid=187, reqid=client.4691.0:292, at_version=41'9, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:21.082 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:54.162954+0000", 2023-11-05T08:33:21.082 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 26.866506204, 2023-11-05T08:33:21.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0045082990000000003, 2023-11-05T08:33:21.082 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:21.082 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:21.083 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:21.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:21.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.162954+0000", 2023-11-05T08:33:21.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:21.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.162954+0000", 2023-11-05T08:33:21.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:21.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.162956+0000", 2023-11-05T08:33:21.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.86e-06 2023-11-05T08:33:21.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:21.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.163012+0000", 2023-11-05T08:33:21.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5744e-05 2023-11-05T08:33:21.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:21.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.163048+0000", 2023-11-05T08:33:21.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6346000000000001e-05 2023-11-05T08:33:21.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:21.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.163075+0000", 2023-11-05T08:33:21.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6622e-05 2023-11-05T08:33:21.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:21.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.163236+0000", 2023-11-05T08:33:21.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016064599999999999 2023-11-05T08:33:21.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:21.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.163262+0000", 2023-11-05T08:33:21.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6356e-05 2023-11-05T08:33:21.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:21.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.167302+0000", 2023-11-05T08:33:21.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040403000000000001 2023-11-05T08:33:21.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:21.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.167463+0000", 2023-11-05T08:33:21.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016042500000000001 2023-11-05T08:33:21.093 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.093 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:21.093 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.094 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.2s4 41/40 ECSubWrite(tid=188, reqid=osd.13.0:188, at_version=41'11, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:21.094 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:54.171988+0000", 2023-11-05T08:33:21.095 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 26.857472674, 2023-11-05T08:33:21.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037888499999999999, 2023-11-05T08:33:21.095 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:21.095 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:21.095 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:21.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:21.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.171988+0000", 2023-11-05T08:33:21.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:21.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.171988+0000", 2023-11-05T08:33:21.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:21.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.171988+0000", 2023-11-05T08:33:21.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1400000000000001e-07 2023-11-05T08:33:21.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:21.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.172004+0000", 2023-11-05T08:33:21.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6308e-05 2023-11-05T08:33:21.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:21.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.172011+0000", 2023-11-05T08:33:21.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9430000000000003e-06 2023-11-05T08:33:21.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:21.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.172018+0000", 2023-11-05T08:33:21.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6250000000000001e-06 2023-11-05T08:33:21.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:21.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.172082+0000", 2023-11-05T08:33:21.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4311000000000001e-05 2023-11-05T08:33:21.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:21.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.172093+0000", 2023-11-05T08:33:21.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1045e-05 2023-11-05T08:33:21.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:21.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.175705+0000", 2023-11-05T08:33:21.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036118819999999999 2023-11-05T08:33:21.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:21.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.175777+0000", 2023-11-05T08:33:21.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1521999999999997e-05 2023-11-05T08:33:21.106 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.106 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:21.106 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.107 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.13s3 41/40 ECSubWrite(tid=190, reqid=osd.13.0:190, at_version=41'14, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:21.107 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:54.189971+0000", 2023-11-05T08:33:21.107 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 26.839489471, 2023-11-05T08:33:21.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003736746, 2023-11-05T08:33:21.108 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:21.108 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:21.108 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:21.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:21.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.189971+0000", 2023-11-05T08:33:21.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:21.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.189971+0000", 2023-11-05T08:33:21.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:21.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.189972+0000", 2023-11-05T08:33:21.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4900000000000005e-07 2023-11-05T08:33:21.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:21.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.190023+0000", 2023-11-05T08:33:21.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1162999999999998e-05 2023-11-05T08:33:21.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:21.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.190044+0000", 2023-11-05T08:33:21.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0761999999999999e-05 2023-11-05T08:33:21.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:21.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.190062+0000", 2023-11-05T08:33:21.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7977000000000001e-05 2023-11-05T08:33:21.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:21.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.190138+0000", 2023-11-05T08:33:21.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5907000000000004e-05 2023-11-05T08:33:21.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:21.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.190148+0000", 2023-11-05T08:33:21.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0344e-05 2023-11-05T08:33:21.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:21.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.193643+0000", 2023-11-05T08:33:21.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003494718 2023-11-05T08:33:21.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:21.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.193708+0000", 2023-11-05T08:33:21.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5025999999999997e-05 2023-11-05T08:33:21.118 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.118 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:21.118 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.119 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.2s4 42/40 ECSubWrite(tid=192, reqid=osd.13.0:192, at_version=42'13, trim_to=0'0, roll_forward_to=41'11))", 2023-11-05T08:33:21.119 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:55.150758+0000", 2023-11-05T08:33:21.119 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 25.878703115, 2023-11-05T08:33:21.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032901889999999998, 2023-11-05T08:33:21.120 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:21.120 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:21.120 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:21.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:21.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.150758+0000", 2023-11-05T08:33:21.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:21.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.150758+0000", 2023-11-05T08:33:21.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:21.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.150758+0000", 2023-11-05T08:33:21.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9999999999999998e-07 2023-11-05T08:33:21.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:21.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.150792+0000", 2023-11-05T08:33:21.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4369999999999998e-05 2023-11-05T08:33:21.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:21.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.150820+0000", 2023-11-05T08:33:21.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7285999999999999e-05 2023-11-05T08:33:21.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:21.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.150833+0000", 2023-11-05T08:33:21.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3603e-05 2023-11-05T08:33:21.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:21.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.150981+0000", 2023-11-05T08:33:21.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000148028 2023-11-05T08:33:21.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:21.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.151008+0000", 2023-11-05T08:33:21.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7169e-05 2023-11-05T08:33:21.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:21.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.153963+0000", 2023-11-05T08:33:21.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029548700000000001 2023-11-05T08:33:21.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:21.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.154048+0000", 2023-11-05T08:33:21.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4363000000000005e-05 2023-11-05T08:33:21.130 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.130 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:21.131 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.131 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.16s1 46/45 ECSubWrite(tid=45, reqid=osd.11.0:45, at_version=46'11, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:21.131 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:59.219281+0000", 2023-11-05T08:33:21.131 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 21.810179188999999, 2023-11-05T08:33:21.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.005540456, 2023-11-05T08:33:21.132 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:21.132 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:21.132 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:21.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:21.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219281+0000", 2023-11-05T08:33:21.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:21.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219281+0000", 2023-11-05T08:33:21.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:21.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219283+0000", 2023-11-05T08:33:21.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.637e-06 2023-11-05T08:33:21.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:21.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219344+0000", 2023-11-05T08:33:21.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0831000000000001e-05 2023-11-05T08:33:21.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:21.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219376+0000", 2023-11-05T08:33:21.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1945e-05 2023-11-05T08:33:21.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:21.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219417+0000", 2023-11-05T08:33:21.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1335000000000003e-05 2023-11-05T08:33:21.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:21.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219601+0000", 2023-11-05T08:33:21.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018345699999999999 2023-11-05T08:33:21.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:21.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219627+0000", 2023-11-05T08:33:21.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6732000000000001e-05 2023-11-05T08:33:21.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:21.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.224749+0000", 2023-11-05T08:33:21.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0051213129999999997 2023-11-05T08:33:21.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:21.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.224822+0000", 2023-11-05T08:33:21.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3206000000000003e-05 2023-11-05T08:33:21.142 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.142 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:21.142 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.143 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.11s2 46/45 ECSubWrite(tid=0, reqid=osd.10.0:0, at_version=46'17, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:21.143 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:59.222006+0000", 2023-11-05T08:33:21.143 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 21.807454496999998, 2023-11-05T08:33:21.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053541609999999996, 2023-11-05T08:33:21.143 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:21.144 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:21.144 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:21.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:21.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222006+0000", 2023-11-05T08:33:21.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:21.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222006+0000", 2023-11-05T08:33:21.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:21.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222008+0000", 2023-11-05T08:33:21.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7430000000000001e-06 2023-11-05T08:33:21.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:21.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222062+0000", 2023-11-05T08:33:21.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3891000000000003e-05 2023-11-05T08:33:21.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:21.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222095+0000", 2023-11-05T08:33:21.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3148999999999997e-05 2023-11-05T08:33:21.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.149 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:21.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222118+0000", 2023-11-05T08:33:21.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3258999999999999e-05 2023-11-05T08:33:21.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:21.150 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222279+0000", 2023-11-05T08:33:21.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016113 2023-11-05T08:33:21.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:21.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222303+0000", 2023-11-05T08:33:21.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4088999999999999e-05 2023-11-05T08:33:21.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:21.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.227275+0000", 2023-11-05T08:33:21.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049715599999999999 2023-11-05T08:33:21.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:21.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.227360+0000", 2023-11-05T08:33:21.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5340000000000006e-05 2023-11-05T08:33:21.153 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.153 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:21.154 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.154 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.11s2 46/45 ECSubWrite(tid=1, reqid=osd.10.0:0, at_version=0'0, trim_to=0'0, roll_forward_to=46'17))", 2023-11-05T08:33:21.154 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:59.232093+0000", 2023-11-05T08:33:21.155 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 21.797368021, 2023-11-05T08:33:21.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037473570000000002, 2023-11-05T08:33:21.155 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:21.155 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:21.155 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:21.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.156 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:21.156 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232093+0000", 2023-11-05T08:33:21.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.157 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:21.157 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232093+0000", 2023-11-05T08:33:21.157 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.158 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:21.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232094+0000", 2023-11-05T08:33:21.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0380000000000001e-06 2023-11-05T08:33:21.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:21.159 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232122+0000", 2023-11-05T08:33:21.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8585000000000002e-05 2023-11-05T08:33:21.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:21.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232142+0000", 2023-11-05T08:33:21.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9357000000000001e-05 2023-11-05T08:33:21.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.161 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:21.161 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232157+0000", 2023-11-05T08:33:21.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5141e-05 2023-11-05T08:33:21.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.162 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:21.162 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232272+0000", 2023-11-05T08:33:21.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114945 2023-11-05T08:33:21.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.162 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:21.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232285+0000", 2023-11-05T08:33:21.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2917e-05 2023-11-05T08:33:21.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:21.164 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.235777+0000", 2023-11-05T08:33:21.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003492069 2023-11-05T08:33:21.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.164 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:21.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.235840+0000", 2023-11-05T08:33:21.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3305000000000005e-05 2023-11-05T08:33:21.165 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.165 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:21.165 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.166 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.16s1 46/45 ECSubWrite(tid=47, reqid=osd.11.0:47, at_version=46'13, trim_to=0'0, roll_forward_to=46'11))", 2023-11-05T08:33:21.166 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:59.233948+0000", 2023-11-05T08:33:21.166 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 21.795512268, 2023-11-05T08:33:21.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048084130000000001, 2023-11-05T08:33:21.167 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:21.167 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:21.167 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:21.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:21.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.233948+0000", 2023-11-05T08:33:21.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:21.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.233948+0000", 2023-11-05T08:33:21.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:21.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.233950+0000", 2023-11-05T08:33:21.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5969999999999999e-06 2023-11-05T08:33:21.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:21.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.234005+0000", 2023-11-05T08:33:21.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5414999999999997e-05 2023-11-05T08:33:21.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:21.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.234040+0000", 2023-11-05T08:33:21.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.455e-05 2023-11-05T08:33:21.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:21.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.234061+0000", 2023-11-05T08:33:21.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1036e-05 2023-11-05T08:33:21.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:21.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.234178+0000", 2023-11-05T08:33:21.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000116822 2023-11-05T08:33:21.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:21.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.234195+0000", 2023-11-05T08:33:21.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6801999999999999e-05 2023-11-05T08:33:21.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:21.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.238676+0000", 2023-11-05T08:33:21.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004481044 2023-11-05T08:33:21.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:21.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.238757+0000", 2023-11-05T08:33:21.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1147000000000001e-05 2023-11-05T08:33:21.176 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.177 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:21.177 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.177 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.16s1 46/45 ECSubWrite(tid=48, reqid=osd.11.0:47, at_version=0'0, trim_to=0'0, roll_forward_to=46'13))", 2023-11-05T08:33:21.177 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:59.240171+0000", 2023-11-05T08:33:21.178 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 21.789289661000002, 2023-11-05T08:33:21.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0045632529999999998, 2023-11-05T08:33:21.178 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:21.178 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:21.178 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:21.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:21.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240171+0000", 2023-11-05T08:33:21.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:21.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240171+0000", 2023-11-05T08:33:21.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:21.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240172+0000", 2023-11-05T08:33:21.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.282e-06 2023-11-05T08:33:21.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:21.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240193+0000", 2023-11-05T08:33:21.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0601000000000001e-05 2023-11-05T08:33:21.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:21.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240214+0000", 2023-11-05T08:33:21.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1574999999999999e-05 2023-11-05T08:33:21.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:21.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240230+0000", 2023-11-05T08:33:21.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5656999999999999e-05 2023-11-05T08:33:21.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:21.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240361+0000", 2023-11-05T08:33:21.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000130488 2023-11-05T08:33:21.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:21.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240372+0000", 2023-11-05T08:33:21.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1158999999999999e-05 2023-11-05T08:33:21.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:21.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.244663+0000", 2023-11-05T08:33:21.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004291563 2023-11-05T08:33:21.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:21.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.244734+0000", 2023-11-05T08:33:21.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0927999999999996e-05 2023-11-05T08:33:21.188 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.188 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:21.188 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.189 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.16s1 51/45 ECSubWrite(tid=58, reqid=osd.11.0:58, at_version=51'19, trim_to=0'0, roll_forward_to=49'17))", 2023-11-05T08:33:21.189 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:04.226221+0000", 2023-11-05T08:33:21.189 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 16.803239889, 2023-11-05T08:33:21.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.005713414, 2023-11-05T08:33:21.189 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:21.189 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:21.190 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:21.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:21.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226221+0000", 2023-11-05T08:33:21.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:21.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226221+0000", 2023-11-05T08:33:21.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:21.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226221+0000", 2023-11-05T08:33:21.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.53e-07 2023-11-05T08:33:21.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:21.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226253+0000", 2023-11-05T08:33:21.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1671999999999998e-05 2023-11-05T08:33:21.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:21.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226269+0000", 2023-11-05T08:33:21.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6150999999999998e-05 2023-11-05T08:33:21.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:21.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226282+0000", 2023-11-05T08:33:21.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3465000000000001e-05 2023-11-05T08:33:21.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:21.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226474+0000", 2023-11-05T08:33:21.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019140499999999999 2023-11-05T08:33:21.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:21.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226501+0000", 2023-11-05T08:33:21.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7523000000000001e-05 2023-11-05T08:33:21.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:21.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.231902+0000", 2023-11-05T08:33:21.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0054011019999999996 2023-11-05T08:33:21.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:21.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.231934+0000", 2023-11-05T08:33:21.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1742999999999997e-05 2023-11-05T08:33:21.199 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.199 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:21.200 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.200 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.9s4 52/45 ECSubWrite(tid=75, reqid=client.4691.0:341, at_version=52'24, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:21.200 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:05.257804+0000", 2023-11-05T08:33:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.771656701, 2023-11-05T08:33:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010353123000000001, 2023-11-05T08:33:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:21.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.257804+0000", 2023-11-05T08:33:21.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:21.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.257804+0000", 2023-11-05T08:33:21.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:21.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.257805+0000", 2023-11-05T08:33:21.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3399999999999999e-07 2023-11-05T08:33:21.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:21.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.258142+0000", 2023-11-05T08:33:21.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033764100000000002 2023-11-05T08:33:21.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:21.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.258157+0000", 2023-11-05T08:33:21.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4625e-05 2023-11-05T08:33:21.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:21.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.258168+0000", 2023-11-05T08:33:21.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0992000000000001e-05 2023-11-05T08:33:21.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:21.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.258371+0000", 2023-11-05T08:33:21.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020296299999999999 2023-11-05T08:33:21.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:21.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.258404+0000", 2023-11-05T08:33:21.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2765999999999998e-05 2023-11-05T08:33:21.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:21.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.268067+0000", 2023-11-05T08:33:21.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0096629479999999993 2023-11-05T08:33:21.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:21.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.268157+0000", 2023-11-05T08:33:21.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0554000000000001e-05 2023-11-05T08:33:21.211 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.211 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:21.211 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.212 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(osd.14.22:3 3.5s0 3:a287d13d:::smithi09039252-33:head [assert-version v4,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:21.212 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:05.264331+0000", 2023-11-05T08:33:21.212 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.765130097, 2023-11-05T08:33:21.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010947936, 2023-11-05T08:33:21.213 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:21.213 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:21.213 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:21.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.264331+0000", 2023-11-05T08:33:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.264331+0000", 2023-11-05T08:33:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.264334+0000", 2023-11-05T08:33:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.467e-06 2023-11-05T08:33:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.264350+0000", 2023-11-05T08:33:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6200000000000001e-05 2023-11-05T08:33:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.264387+0000", 2023-11-05T08:33:21.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6684999999999999e-05 2023-11-05T08:33:21.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:21.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.264489+0000", 2023-11-05T08:33:21.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010218 2023-11-05T08:33:21.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:21.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.265518+0000", 2023-11-05T08:33:21.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0010291899999999999 2023-11-05T08:33:21.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-11-05T08:33:21.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.265535+0000", 2023-11-05T08:33:21.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6697e-05 2023-11-05T08:33:21.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:21.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.269530+0000", 2023-11-05T08:33:21.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039951769999999999 2023-11-05T08:33:21.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:21.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.269646+0000", 2023-11-05T08:33:21.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011579299999999999 2023-11-05T08:33:21.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:21.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.275278+0000", 2023-11-05T08:33:21.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0056325469999999999 2023-11-05T08:33:21.223 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.224 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:21.224 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.224 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpReadReply(3.5s0 52/40 ECSubReadReply(tid=27, attrs_read=0))", 2023-11-05T08:33:21.225 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:05.271241+0000", 2023-11-05T08:33:21.225 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.758219682, 2023-11-05T08:33:21.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040795400000000004, 2023-11-05T08:33:21.225 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:21.225 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:21.226 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:21.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:21.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.271241+0000", 2023-11-05T08:33:21.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:21.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.271241+0000", 2023-11-05T08:33:21.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:21.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.271242+0000", 2023-11-05T08:33:21.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1370000000000001e-06 2023-11-05T08:33:21.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:21.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.272078+0000", 2023-11-05T08:33:21.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00083553299999999996 2023-11-05T08:33:21.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:21.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.272107+0000", 2023-11-05T08:33:21.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9195000000000001e-05 2023-11-05T08:33:21.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:21.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.272124+0000", 2023-11-05T08:33:21.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6959000000000001e-05 2023-11-05T08:33:21.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:21.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.272391+0000", 2023-11-05T08:33:21.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026688200000000002 2023-11-05T08:33:21.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:21.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.275320+0000", 2023-11-05T08:33:21.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029298340000000001 2023-11-05T08:33:21.234 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.234 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:21.234 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.235 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.16s1 56/45 ECSubWrite(tid=63, reqid=client.4691.0:369, at_version=56'20, trim_to=0'0, roll_forward_to=51'19))", 2023-11-05T08:33:21.235 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:09.292024+0000", 2023-11-05T08:33:21.235 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.737436321000001, 2023-11-05T08:33:21.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0035291490000000001, 2023-11-05T08:33:21.235 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:21.235 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:21.236 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:21.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:21.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.292024+0000", 2023-11-05T08:33:21.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:21.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.292024+0000", 2023-11-05T08:33:21.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:21.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.292025+0000", 2023-11-05T08:33:21.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.083e-06 2023-11-05T08:33:21.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:21.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.292178+0000", 2023-11-05T08:33:21.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015233600000000001 2023-11-05T08:33:21.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:21.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.292213+0000", 2023-11-05T08:33:21.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4779e-05 2023-11-05T08:33:21.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:21.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.292239+0000", 2023-11-05T08:33:21.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6758999999999999e-05 2023-11-05T08:33:21.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:21.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.292487+0000", 2023-11-05T08:33:21.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024808600000000001 2023-11-05T08:33:21.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:21.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.292519+0000", 2023-11-05T08:33:21.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2125999999999997e-05 2023-11-05T08:33:21.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:21.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.295403+0000", 2023-11-05T08:33:21.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028833539999999999 2023-11-05T08:33:21.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:21.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.295553+0000", 2023-11-05T08:33:21.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000150626 2023-11-05T08:33:21.245 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.245 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:21.245 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.246 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.16s1 56/45 ECSubWrite(tid=64, reqid=client.4691.0:370, at_version=56'21, trim_to=0'0, roll_forward_to=51'19))", 2023-11-05T08:33:21.246 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:09.297268+0000", 2023-11-05T08:33:21.246 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.732192285, 2023-11-05T08:33:21.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053142609999999998, 2023-11-05T08:33:21.247 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:21.247 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:21.247 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:21.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:21.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297268+0000", 2023-11-05T08:33:21.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:21.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297268+0000", 2023-11-05T08:33:21.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:21.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297270+0000", 2023-11-05T08:33:21.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7570000000000001e-06 2023-11-05T08:33:21.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:21.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297426+0000", 2023-11-05T08:33:21.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015544199999999999 2023-11-05T08:33:21.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:21.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297457+0000", 2023-11-05T08:33:21.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1615999999999997e-05 2023-11-05T08:33:21.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:21.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297480+0000", 2023-11-05T08:33:21.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3326e-05 2023-11-05T08:33:21.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:21.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.299414+0000", 2023-11-05T08:33:21.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001933518 2023-11-05T08:33:21.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:21.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.299445+0000", 2023-11-05T08:33:21.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0701999999999998e-05 2023-11-05T08:33:21.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:21.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.302505+0000", 2023-11-05T08:33:21.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030607590000000001 2023-11-05T08:33:21.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:21.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.302583+0000", 2023-11-05T08:33:21.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7140999999999999e-05 2023-11-05T08:33:21.257 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.257 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:21.257 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.257 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.16s1 56/45 ECSubWrite(tid=65, reqid=client.4691.0:371, at_version=56'22, trim_to=0'0, roll_forward_to=56'20))", 2023-11-05T08:33:21.258 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:09.304840+0000", 2023-11-05T08:33:21.258 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.724620609, 2023-11-05T08:33:21.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032268710000000001, 2023-11-05T08:33:21.258 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:21.258 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:21.259 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:21.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:21.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.304840+0000", 2023-11-05T08:33:21.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:21.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.304840+0000", 2023-11-05T08:33:21.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:21.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:21.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.304842+0000", 2023-11-05T08:33:21.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5939999999999999e-06 2023-11-05T08:33:21.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:21.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.304981+0000", 2023-11-05T08:33:21.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013962600000000001 2023-11-05T08:33:21.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:21.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.305009+0000", 2023-11-05T08:33:21.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7426000000000001e-05 2023-11-05T08:33:21.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:21.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.305024+0000", 2023-11-05T08:33:21.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5449000000000001e-05 2023-11-05T08:33:21.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:21.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.305212+0000", 2023-11-05T08:33:21.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000188252 2023-11-05T08:33:21.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:21.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.305246+0000", 2023-11-05T08:33:21.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3321999999999997e-05 2023-11-05T08:33:21.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:21.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.307927+0000", 2023-11-05T08:33:21.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026809889999999999 2023-11-05T08:33:21.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:21.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:21.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:21.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.308067+0000", 2023-11-05T08:33:21.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014021300000000001 2023-11-05T08:33:21.268 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.269 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:21.269 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.269 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:21.269 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:21.269 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:33:21.270 INFO:teuthology.orchestra.run.smithi002.stdout:osd.12: {} 2023-11-05T08:33:21.271 INFO:teuthology.orchestra.run.smithi002.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-11-05T08:33:21.272 INFO:teuthology.orchestra.run.smithi002.stdout:osd.13: {} 2023-11-05T08:33:21.272 INFO:teuthology.orchestra.run.smithi002.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-11-05T08:33:21.272 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:21.263+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:21.273 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-11-05T08:33:21.273 INFO:teuthology.orchestra.run.smithi002.stdout:osd.14: {} 2023-11-05T08:33:21.274 INFO:teuthology.orchestra.run.smithi002.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-11-05T08:33:21.274 INFO:teuthology.orchestra.run.smithi002.stdout:osd.15: {} 2023-11-05T08:33:21.275 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-05T08:33:21.276 INFO:teuthology.orchestra.run.smithi002.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-11-05T08:33:21.276 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-05T08:33:21.372 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:33:21.370+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:33:21.376 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [3] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 7, 11, 15] 2023-11-05T08:33:21.377 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-11-05T08:33:21.377 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2023-11-05T08:33:21.378 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-05T08:33:21.399 INFO:tasks.rados.rados.0.smithi090.stdout:211: done (1 left) 2023-11-05T08:33:21.400 INFO:tasks.rados.rados.0.smithi090.stdout:212: rollback oid 27 current snap is 17 2023-11-05T08:33:21.400 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 27 to 4 2023-11-05T08:33:21.400 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 49 current snap is 17 2023-11-05T08:33:21.400 INFO:tasks.rados.rados.0.smithi090.stdout:213: seq_num 95 ranges {819200=442368,1261568=540672,1802240=622592} 2023-11-05T08:33:21.421 INFO:tasks.rados.rados.0.smithi090.stdout:213: writing smithi09039252-49 from 819200 to 1261568 tid 1 2023-11-05T08:33:21.431 INFO:tasks.rados.rados.0.smithi090.stdout:213: writing smithi09039252-49 from 1261568 to 1802240 tid 2 2023-11-05T08:33:21.443 INFO:tasks.rados.rados.0.smithi090.stdout:213: writing smithi09039252-49 from 1802240 to 2424832 tid 3 2023-11-05T08:33:21.444 INFO:tasks.rados.rados.0.smithi090.stdout:214: delete oid 5 current snap is 17 2023-11-05T08:33:21.444 INFO:tasks.rados.rados.0.smithi090.stdout:212: finishing rollback tid 0 to smithi09039252-27 2023-11-05T08:33:21.444 INFO:tasks.rados.rados.0.smithi090.stdout:212: finishing rollback tid 1 to smithi09039252-27 2023-11-05T08:33:21.444 INFO:tasks.rados.rados.0.smithi090.stdout:212: finishing rollback tid 2 to smithi09039252-27 2023-11-05T08:33:21.444 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 27 v 75 (ObjNum 26 snap 0 seq_num 26) dirty exists 2023-11-05T08:33:21.445 INFO:tasks.rados.rados.0.smithi090.stdout:213: finishing write tid 1 to smithi09039252-49 2023-11-05T08:33:21.445 INFO:tasks.rados.rados.0.smithi090.stdout:213: finishing write tid 2 to smithi09039252-49 2023-11-05T08:33:21.448 INFO:teuthology.orchestra.run.smithi002.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-05T08:33:21.450 INFO:tasks.rados.rados.0.smithi090.stdout:212: done (3 left) 2023-11-05T08:33:21.450 INFO:tasks.rados.rados.0.smithi090.stdout:214: done (2 left) 2023-11-05T08:33:21.450 INFO:tasks.rados.rados.0.smithi090.stdout:215: delete oid 39 current snap is 17 2023-11-05T08:33:21.451 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-05T08:33:21.451 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-11-05T08:33:21.455 INFO:tasks.rados.rados.0.smithi090.stdout:213: finishing write tid 3 to smithi09039252-49 2023-11-05T08:33:21.456 INFO:tasks.rados.rados.0.smithi090.stdout:213: finishing write tid 4 to smithi09039252-49 2023-11-05T08:33:21.457 INFO:tasks.rados.rados.0.smithi090.stdout:215: done (2 left) 2023-11-05T08:33:21.458 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 5 current snap is 17 2023-11-05T08:33:21.458 INFO:tasks.rados.rados.0.smithi090.stdout:216: seq_num 98 ranges {0=491520,491520=589824,1081344=557056,1638400=16384} 2023-11-05T08:33:21.466 INFO:tasks.rados.rados.0.smithi090.stdout:216: writing smithi09039252-5 from 0 to 491520 tid 1 2023-11-05T08:33:21.472 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:33:21.470+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:33:21.476 INFO:tasks.rados.rados.0.smithi090.stdout:216: writing smithi09039252-5 from 491520 to 1081344 tid 2 2023-11-05T08:33:21.485 INFO:tasks.rados.rados.0.smithi090.stdout:216: writing smithi09039252-5 from 1081344 to 1638400 tid 3 2023-11-05T08:33:21.486 INFO:tasks.rados.rados.0.smithi090.stdout:216: writing smithi09039252-5 from 1638400 to 1654784 tid 4 2023-11-05T08:33:21.487 INFO:tasks.rados.rados.0.smithi090.stdout:213: finishing write tid 5 to smithi09039252-49 2023-11-05T08:33:21.487 INFO:tasks.rados.rados.0.smithi090.stdout:213: oid 49 updating version 0 to 33 2023-11-05T08:33:21.487 INFO:tasks.rados.rados.0.smithi090.stdout:213: oid 49 version 33 is already newer than 30 2023-11-05T08:33:21.487 INFO:tasks.rados.rados.0.smithi090.stdout:213: oid 49 version 33 is already newer than 32 2023-11-05T08:33:21.487 INFO:tasks.rados.rados.0.smithi090.stdout:213: oid 49 version 33 is already newer than 31 2023-11-05T08:33:21.488 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 49 v 33 (ObjNum 95 snap 17 seq_num 95) dirty exists 2023-11-05T08:33:21.488 INFO:tasks.rados.rados.0.smithi090.stdout:213: left oid 49 (ObjNum 95 snap 17 seq_num 95) 2023-11-05T08:33:21.488 INFO:tasks.rados.rados.0.smithi090.stdout:216: finishing write tid 1 to smithi09039252-5 2023-11-05T08:33:21.488 INFO:tasks.rados.rados.0.smithi090.stdout:216: finishing write tid 2 to smithi09039252-5 2023-11-05T08:33:21.488 INFO:tasks.rados.rados.0.smithi090.stdout:217: read oid 40 snap 4 2023-11-05T08:33:21.489 INFO:tasks.rados.rados.0.smithi090.stdout:217: expect (ObjNum 59 snap 1 seq_num 59) 2023-11-05T08:33:21.489 INFO:tasks.rados.rados.0.smithi090.stdout:213: done (3 left) 2023-11-05T08:33:21.489 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 39 current snap is 17 2023-11-05T08:33:21.489 INFO:tasks.rados.rados.0.smithi090.stdout:218: seq_num 99 ranges {0=507904,507904=606208,1114112=557056} 2023-11-05T08:33:21.493 INFO:tasks.rados.rados.0.smithi090.stdout:218: writing smithi09039252-39 from 0 to 507904 tid 1 2023-11-05T08:33:21.503 INFO:tasks.rados.rados.0.smithi090.stdout:218: writing smithi09039252-39 from 507904 to 1114112 tid 2 2023-11-05T08:33:21.511 INFO:tasks.rados.rados.0.smithi090.stdout:218: writing smithi09039252-39 from 1114112 to 1671168 tid 3 2023-11-05T08:33:21.511 INFO:tasks.rados.rados.0.smithi090.stdout:219: read oid 43 snap -1 2023-11-05T08:33:21.511 INFO:tasks.rados.rados.0.smithi090.stdout:219: expect (ObjNum 82 snap 15 seq_num 82) 2023-11-05T08:33:21.511 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 49 current snap is 17 2023-11-05T08:33:21.512 INFO:tasks.rados.rados.0.smithi090.stdout:220: seq_num 100 ranges {2424832=524288,2949120=753664,3702784=409600} 2023-11-05T08:33:21.541 INFO:tasks.rados.rados.0.smithi090.stdout:220: writing smithi09039252-49 from 2424832 to 2949120 tid 1 2023-11-05T08:33:21.547 INFO:tasks.rados.rados.0.smithi090.stdout:220: writing smithi09039252-49 from 2949120 to 3702784 tid 2 2023-11-05T08:33:21.550 INFO:tasks.rados.rados.0.smithi090.stdout:220: writing smithi09039252-49 from 3702784 to 4112384 tid 3 2023-11-05T08:33:21.550 INFO:tasks.rados.rados.0.smithi090.stdout:221: delete oid 34 current snap is 17 2023-11-05T08:33:21.550 INFO:tasks.rados.rados.0.smithi090.stdout:216: finishing write tid 3 to smithi09039252-5 2023-11-05T08:33:21.550 INFO:tasks.rados.rados.0.smithi090.stdout:217: expect (ObjNum 59 snap 1 seq_num 59) 2023-11-05T08:33:21.565 INFO:teuthology.orchestra.run.smithi002.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-05T08:33:21.567 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-05T08:33:21.567 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-11-05T08:33:21.574 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:33:21.568+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:33:21.611 INFO:tasks.rados.rados.0.smithi090.stdout:216: finishing write tid 4 to smithi09039252-5 2023-11-05T08:33:21.612 INFO:tasks.rados.rados.0.smithi090.stdout:216: finishing write tid 5 to smithi09039252-5 2023-11-05T08:33:21.612 INFO:tasks.rados.rados.0.smithi090.stdout:217: done (6 left) 2023-11-05T08:33:21.612 INFO:tasks.rados.rados.0.smithi090.stdout:221: done (5 left) 2023-11-05T08:33:21.612 INFO:tasks.rados.rados.0.smithi090.stdout:222: rollback oid 23 current snap is 17 2023-11-05T08:33:21.613 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 23 to 4 2023-11-05T08:33:21.613 INFO:tasks.rados.rados.0.smithi090.stdout:216: finishing write tid 6 to smithi09039252-5 2023-11-05T08:33:21.613 INFO:tasks.rados.rados.0.smithi090.stdout:216: oid 5 updating version 0 to 24 2023-11-05T08:33:21.613 INFO:tasks.rados.rados.0.smithi090.stdout:216: oid 5 updating version 24 to 27 2023-11-05T08:33:21.613 INFO:tasks.rados.rados.0.smithi090.stdout:216: oid 5 updating version 27 to 28 2023-11-05T08:33:21.614 INFO:tasks.rados.rados.0.smithi090.stdout:216: oid 5 version 28 is already newer than 25 2023-11-05T08:33:21.614 INFO:tasks.rados.rados.0.smithi090.stdout:216: oid 5 version 28 is already newer than 26 2023-11-05T08:33:21.614 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 5 v 28 (ObjNum 98 snap 17 seq_num 98) dirty exists 2023-11-05T08:33:21.614 INFO:tasks.rados.rados.0.smithi090.stdout:216: left oid 5 (ObjNum 98 snap 17 seq_num 98) 2023-11-05T08:33:21.614 INFO:tasks.rados.rados.0.smithi090.stdout:218: finishing write tid 1 to smithi09039252-39 2023-11-05T08:33:21.615 INFO:tasks.rados.rados.0.smithi090.stdout:218: finishing write tid 2 to smithi09039252-39 2023-11-05T08:33:21.615 INFO:tasks.rados.rados.0.smithi090.stdout:218: finishing write tid 3 to smithi09039252-39 2023-11-05T08:33:21.615 INFO:tasks.rados.rados.0.smithi090.stdout:218: finishing write tid 4 to smithi09039252-39 2023-11-05T08:33:21.615 INFO:tasks.rados.rados.0.smithi090.stdout:218: finishing write tid 5 to smithi09039252-39 2023-11-05T08:33:21.616 INFO:tasks.rados.rados.0.smithi090.stdout:218: oid 39 updating version 0 to 40 2023-11-05T08:33:21.616 INFO:tasks.rados.rados.0.smithi090.stdout:218: oid 39 updating version 40 to 41 2023-11-05T08:33:21.616 INFO:tasks.rados.rados.0.smithi090.stdout:218: oid 39 version 41 is already newer than 39 2023-11-05T08:33:21.616 INFO:tasks.rados.rados.0.smithi090.stdout:218: oid 39 version 41 is already newer than 38 2023-11-05T08:33:21.616 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 39 v 41 (ObjNum 99 snap 17 seq_num 99) dirty exists 2023-11-05T08:33:21.616 INFO:tasks.rados.rados.0.smithi090.stdout:218: left oid 39 (ObjNum 99 snap 17 seq_num 99) 2023-11-05T08:33:21.617 INFO:tasks.rados.rados.0.smithi090.stdout:219: expect (ObjNum 82 snap 15 seq_num 82) 2023-11-05T08:33:21.670 INFO:teuthology.orchestra.run.smithi002.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-05T08:33:21.672 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-05T08:33:21.673 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-11-05T08:33:21.674 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:33:21.667+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:33:21.702 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:33:21.703 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-11-05T08:33:21.721 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":79,"stamp":"2023-11-05T08:33:21.393685+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":57115168,"num_objects":92,"num_object_clones":38,"num_object_copies":634,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":57,"num_objects_misplaced":17,"num_objects_unfound":0,"num_objects_dirty":92,"num_whiteouts":5,"num_read":311,"num_read_kb":34254,"num_write":460,"num_write_kb":44676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":673,"ondisk_log_size":673,"up":197,"acting":197,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":197,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":532716,"kb_used_data":100368,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509416724,"statfs":{"total":1546188226560,"available":1545642725376,"internally_reserved":0,"allocated":102776832,"data_stored":129807701,"data_compressed":1182761,"data_compressed_allocated":2629632,"data_compressed_original":6152192,"omap_allocated":25505,"internal_metadata":442473567},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9199616,"num_objects":13,"num_object_clones":11,"num_object_copies":91,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":10,"num_read_kb":4669,"num_write":1,"num_write_kb":4664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-5,"num_bytes_recovered":-1654784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.044536"},"pg_stats":[{"pgid":"3.19","version":"59'29","reported_seq":151,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-05T08:33:17.380491+0000","last_change":"2023-11-05T08:33:17.381937+0000","last_active":"2023-11-05T08:33:17.380491+0000","last_peered":"2023-11-05T08:33:17.380491+0000","last_clean":"2023-11-05T08:33:17.380491+0000","last_became_active":"2023-11-05T08:32:59.211522+0000","last_became_peered":"2023-11-05T08:32:59.211522+0000","last_unstale":"2023-11-05T08:33:17.380491+0000","last_undegraded":"2023-11-05T08:33:17.380491+0000","last_fullsized":"2023-11-05T08:33:17.380491+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":5,"last_scrub":"39'22","last_scrub_stamp":"2023-11-05T08:33:01.870959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:01.870959+0000","objects_scrubbed":3,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:03.063246+0000","scrub_duration":0.0087276880000000008,"objects_trimmed":0,"snaptrim_duration":0.001980468,"stat_sum":{"num_bytes":3440640,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,11,9,13,7,1],"acting":[8,4,11,9,13,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"2"},{"start":"8","length":"1"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1b","length":"2"}]},{"pgid":"3.18","version":"65'36","reported_seq":157,"reported_epoch":65,"state":"active+clean+remapped","last_fresh":"2023-11-05T08:33:18.387266+0000","last_change":"2023-11-05T08:33:18.391087+0000","last_active":"2023-11-05T08:33:18.387266+0000","last_peered":"2023-11-05T08:33:18.387266+0000","last_clean":"2023-11-05T08:33:18.387266+0000","last_became_active":"2023-11-05T08:32:59.217241+0000","last_became_peered":"2023-11-05T08:32:59.217241+0000","last_unstale":"2023-11-05T08:33:18.387266+0000","last_undegraded":"2023-11-05T08:33:18.387266+0000","last_fullsized":"2023-11-05T08:33:18.387266+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":5,"last_scrub":"52'28","last_scrub_stamp":"2023-11-05T08:33:13.971775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:13.971775+0000","objects_scrubbed":5,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:24.355124+0000","scrub_duration":0.019915236999999999,"objects_trimmed":0,"snaptrim_duration":0.026542824,"stat_sum":{"num_bytes":3211264,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,2147483647,4,5,7,12],"acting":[13,6,2,4,5,7,12],"avail_no_missing":["13(0)","2(2)","4(3)","5(4)","6(1)","7(5)","12(6)"],"object_location_counts":[{"shards":"2(2),4(3),5(4),6(1),7(5),12(6),13(0)","objects":6}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"2"},{"start":"8","length":"1"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1b","length":"2"}]},{"pgid":"3.17","version":"49'17","reported_seq":161,"reported_epoch":64,"state":"active+clean+remapped","last_fresh":"2023-11-05T08:33:17.376843+0000","last_change":"2023-11-05T08:33:17.383018+0000","last_active":"2023-11-05T08:33:17.376843+0000","last_peered":"2023-11-05T08:33:17.376843+0000","last_clean":"2023-11-05T08:33:17.376843+0000","last_became_active":"2023-11-05T08:33:02.240933+0000","last_became_peered":"2023-11-05T08:33:02.240933+0000","last_unstale":"2023-11-05T08:33:17.376843+0000","last_undegraded":"2023-11-05T08:33:17.376843+0000","last_fullsized":"2023-11-05T08:33:17.376843+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":5,"last_scrub":"49'17","last_scrub_stamp":"2023-11-05T08:33:13.975356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:13.975356+0000","objects_scrubbed":3,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:33.888115+0000","scrub_duration":0.014066680999999999,"objects_trimmed":0,"snaptrim_duration":0.0089184820000000001,"stat_sum":{"num_bytes":638976,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2147483647,10,9,7,1,13],"acting":[5,6,10,9,7,1,13],"avail_no_missing":["5(0)","1(5)","6(1)","7(4)","9(3)","10(2)","13(6)"],"object_location_counts":[{"shards":"1(5),5(0),6(1),7(4),9(3),10(2),13(6)","objects":3}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"2"},{"start":"8","length":"1"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1b","length":"2"}]},{"pgid":"3.16","version":"64'30","reported_seq":134,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-05T08:33:17.387425+0000","last_change":"2023-11-05T08:33:17.391439+0000","last_active":"2023-11-05T08:33:17.387425+0000","last_peered":"2023-11-05T08:33:17.387425+0000","last_clean":"2023-11-05T08:33:17.387425+0000","last_became_active":"2023-11-05T08:32:59.210858+0000","last_became_peered":"2023-11-05T08:32:59.210858+0000","last_unstale":"2023-11-05T08:33:17.387425+0000","last_undegraded":"2023-11-05T08:33:17.387425+0000","last_fullsized":"2023-11-05T08:33:17.387425+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":5,"last_scrub":"46'13","last_scrub_stamp":"2023-11-05T08:33:00.659926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:00.659926+0000","objects_scrubbed":1,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:26.111908+0000","scrub_duration":0.0077151709999999998,"objects_trimmed":0,"snaptrim_duration":0.018048692000000002,"stat_sum":{"num_bytes":4440064,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":9,"num_write_kb":2704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15,4,8,9,5,10],"acting":[11,15,4,8,9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"2"},{"start":"8","length":"1"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1b","length":"2"}]},{"pgid":"3.15","version":"36'15","reported_seq":32,"reported_epoch":63,"state":"down+remapped","last_fresh":"2023-11-05T08:33:16.340622+0000","last_change":"2023-11-05T08:32:58.174545+0000","last_active":"2023-11-05T08:32:57.142227+0000","last_peered":"2023-11-05T08:32:49.086936+0000","last_clean":"2023-11-05T08:32:49.086936+0000","last_became_active":"2023-11-05T08:32:44.551939+0000","last_became_peered":"2023-11-05T08:32:44.551939+0000","last_unstale":"2023-11-05T08:33:16.340622+0000","last_undegraded":"2023-11-05T08:33:16.340622+0000","last_fullsized":"2023-11-05T08:33:16.340622+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1630208,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":795,"num_write":6,"num_write_kb":1592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,2147483647,13,14,2,15],"acting":[9,6,2147483647,13,2147483647,2,2147483647],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.14","version":"56'19","reported_seq":212,"reported_epoch":64,"state":"active+clean+remapped","last_fresh":"2023-11-05T08:33:17.381429+0000","last_change":"2023-11-05T08:33:17.383812+0000","last_active":"2023-11-05T08:33:17.381429+0000","last_peered":"2023-11-05T08:33:17.381429+0000","last_clean":"2023-11-05T08:33:17.381429+0000","last_became_active":"2023-11-05T08:33:02.227671+0000","last_became_peered":"2023-11-05T08:33:02.227671+0000","last_unstale":"2023-11-05T08:33:17.381429+0000","last_undegraded":"2023-11-05T08:33:17.381429+0000","last_fullsized":"2023-11-05T08:33:17.381429+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":5,"last_scrub":"49'18","last_scrub_stamp":"2023-11-05T08:33:04.812402+0000","last_deep_scrub":"49'18","last_deep_scrub_stamp":"2023-11-05T08:33:04.812402+0000","last_clean_scrub_stamp":"2023-11-05T08:33:04.812402+0000","objects_scrubbed":2,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:24.862241+0000","scrub_duration":0.0086643829999999995,"objects_trimmed":0,"snaptrim_duration":0.003589961,"stat_sum":{"num_bytes":1966080,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,9,15,1,13,8,4],"acting":[8,9,15,1,13,8,4],"avail_no_missing":["8(0)","1(3)","4(6)","8(5)","9(1)","13(4)","15(2)"],"object_location_counts":[{"shards":"1(3),4(6),8(0),8(5),9(1),13(4),15(2)","objects":2}],"blocked_by":[],"up_primary":9,"acting_primary":8,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"2"},{"start":"8","length":"1"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1b","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":47,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-05T08:33:16.379545+0000","last_change":"2023-11-05T08:32:31.942607+0000","last_active":"2023-11-05T08:33:16.379545+0000","last_peered":"2023-11-05T08:33:16.379545+0000","last_clean":"2023-11-05T08:33:16.379545+0000","last_became_active":"2023-11-05T08:32:31.941858+0000","last_became_peered":"2023-11-05T08:32:31.941858+0000","last_unstale":"2023-11-05T08:33:16.379545+0000","last_undegraded":"2023-11-05T08:33:16.379545+0000","last_fullsized":"2023-11-05T08:33:16.379545+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:48.710632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"35'9","reported_seq":102,"reported_epoch":65,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:33:18.363318+0000","last_change":"2023-11-05T08:32:54.153555+0000","last_active":"2023-11-05T08:33:18.363318+0000","last_peered":"2023-11-05T08:33:18.363318+0000","last_clean":"2023-11-05T08:32:48.102597+0000","last_became_active":"2023-11-05T08:32:54.153555+0000","last_became_peered":"2023-11-05T08:32:54.153555+0000","last_unstale":"2023-11-05T08:33:18.363318+0000","last_undegraded":"2023-11-05T08:32:54.121924+0000","last_fullsized":"2023-11-05T08:32:54.120000+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":14,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1089536,"num_objects":1,"num_object_clones":1,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":1064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,13,1,15,12,7,9],"acting":[2147483647,13,1,15,12,7,9],"avail_no_missing":["13(1)","1(2)","7(5)","9(6)","12(4)","15(3)"],"object_location_counts":[{"shards":"1(2),7(5),9(6),12(4),13(1),15(3)","objects":1}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":47,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-05T08:33:16.342806+0000","last_change":"2023-11-05T08:32:31.941613+0000","last_active":"2023-11-05T08:33:16.342806+0000","last_peered":"2023-11-05T08:33:16.342806+0000","last_clean":"2023-11-05T08:33:16.342806+0000","last_became_active":"2023-11-05T08:32:31.941207+0000","last_became_peered":"2023-11-05T08:32:31.941207+0000","last_unstale":"2023-11-05T08:33:16.342806+0000","last_undegraded":"2023-11-05T08:33:16.342806+0000","last_fullsized":"2023-11-05T08:33:16.342806+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:44.932076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"39'12","reported_seq":109,"reported_epoch":64,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:33:17.368213+0000","last_change":"2023-11-05T08:32:54.151639+0000","last_active":"2023-11-05T08:33:17.368213+0000","last_peered":"2023-11-05T08:33:17.368213+0000","last_clean":"2023-11-05T08:32:49.074431+0000","last_became_active":"2023-11-05T08:32:54.151639+0000","last_became_peered":"2023-11-05T08:32:54.151639+0000","last_unstale":"2023-11-05T08:33:17.368213+0000","last_undegraded":"2023-11-05T08:32:54.120694+0000","last_fullsized":"2023-11-05T08:32:54.119652+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":12,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030081180000000002,"stat_sum":{"num_bytes":319488,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,2147483647,8,5,11],"acting":[4,6,9,2147483647,8,5,11],"avail_no_missing":["4(0)","5(5)","6(1)","8(4)","9(2)","11(6)"],"object_location_counts":[{"shards":"4(0),5(5),6(1),8(4),9(2),11(6)","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":47,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-05T08:33:15.376826+0000","last_change":"2023-11-05T08:32:31.940904+0000","last_active":"2023-11-05T08:33:15.376826+0000","last_peered":"2023-11-05T08:33:15.376826+0000","last_clean":"2023-11-05T08:33:15.376826+0000","last_became_active":"2023-11-05T08:32:31.940312+0000","last_became_peered":"2023-11-05T08:32:31.940312+0000","last_unstale":"2023-11-05T08:33:15.376826+0000","last_undegraded":"2023-11-05T08:33:15.376826+0000","last_fullsized":"2023-11-05T08:33:15.376826+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:43.223063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"39'12","reported_seq":108,"reported_epoch":64,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:33:17.381149+0000","last_change":"2023-11-05T08:32:54.141915+0000","last_active":"2023-11-05T08:33:17.381149+0000","last_peered":"2023-11-05T08:33:17.381149+0000","last_clean":"2023-11-05T08:32:49.088117+0000","last_became_active":"2023-11-05T08:32:54.141915+0000","last_became_peered":"2023-11-05T08:32:54.141915+0000","last_unstale":"2023-11-05T08:33:17.381149+0000","last_undegraded":"2023-11-05T08:32:54.115420+0000","last_fullsized":"2023-11-05T08:32:54.114438+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":12,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.011061272,"stat_sum":{"num_bytes":1376256,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,4,1,2147483647,14,13],"acting":[8,2,4,1,2147483647,14,13],"avail_no_missing":["8(0)","1(3)","2(1)","4(2)","13(6)","14(5)"],"object_location_counts":[{"shards":"1(3),2(1),4(2),8(0),13(6),14(5)","objects":1}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":52,"reported_epoch":63,"state":"active+undersized","last_fresh":"2023-11-05T08:33:16.350123+0000","last_change":"2023-11-05T08:32:51.087583+0000","last_active":"2023-11-05T08:33:16.350123+0000","last_peered":"2023-11-05T08:33:16.350123+0000","last_clean":"2023-11-05T08:32:49.071396+0000","last_became_active":"2023-11-05T08:32:51.087583+0000","last_became_peered":"2023-11-05T08:32:51.087583+0000","last_unstale":"2023-11-05T08:33:16.350123+0000","last_undegraded":"2023-11-05T08:33:16.350123+0000","last_fullsized":"2023-11-05T08:32:51.085658+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:52.667515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"36'15","reported_seq":118,"reported_epoch":64,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:33:17.390852+0000","last_change":"2023-11-05T08:32:54.139002+0000","last_active":"2023-11-05T08:33:17.390852+0000","last_peered":"2023-11-05T08:33:17.390852+0000","last_clean":"2023-11-05T08:32:49.088591+0000","last_became_active":"2023-11-05T08:32:54.139002+0000","last_became_peered":"2023-11-05T08:32:54.139002+0000","last_unstale":"2023-11-05T08:33:17.390852+0000","last_undegraded":"2023-11-05T08:32:54.118261+0000","last_fullsized":"2023-11-05T08:32:54.117408+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":12,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.020413489999999999,"stat_sum":{"num_bytes":1630208,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":795,"num_write":6,"num_write_kb":1592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,4,13,7,2147483647,6],"acting":[15,8,4,13,7,2147483647,6],"avail_no_missing":["15(0)","4(2)","6(6)","7(4)","8(1)","13(3)"],"object_location_counts":[{"shards":"4(2),6(6),7(4),8(1),13(3),15(0)","objects":2}],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.3","version":"61'14","reported_seq":130,"reported_epoch":65,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:33:18.361697+0000","last_change":"2023-11-05T08:32:59.214416+0000","last_active":"2023-11-05T08:33:18.361697+0000","last_peered":"2023-11-05T08:33:18.361697+0000","last_clean":"2023-11-05T08:32:57.168145+0000","last_became_active":"2023-11-05T08:32:59.214416+0000","last_became_peered":"2023-11-05T08:32:59.214416+0000","last_unstale":"2023-11-05T08:33:18.361697+0000","last_undegraded":"2023-11-05T08:32:59.185849+0000","last_fullsized":"2023-11-05T08:32:59.184877+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0045341740000000002,"stat_sum":{"num_bytes":786432,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":4,"num_read_kb":1,"num_write":5,"num_write_kb":768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,4,15,2,5,2147483647],"acting":[13,7,4,15,2,5,2147483647],"avail_no_missing":["13(0)","2(4)","4(2)","5(5)","7(1)","15(3)"],"object_location_counts":[{"shards":"2(4),4(2),5(5),7(1),13(0),15(3)","objects":3}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"2"},{"start":"6","length":"1"},{"start":"8","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":46,"reported_epoch":63,"state":"active+undersized","last_fresh":"2023-11-05T08:33:16.358488+0000","last_change":"2023-11-05T08:32:59.192112+0000","last_active":"2023-11-05T08:33:16.358488+0000","last_peered":"2023-11-05T08:33:16.358488+0000","last_clean":"2023-11-05T08:32:57.141279+0000","last_became_active":"2023-11-05T08:32:59.192112+0000","last_became_peered":"2023-11-05T08:32:59.192112+0000","last_unstale":"2023-11-05T08:33:16.358488+0000","last_undegraded":"2023-11-05T08:33:16.358488+0000","last_fullsized":"2023-11-05T08:32:59.189202+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:58.324718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12],"acting":[12],"avail_no_missing":["12"],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.0","version":"33'9","reported_seq":166,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-05T08:33:17.536965+0000","last_change":"2023-11-05T08:33:17.537759+0000","last_active":"2023-11-05T08:33:17.536965+0000","last_peered":"2023-11-05T08:33:17.536965+0000","last_clean":"2023-11-05T08:33:17.536965+0000","last_became_active":"2023-11-05T08:32:54.144790+0000","last_became_peered":"2023-11-05T08:32:54.144790+0000","last_unstale":"2023-11-05T08:33:17.536965+0000","last_undegraded":"2023-11-05T08:33:17.536965+0000","last_fullsized":"2023-11-05T08:33:17.536965+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'9","last_scrub_stamp":"2023-11-05T08:33:03.810524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:03.810524+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:16.369751+0000","scrub_duration":0.0066041449999999996,"objects_trimmed":0,"snaptrim_duration":0.00096708199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,15,1,5,6,4],"acting":[14,2,15,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"2"},{"start":"8","length":"1"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1b","length":"2"}]},{"pgid":"2.1","version":"0'0","reported_seq":49,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-05T08:33:17.359436+0000","last_change":"2023-11-05T08:32:31.941357+0000","last_active":"2023-11-05T08:33:17.359436+0000","last_peered":"2023-11-05T08:33:17.359436+0000","last_clean":"2023-11-05T08:33:17.359436+0000","last_became_active":"2023-11-05T08:32:31.940564+0000","last_became_peered":"2023-11-05T08:32:31.940564+0000","last_unstale":"2023-11-05T08:33:17.359436+0000","last_undegraded":"2023-11-05T08:33:17.359436+0000","last_fullsized":"2023-11-05T08:33:17.359436+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:58.168161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.1","version":"64'19","reported_seq":181,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-05T08:33:17.375916+0000","last_change":"2023-11-05T08:33:17.379110+0000","last_active":"2023-11-05T08:33:17.375916+0000","last_peered":"2023-11-05T08:33:17.375916+0000","last_clean":"2023-11-05T08:33:17.375916+0000","last_became_active":"2023-11-05T08:32:54.144895+0000","last_became_peered":"2023-11-05T08:32:54.144895+0000","last_unstale":"2023-11-05T08:33:17.375916+0000","last_undegraded":"2023-11-05T08:33:17.375916+0000","last_fullsized":"2023-11-05T08:33:17.375916+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'11","last_scrub_stamp":"2023-11-05T08:33:04.003557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:04.003557+0000","objects_scrubbed":3,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:10.794932+0000","scrub_duration":0.010761797,"objects_trimmed":0,"snaptrim_duration":0.011260671,"stat_sum":{"num_bytes":2293760,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":290,"num_write":3,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,5,10,2,13],"acting":[4,11,9,5,10,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"2"},{"start":"8","length":"1"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1b","length":"2"}]},{"pgid":"2.0","version":"0'0","reported_seq":45,"reported_epoch":62,"state":"active+undersized","last_fresh":"2023-11-05T08:33:15.377729+0000","last_change":"2023-11-05T08:32:59.200499+0000","last_active":"2023-11-05T08:33:15.377729+0000","last_peered":"2023-11-05T08:33:15.377729+0000","last_clean":"2023-11-05T08:32:57.169235+0000","last_became_active":"2023-11-05T08:32:59.200499+0000","last_became_peered":"2023-11-05T08:32:59.200499+0000","last_unstale":"2023-11-05T08:33:15.377729+0000","last_undegraded":"2023-11-05T08:33:15.377729+0000","last_fullsized":"2023-11-05T08:32:59.194018+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:35.798739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"65'21","reported_seq":136,"reported_epoch":65,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:33:18.576548+0000","last_change":"2023-11-05T08:32:59.207989+0000","last_active":"2023-11-05T08:33:18.576548+0000","last_peered":"2023-11-05T08:33:18.576548+0000","last_clean":"2023-11-05T08:32:57.163393+0000","last_became_active":"2023-11-05T08:32:59.207989+0000","last_became_peered":"2023-11-05T08:32:59.207989+0000","last_unstale":"2023-11-05T08:33:18.576548+0000","last_undegraded":"2023-11-05T08:32:59.181953+0000","last_fullsized":"2023-11-05T08:32:59.181602+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018755122999999999,"stat_sum":{"num_bytes":2015232,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":8,"num_write_kb":1968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,2147483647,6,15,1,11],"acting":[13,2,2147483647,6,15,1,11],"avail_no_missing":["13(0)","1(5)","2(1)","6(3)","11(6)","15(4)"],"object_location_counts":[{"shards":"1(5),2(1),6(3),11(6),13(0),15(4)","objects":2}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"2"},{"start":"6","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":48,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-05T08:33:16.356095+0000","last_change":"2023-11-05T08:32:31.940561+0000","last_active":"2023-11-05T08:33:16.356095+0000","last_peered":"2023-11-05T08:33:16.356095+0000","last_clean":"2023-11-05T08:33:16.356095+0000","last_became_active":"2023-11-05T08:32:31.939705+0000","last_became_peered":"2023-11-05T08:32:31.939705+0000","last_unstale":"2023-11-05T08:33:16.356095+0000","last_undegraded":"2023-11-05T08:33:16.356095+0000","last_fullsized":"2023-11-05T08:33:16.356095+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:54.088838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"25'157","reported_seq":248,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-05T08:33:17.361232+0000","last_change":"2023-11-05T08:32:28.761556+0000","last_active":"2023-11-05T08:33:17.361232+0000","last_peered":"2023-11-05T08:33:17.361232+0000","last_clean":"2023-11-05T08:33:17.361232+0000","last_became_active":"2023-11-05T08:32:28.758834+0000","last_became_peered":"2023-11-05T08:32:28.758834+0000","last_unstale":"2023-11-05T08:33:17.361232+0000","last_undegraded":"2023-11-05T08:33:17.361232+0000","last_fullsized":"2023-11-05T08:33:17.361232+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:27.693099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:27.693099+0000","last_clean_scrub_stamp":"2023-11-05T08:32:27.693099+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:40.584365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.8","version":"61'21","reported_seq":121,"reported_epoch":64,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:33:17.373722+0000","last_change":"2023-11-05T08:32:54.151946+0000","last_active":"2023-11-05T08:33:17.373722+0000","last_peered":"2023-11-05T08:33:17.373722+0000","last_clean":"2023-11-05T08:32:49.090022+0000","last_became_active":"2023-11-05T08:32:54.151946+0000","last_became_peered":"2023-11-05T08:32:54.151946+0000","last_unstale":"2023-11-05T08:33:17.373722+0000","last_undegraded":"2023-11-05T08:32:54.119646+0000","last_fullsized":"2023-11-05T08:32:54.118811+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":12,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.013731910999999999,"stat_sum":{"num_bytes":2523136,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":18,"num_write_kb":1904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2147483647,2,1,14,15,6],"acting":[5,2147483647,2,1,14,15,6],"avail_no_missing":["5(0)","1(3)","2(2)","6(6)","14(4)","15(5)"],"object_location_counts":[{"shards":"1(3),2(2),5(0),6(6),14(4),15(5)","objects":4}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.9","version":"52'26","reported_seq":148,"reported_epoch":64,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:33:17.536753+0000","last_change":"2023-11-05T08:32:59.206152+0000","last_active":"2023-11-05T08:33:17.536753+0000","last_peered":"2023-11-05T08:33:17.536753+0000","last_clean":"2023-11-05T08:32:49.072716+0000","last_became_active":"2023-11-05T08:32:59.206152+0000","last_became_peered":"2023-11-05T08:32:59.206152+0000","last_unstale":"2023-11-05T08:33:17.536753+0000","last_undegraded":"2023-11-05T08:32:59.188430+0000","last_fullsized":"2023-11-05T08:32:59.187299+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":12,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015323719999999999,"stat_sum":{"num_bytes":3407872,"num_objects":6,"num_object_clones":2,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":21,"num_read_kb":8311,"num_write":31,"num_write_kb":3328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,2147483647,2147483647,15,8,10],"acting":[14,1,2147483647,2147483647,15,8,10],"avail_no_missing":["14(0)","1(1)","8(5)","10(6)","15(4)"],"object_location_counts":[{"shards":"1(1),8(5),10(6),14(0),15(4)","objects":6}],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.a","version":"63'30","reported_seq":132,"reported_epoch":64,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:33:17.367931+0000","last_change":"2023-11-05T08:32:59.213422+0000","last_active":"2023-11-05T08:33:17.367931+0000","last_peered":"2023-11-05T08:33:17.367931+0000","last_clean":"2023-11-05T08:32:57.149380+0000","last_became_active":"2023-11-05T08:32:59.213422+0000","last_became_peered":"2023-11-05T08:32:59.213422+0000","last_unstale":"2023-11-05T08:33:17.367931+0000","last_undegraded":"2023-11-05T08:32:59.183619+0000","last_fullsized":"2023-11-05T08:32:59.182610+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:11.951715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033597206999999997,"stat_sum":{"num_bytes":3784704,"num_objects":6,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":15,"num_read_kb":7542,"num_write":34,"num_write_kb":3696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,7,1,9,14,6,11],"acting":[2147483647,7,1,9,14,6,11],"avail_no_missing":["7(1)","1(2)","6(5)","9(3)","11(6)","14(4)"],"object_location_counts":[{"shards":"1(2),6(5),7(1),9(3),11(6),14(4)","objects":6}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"2"},{"start":"6","length":"1"}]},{"pgid":"3.b","version":"63'18","reported_seq":118,"reported_epoch":64,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:33:17.368742+0000","last_change":"2023-11-05T08:32:51.095575+0000","last_active":"2023-11-05T08:33:17.368742+0000","last_peered":"2023-11-05T08:33:17.368742+0000","last_clean":"2023-11-05T08:32:49.072970+0000","last_became_active":"2023-11-05T08:32:51.095575+0000","last_became_peered":"2023-11-05T08:32:51.095575+0000","last_unstale":"2023-11-05T08:33:17.368742+0000","last_undegraded":"2023-11-05T08:32:51.085902+0000","last_fullsized":"2023-11-05T08:32:51.084936+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":12,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:45.701694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021279290000000002,"stat_sum":{"num_bytes":2637824,"num_objects":6,"num_object_clones":2,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":19,"num_read_kb":2647,"num_write":15,"num_write_kb":2896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2147483647,1,12,6,7,9],"acting":[4,2147483647,1,12,6,7,9],"avail_no_missing":["4(0)","1(2)","6(4)","7(5)","9(6)","12(3)"],"object_location_counts":[{"shards":"1(2),4(0),6(4),7(5),9(6),12(3)","objects":6}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.c","version":"33'2","reported_seq":97,"reported_epoch":64,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:33:17.373285+0000","last_change":"2023-11-05T08:32:51.099852+0000","last_active":"2023-11-05T08:33:17.373285+0000","last_peered":"2023-11-05T08:33:17.373285+0000","last_clean":"2023-11-05T08:32:49.087645+0000","last_became_active":"2023-11-05T08:32:51.099852+0000","last_became_peered":"2023-11-05T08:32:51.099852+0000","last_unstale":"2023-11-05T08:33:17.373285+0000","last_undegraded":"2023-11-05T08:32:51.089699+0000","last_fullsized":"2023-11-05T08:32:51.089041+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":12,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:08.002577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015630430000000001,"stat_sum":{"num_bytes":311296,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":914,"num_write":2,"num_write_kb":304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,2147483647,10,6,12,5],"acting":[11,8,2147483647,10,6,12,5],"avail_no_missing":["11(0)","5(6)","6(4)","8(1)","10(3)","12(5)"],"object_location_counts":[{"shards":"5(6),6(4),8(1),10(3),11(0),12(5)","objects":1}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.d","version":"52'16","reported_seq":118,"reported_epoch":64,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:33:17.368300+0000","last_change":"2023-11-05T08:32:59.204042+0000","last_active":"2023-11-05T08:33:17.368300+0000","last_peered":"2023-11-05T08:33:17.368300+0000","last_clean":"2023-11-05T08:32:49.088145+0000","last_became_active":"2023-11-05T08:32:59.204042+0000","last_became_peered":"2023-11-05T08:32:59.204042+0000","last_unstale":"2023-11-05T08:33:17.368300+0000","last_undegraded":"2023-11-05T08:32:59.184163+0000","last_fullsized":"2023-11-05T08:32:59.183315+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":12,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:49.779651+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.024583713,"stat_sum":{"num_bytes":1556480,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":13,"num_read_kb":965,"num_write":10,"num_write_kb":1520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2147483647,14,10,2147483647,8,4],"acting":[7,2147483647,14,10,2147483647,8,4],"avail_no_missing":["7(0)","4(6)","8(5)","10(3)","14(2)"],"object_location_counts":[{"shards":"4(6),7(0),8(5),10(3),14(2)","objects":4}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.e","version":"64'53","reported_seq":190,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-05T08:33:17.383460+0000","last_change":"2023-11-05T08:33:17.384386+0000","last_active":"2023-11-05T08:33:17.383460+0000","last_peered":"2023-11-05T08:33:17.383460+0000","last_clean":"2023-11-05T08:33:17.383460+0000","last_became_active":"2023-11-05T08:32:44.949185+0000","last_became_peered":"2023-11-05T08:32:44.949185+0000","last_unstale":"2023-11-05T08:33:17.383460+0000","last_undegraded":"2023-11-05T08:33:17.383460+0000","last_fullsized":"2023-11-05T08:33:17.383460+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:44.922652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011658854999999999,"stat_sum":{"num_bytes":6291456,"num_objects":10,"num_object_clones":4,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":21,"num_read_kb":1643,"num_write":29,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1,11,8,6,12],"acting":[2,9,1,11,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"2"},{"start":"8","length":"1"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1b","length":"2"}]},{"pgid":"3.f","version":"48'29","reported_seq":133,"reported_epoch":64,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:33:17.373742+0000","last_change":"2023-11-05T08:32:51.104425+0000","last_active":"2023-11-05T08:33:17.373742+0000","last_peered":"2023-11-05T08:33:17.373742+0000","last_clean":"2023-11-05T08:32:49.098585+0000","last_became_active":"2023-11-05T08:32:51.104425+0000","last_became_peered":"2023-11-05T08:32:51.104425+0000","last_unstale":"2023-11-05T08:33:17.373742+0000","last_undegraded":"2023-11-05T08:32:51.091474+0000","last_fullsized":"2023-11-05T08:32:51.089784+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":12,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:46.135425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012093372,"stat_sum":{"num_bytes":5767168,"num_objects":9,"num_object_clones":5,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":28,"num_read_kb":7196,"num_write":40,"num_write_kb":5632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,14,13,8,10,2147483647],"acting":[11,2,14,13,8,10,2147483647],"avail_no_missing":["11(0)","2(1)","8(4)","10(5)","13(3)","14(2)"],"object_location_counts":[{"shards":"2(1),8(4),10(5),11(0),13(3),14(2)","objects":9}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.10","version":"65'20","reported_seq":133,"reported_epoch":65,"state":"active+clean+remapped","last_fresh":"2023-11-05T08:33:18.392720+0000","last_change":"2023-11-05T08:33:18.396459+0000","last_active":"2023-11-05T08:33:18.392720+0000","last_peered":"2023-11-05T08:33:18.392720+0000","last_clean":"2023-11-05T08:33:18.392720+0000","last_became_active":"2023-11-05T08:32:59.210399+0000","last_became_peered":"2023-11-05T08:32:59.210399+0000","last_unstale":"2023-11-05T08:33:18.392720+0000","last_undegraded":"2023-11-05T08:33:18.392720+0000","last_fullsized":"2023-11-05T08:33:18.392720+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":5,"last_scrub":"33'9","last_scrub_stamp":"2023-11-05T08:33:00.792248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:00.792248+0000","objects_scrubbed":3,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:08.857875+0000","scrub_duration":0.0073349119999999999,"objects_trimmed":0,"snaptrim_duration":0.034016435999999997,"stat_sum":{"num_bytes":3063808,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":3028,"num_write":4,"num_write_kb":1424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,7,4,8,2147483647,11],"acting":[13,6,7,4,8,6,11],"avail_no_missing":["13(0)","4(3)","6(1)","6(5)","7(2)","8(4)","11(6)"],"object_location_counts":[{"shards":"4(3),6(1),6(5),7(2),8(4),11(6),13(0)","objects":4}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"2"},{"start":"8","length":"1"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1b","length":"2"}]},{"pgid":"3.11","version":"49'19","reported_seq":185,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-05T08:33:17.374811+0000","last_change":"2023-11-05T08:33:17.375757+0000","last_active":"2023-11-05T08:33:17.374811+0000","last_peered":"2023-11-05T08:33:17.374811+0000","last_clean":"2023-11-05T08:33:17.374811+0000","last_became_active":"2023-11-05T08:32:59.214193+0000","last_became_peered":"2023-11-05T08:32:59.214193+0000","last_unstale":"2023-11-05T08:33:17.374811+0000","last_undegraded":"2023-11-05T08:33:17.374811+0000","last_fullsized":"2023-11-05T08:33:17.374811+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":5,"last_scrub":"46'17","last_scrub_stamp":"2023-11-05T08:33:00.070510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:00.070510+0000","objects_scrubbed":3,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:19.560298+0000","scrub_duration":0.0081678800000000006,"objects_trimmed":0,"snaptrim_duration":0.0016583240000000001,"stat_sum":{"num_bytes":1998848,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,15,9,5,6,1],"acting":[10,11,15,9,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"2"},{"start":"8","length":"1"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1b","length":"2"}]},{"pgid":"3.12","version":"37'7","reported_seq":104,"reported_epoch":64,"state":"active+clean+remapped","last_fresh":"2023-11-05T08:33:17.357887+0000","last_change":"2023-11-05T08:33:17.361005+0000","last_active":"2023-11-05T08:33:17.357887+0000","last_peered":"2023-11-05T08:33:17.357887+0000","last_clean":"2023-11-05T08:33:17.357887+0000","last_became_active":"2023-11-05T08:33:00.206236+0000","last_became_peered":"2023-11-05T08:33:00.206236+0000","last_unstale":"2023-11-05T08:33:17.357887+0000","last_undegraded":"2023-11-05T08:33:17.357887+0000","last_fullsized":"2023-11-05T08:33:17.357887+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":5,"last_scrub":"37'7","last_scrub_stamp":"2023-11-05T08:33:02.650387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:02.650387+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:08.602913+0000","scrub_duration":0.0067664470000000001,"objects_trimmed":0,"snaptrim_duration":0.0049863559999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8,6,4,2147483647,15],"acting":[9,7,8,6,4,1,15],"avail_no_missing":["9(0)","1(5)","4(4)","6(3)","7(1)","8(2)","15(6)"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"2"},{"start":"8","length":"1"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1b","length":"2"}]},{"pgid":"3.13","version":"59'18","reported_seq":176,"reported_epoch":64,"state":"active+clean+remapped","last_fresh":"2023-11-05T08:33:17.374833+0000","last_change":"2023-11-05T08:33:17.380146+0000","last_active":"2023-11-05T08:33:17.374833+0000","last_peered":"2023-11-05T08:33:17.374833+0000","last_clean":"2023-11-05T08:33:17.374833+0000","last_became_active":"2023-11-05T08:32:59.212969+0000","last_became_peered":"2023-11-05T08:32:59.212969+0000","last_unstale":"2023-11-05T08:33:17.374833+0000","last_undegraded":"2023-11-05T08:33:17.374833+0000","last_fullsized":"2023-11-05T08:33:17.374833+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'14","last_scrub_stamp":"2023-11-05T08:33:02.644028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:02.644028+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:10.443168+0000","scrub_duration":0.007386386,"objects_trimmed":0,"snaptrim_duration":0.0068545480000000002,"stat_sum":{"num_bytes":344064,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,9,13,2147483647,4,10],"acting":[11,7,9,13,2,4,10],"avail_no_missing":["11(0)","2(4)","4(5)","7(1)","9(2)","10(6)","13(3)"],"object_location_counts":[{"shards":"2(4),4(5),7(1),9(2),10(6),11(0),13(3)","objects":2}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"2"},{"start":"8","length":"1"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1b","length":"2"}]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":56524800,"num_objects":90,"num_object_clones":38,"num_object_copies":630,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":57,"num_objects_misplaced":17,"num_objects_unfound":0,"num_objects_dirty":90,"num_whiteouts":5,"num_read":175,"num_read_kb":34136,"num_write":257,"num_write_kb":41216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":92459008,"data_stored":122019840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":516,"ondisk_log_size":516,"up":182,"acting":182,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":13898,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":22,"seq":94489280525,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34164,"kb_used_data":7140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337676,"statfs":{"total":96636764160,"available":96601780224,"internally_reserved":0,"allocated":7311360,"data_stored":9772926,"data_compressed":87442,"data_compressed_allocated":208896,"data_compressed_original":864256,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,4,5,6,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0189999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1339999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.887}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.081}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.669}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.96}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.613}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2429999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5129999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0099999999999998}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.871}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6829999999999998}]}]},{"osd":15,"up_from":21,"seq":90194313229,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32484,"kb_used_data":5460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339356,"statfs":{"total":96636764160,"available":96603500544,"internally_reserved":0,"allocated":5591040,"data_stored":6684924,"data_compressed":78565,"data_compressed_allocated":172032,"data_compressed_original":344064,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,4,5,6,7,8,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0089999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.131}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96399999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0780000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.22}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.873}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.157}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.236}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1970000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1339999999999999}]}]},{"osd":14,"up_from":22,"seq":94489280524,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31772,"kb_used_data":4744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340068,"statfs":{"total":96636764160,"available":96604229632,"internally_reserved":0,"allocated":4857856,"data_stored":7097694,"data_compressed":80493,"data_compressed_allocated":176128,"data_compressed_original":352256,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,4,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.282}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.018}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.22}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0049999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2330000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.133}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4039999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4319999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.47}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.766}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6879999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313223,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":4392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":4497408,"data_stored":4339168,"data_compressed":21230,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,9,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]}]},{"osd":3,"up_from":21,"seq":90194313225,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30380,"kb_used_data":3356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341460,"statfs":{"total":96636764160,"available":96605655040,"internally_reserved":0,"allocated":3436544,"data_stored":4001321,"data_compressed":40011,"data_compressed_allocated":90112,"data_compressed_original":180224,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,4,5,6,7,9,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.333}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0740000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1479999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.361}]}]},{"osd":6,"up_from":21,"seq":90194313228,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35208,"kb_used_data":8188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336632,"statfs":{"total":96636764160,"available":96600711168,"internally_reserved":0,"allocated":8384512,"data_stored":10625276,"data_compressed":78565,"data_compressed_allocated":172032,"data_compressed_original":344064,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93300000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96299999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83399999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0189999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1830000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6060000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.081}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3979999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3999999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6399999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7150000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.861}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4410000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.954}]}]},{"osd":1,"up_from":22,"seq":94489280524,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34404,"kb_used_data":7384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337436,"statfs":{"total":96636764160,"available":96601534464,"internally_reserved":0,"allocated":7561216,"data_stored":9294662,"data_compressed":78565,"data_compressed_allocated":172032,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[2,4,5,6,7,8,9,10,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.873}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91100000000000003}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97199999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94199999999999995}]}]},{"osd":2,"up_from":22,"seq":94489280524,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32972,"kb_used_data":5952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":6094848,"data_stored":8283494,"data_compressed":85514,"data_compressed_allocated":204800,"data_compressed_original":856064,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,4,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81599999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63200000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.224}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86699999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.177}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1479999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96499999999999997}]}]},{"osd":4,"up_from":22,"seq":94489280524,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34404,"kb_used_data":7384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337436,"statfs":{"total":96636764160,"available":96601534464,"internally_reserved":0,"allocated":7561216,"data_stored":9519356,"data_compressed":78565,"data_compressed_allocated":172032,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,5,6,7,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6479999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88800000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4590000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.054}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.617}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.536}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7749999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.341}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0539999999999998}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.821}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1949999999999998}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.849}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.911}]}]},{"osd":5,"up_from":22,"seq":94489280524,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31528,"kb_used_data":4508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340312,"statfs":{"total":96636764160,"available":96604479488,"internally_reserved":0,"allocated":4616192,"data_stored":5473094,"data_compressed":78565,"data_compressed_allocated":172032,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,4,6,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78400000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91300000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.474}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.226}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.28}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4470000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5760000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6519999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3700000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4990000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.827}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.601}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0299999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.974}]}]},{"osd":7,"up_from":22,"seq":94489280525,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33364,"kb_used_data":6344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338476,"statfs":{"total":96636764160,"available":96602599424,"internally_reserved":0,"allocated":6496256,"data_stored":8672070,"data_compressed":78565,"data_compressed_allocated":172032,"data_compressed_original":344064,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,4,5,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1930000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90500000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.889}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96699999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.085}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.51}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.167}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.298}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.655}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8080000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0209999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.736}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.843}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0379999999999998}]}]},{"osd":8,"up_from":22,"seq":94489280524,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36560,"kb_used_data":9540,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335280,"statfs":{"total":96636764160,"available":96599326720,"internally_reserved":0,"allocated":9768960,"data_stored":12604230,"data_compressed":78565,"data_compressed_allocated":172032,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,4,5,6,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95599999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0149999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5669999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.411}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.175}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3879999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6459999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.821}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.349}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.895}]}]},{"osd":9,"up_from":21,"seq":90194313228,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34584,"kb_used_data":7560,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337256,"statfs":{"total":96636764160,"available":96601350144,"internally_reserved":0,"allocated":7741440,"data_stored":9403742,"data_compressed":80493,"data_compressed_allocated":176128,"data_compressed_original":352256,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,4,6,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2549999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1479999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.181}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0269999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2669999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0700000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.752}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92400000000000004}]}]},{"osd":10,"up_from":22,"seq":94489280524,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33032,"kb_used_data":6012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338808,"statfs":{"total":96636764160,"available":96602939392,"internally_reserved":0,"allocated":6156288,"data_stored":8033094,"data_compressed":78565,"data_compressed_allocated":172032,"data_compressed_original":344064,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,4,5,6,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95299999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2749999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0680000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4330000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6539999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3620000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5369999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3160000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7090000000000001}]}]},{"osd":11,"up_from":21,"seq":90194313229,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35688,"kb_used_data":8668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336152,"statfs":{"total":96636764160,"available":96600219648,"internally_reserved":0,"allocated":8876032,"data_stored":11481926,"data_compressed":78565,"data_compressed_allocated":172032,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,4,5,6,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91000000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4630000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.399}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80700000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0069999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1899999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2270000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.839}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6459999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.498}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4330000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.921}]}]},{"osd":12,"up_from":22,"seq":94489280524,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30760,"kb_used_data":3736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341080,"statfs":{"total":96636764160,"available":96605265920,"internally_reserved":0,"allocated":3825664,"data_stored":4520724,"data_compressed":80493,"data_compressed_allocated":176128,"data_compressed_original":352256,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,4,5,6,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.123}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0680000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1799999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.097}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5209999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3560000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2290000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6949,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6949,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4165632,"data_stored":4165632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6856704,"data_stored":8802304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5337088,"data_stored":7262208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2977792,"data_stored":3723264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6860800,"data_stored":9027584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3911680,"data_stored":4980736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":7684096,"data_stored":10207232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":5791744,"data_stored":8179712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":9064448,"data_stored":12111872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":7024640,"data_stored":8900608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":5451776,"data_stored":7540736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8171520,"data_stored":10989568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":3112960,"data_stored":4018176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":7016448,"data_stored":9322496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":4141056,"data_stored":6594560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":4890624,"data_stored":6193152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-05T08:33:21.726 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pgp_num 26 2023-11-05T08:33:21.736 INFO:tasks.rados.rados.0.smithi090.stdout:220: finishing write tid 1 to smithi09039252-49 2023-11-05T08:33:21.736 INFO:tasks.rados.rados.0.smithi090.stdout:216: done (5 left) 2023-11-05T08:33:21.736 INFO:tasks.rados.rados.0.smithi090.stdout:218: done (4 left) 2023-11-05T08:33:21.736 INFO:tasks.rados.rados.0.smithi090.stdout:219: done (3 left) 2023-11-05T08:33:21.737 INFO:tasks.rados.rados.0.smithi090.stdout:223: delete oid 24 current snap is 17 2023-11-05T08:33:21.737 INFO:tasks.rados.rados.0.smithi090.stdout:220: finishing write tid 2 to smithi09039252-49 2023-11-05T08:33:21.737 INFO:tasks.rados.rados.0.smithi090.stdout:220: finishing write tid 3 to smithi09039252-49 2023-11-05T08:33:21.737 INFO:tasks.rados.rados.0.smithi090.stdout:220: finishing write tid 4 to smithi09039252-49 2023-11-05T08:33:21.737 INFO:tasks.rados.rados.0.smithi090.stdout:220: finishing write tid 5 to smithi09039252-49 2023-11-05T08:33:21.738 INFO:tasks.rados.rados.0.smithi090.stdout:220: oid 49 updating version 0 to 36 2023-11-05T08:33:21.738 INFO:tasks.rados.rados.0.smithi090.stdout:220: oid 49 version 36 is already newer than 34 2023-11-05T08:33:21.738 INFO:tasks.rados.rados.0.smithi090.stdout:220: oid 49 version 36 is already newer than 35 2023-11-05T08:33:21.738 INFO:tasks.rados.rados.0.smithi090.stdout:220: oid 49 updating version 36 to 37 2023-11-05T08:33:21.738 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 49 v 37 (ObjNum 100 snap 17 seq_num 100) dirty exists 2023-11-05T08:33:21.739 INFO:tasks.rados.rados.0.smithi090.stdout:220: left oid 49 (ObjNum 100 snap 17 seq_num 100) 2023-11-05T08:33:21.739 INFO:tasks.rados.rados.0.smithi090.stdout:222: finishing rollback tid 1 to smithi09039252-23 2023-11-05T08:33:21.739 INFO:tasks.rados.rados.0.smithi090.stdout:222: finishing rollback tid 2 to smithi09039252-23 2023-11-05T08:33:21.739 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 23 v 18 (ObjNum 60 snap 1 seq_num 60) dirty exists 2023-11-05T08:33:21.742 INFO:tasks.rados.rados.0.smithi090.stdout:220: done (3 left) 2023-11-05T08:33:21.742 INFO:tasks.rados.rados.0.smithi090.stdout:222: done (2 left) 2023-11-05T08:33:21.742 INFO:tasks.rados.rados.0.smithi090.stdout:223: done (1 left) 2023-11-05T08:33:21.743 INFO:tasks.rados.rados.0.smithi090.stdout:224: read oid 20 snap -1 2023-11-05T08:33:21.743 INFO:tasks.rados.rados.0.smithi090.stdout:224: expect (ObjNum 56 snap 1 seq_num 56) 2023-11-05T08:33:21.743 INFO:tasks.rados.rados.0.smithi090.stdout:225: copy_from oid 49 from oid 17 current snap is 17 2023-11-05T08:33:21.743 INFO:tasks.rados.rados.0.smithi090.stdout:226: snap_create 2023-11-05T08:33:21.746 INFO:tasks.rados.rados.0.smithi090.stdout:225: finishing copy_from to smithi09039252-49 2023-11-05T08:33:21.746 INFO:tasks.rados.rados.0.smithi090.stdout:225: got expected ENOENT (src dne) 2023-11-05T08:33:21.747 INFO:tasks.rados.rados.0.smithi090.stdout:225: finishing copy_from racing read to smithi09039252-49 2023-11-05T08:33:21.751 INFO:tasks.rados.rados.0.smithi090.stdout:224: expect (ObjNum 56 snap 1 seq_num 56) 2023-11-05T08:33:21.776 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:33:21.772+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:33:21.794 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:21.795 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:21.810 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-11-05T08:33:21.876 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:33:21.874+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:33:21.975 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:21.976 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:21.977 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:33:21.972+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:33:21.990 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-11-05T08:33:22.079 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:33:22.074+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:33:22.108 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:22.108 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:22.123 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2023-11-05T08:33:22.179 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:22.176+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:22.247 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:22.247 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:22.261 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2023-11-05T08:33:22.280 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:22.276+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:22.382 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:33:22.378+0000 7fdc5cd58640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:33:22.387 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:22.387 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:22.394 INFO:teuthology.orchestra.run.smithi002.stderr:set pool 3 pgp_num to 26 2023-11-05T08:33:22.402 INFO:tasks.rados.rados.0.smithi090.stdout:224: done (3 left) 2023-11-05T08:33:22.402 INFO:tasks.rados.rados.0.smithi090.stdout:225: done (2 left) 2023-11-05T08:33:22.402 INFO:tasks.rados.rados.0.smithi090.stdout:226: done (1 left) 2023-11-05T08:33:22.402 INFO:tasks.rados.rados.0.smithi090.stdout:227: snap_remove snap 17 2023-11-05T08:33:22.405 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2023-11-05T08:33:22.482 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:33:22.475+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:33:22.583 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:22.580+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:22.584 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:22.584 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:22.595 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-11-05T08:33:22.598 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2023-11-05T08:33:22.684 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:33:22.680+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:33:22.764 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:22.764 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:22.779 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2023-11-05T08:33:22.785 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:33:22.782+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:33:22.886 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:33:22.879+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:33:22.895 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:22.895 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:22.909 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2023-11-05T08:33:22.987 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:33:22.983+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:33:23.026 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:23.026 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:23.040 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-11-05T08:33:23.088 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:23.084+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:23.175 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:23.175 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:23.188 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-05T08:33:23.191 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-11-05T08:33:23.277 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-05T08:33:23.290 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:33:23.284+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:33:23.367 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:23.367 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:23.381 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-11-05T08:33:23.391 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:33:23.390+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:33:23.413 INFO:teuthology.orchestra.run.smithi002.stderr:noscrub is unset 2023-11-05T08:33:23.419 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-11-05T08:33:23.422 INFO:tasks.rados.rados.0.smithi090.stdout:227: done (1 left) 2023-11-05T08:33:23.422 INFO:tasks.rados.rados.0.smithi090.stdout:228: copy_from oid 39 from oid 26 current snap is 18 2023-11-05T08:33:23.422 INFO:tasks.rados.rados.0.smithi090.stdout:229: snap_create 2023-11-05T08:33:23.442 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1: {} 2023-11-05T08:33:23.442 INFO:teuthology.orchestra.run.smithi002.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-11-05T08:33:23.445 INFO:tasks.rados.rados.0.smithi090.stdout:228: finishing copy_from racing read to smithi09039252-39 2023-11-05T08:33:23.450 INFO:tasks.rados.rados.0.smithi090.stdout:228: finishing copy_from to smithi09039252-39 2023-11-05T08:33:23.450 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 39 v 34 (ObjNum 89 snap 16 seq_num 89) dirty exists 2023-11-05T08:33:23.465 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2: {} 2023-11-05T08:33:23.465 INFO:teuthology.orchestra.run.smithi002.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-11-05T08:33:23.465 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-11-05T08:33:23.482 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4: {} 2023-11-05T08:33:23.482 INFO:teuthology.orchestra.run.smithi002.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-11-05T08:33:23.492 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:33:23.487+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:33:23.501 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5: {} 2023-11-05T08:33:23.501 INFO:teuthology.orchestra.run.smithi002.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-11-05T08:33:23.522 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6: {} 2023-11-05T08:33:23.522 INFO:teuthology.orchestra.run.smithi002.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-11-05T08:33:23.548 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7: {} 2023-11-05T08:33:23.549 INFO:teuthology.orchestra.run.smithi002.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-11-05T08:33:23.559 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:33:23.559 INFO:teuthology.orchestra.run.smithi053.stdout: "size": 20, 2023-11-05T08:33:23.559 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 600, 2023-11-05T08:33:23.559 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2023-11-05T08:33:23.559 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.560 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es2 41/30 ECSubWrite(tid=40, reqid=osd.2.0:40, at_version=41'27, trim_to=0'0, roll_forward_to=40'25))", 2023-11-05T08:33:23.560 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.127197+0000", 2023-11-05T08:33:23.560 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 29.430120929000001, 2023-11-05T08:33:23.560 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.019700031, 2023-11-05T08:33:23.560 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:23.561 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:23.561 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:23.561 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.561 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:23.561 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127197+0000", 2023-11-05T08:33:23.562 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.562 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.562 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.562 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:23.562 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127197+0000", 2023-11-05T08:33:23.563 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.563 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.563 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.563 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:23.563 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127198+0000", 2023-11-05T08:33:23.564 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.0999999999999998e-07 2023-11-05T08:33:23.564 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.564 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.564 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:23.564 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127255+0000", 2023-11-05T08:33:23.564 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6762000000000003e-05 2023-11-05T08:33:23.565 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.565 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.565 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:23.565 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127291+0000", 2023-11-05T08:33:23.566 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5948999999999998e-05 2023-11-05T08:33:23.566 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.566 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.566 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:23.566 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127314+0000", 2023-11-05T08:33:23.566 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3686e-05 2023-11-05T08:33:23.567 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.567 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.567 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:23.567 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127534+0000", 2023-11-05T08:33:23.567 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00021922700000000001 2023-11-05T08:33:23.568 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.568 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.568 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:23.568 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127562+0000", 2023-11-05T08:33:23.568 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8099e-05 2023-11-05T08:33:23.569 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.569 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.569 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:23.569 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.146742+0000", 2023-11-05T08:33:23.569 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.019180135000000001 2023-11-05T08:33:23.569 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.570 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.570 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:23.570 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.146897+0000", 2023-11-05T08:33:23.570 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000155463 2023-11-05T08:33:23.570 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.571 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:23.571 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.571 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.571 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.571 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as2 41/33 ECSubWrite(tid=35, reqid=osd.7.0:35, at_version=41'12, trim_to=0'0, roll_forward_to=40'10))", 2023-11-05T08:33:23.572 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.130596+0000", 2023-11-05T08:33:23.572 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 29.426721705999999, 2023-11-05T08:33:23.572 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022182642999999998, 2023-11-05T08:33:23.572 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:23.572 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:23.572 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:23.573 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.573 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:23.573 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130596+0000", 2023-11-05T08:33:23.573 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.573 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.574 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.574 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:23.574 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130596+0000", 2023-11-05T08:33:23.574 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.574 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.574 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.575 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:23.575 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130597+0000", 2023-11-05T08:33:23.575 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.7600000000000006e-07 2023-11-05T08:33:23.575 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.575 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.576 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:23.576 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130660+0000", 2023-11-05T08:33:23.576 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3015000000000006e-05 2023-11-05T08:33:23.576 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.576 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.577 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:23.577 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130725+0000", 2023-11-05T08:33:23.577 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4988999999999997e-05 2023-11-05T08:33:23.577 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.577 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.578 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:23.578 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130750+0000", 2023-11-05T08:33:23.578 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4802000000000001e-05 2023-11-05T08:33:23.578 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.578 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.579 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:23.579 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.131037+0000", 2023-11-05T08:33:23.579 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00028702900000000001 2023-11-05T08:33:23.579 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.579 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.580 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:23.580 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.131064+0000", 2023-11-05T08:33:23.580 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7022e-05 2023-11-05T08:33:23.580 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.580 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.581 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:23.581 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.152706+0000", 2023-11-05T08:33:23.581 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.021641734999999999 2023-11-05T08:33:23.581 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.581 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.582 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:23.582 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.152779+0000", 2023-11-05T08:33:23.582 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.3075000000000001e-05 2023-11-05T08:33:23.582 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.582 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:23.582 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.583 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.583 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.583 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as2 41/33 ECSubWrite(tid=36, reqid=osd.7.0:36, at_version=41'14, trim_to=0'0, roll_forward_to=40'10))", 2023-11-05T08:33:23.583 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.130806+0000", 2023-11-05T08:33:23.583 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 29.426511825999999, 2023-11-05T08:33:23.584 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.035467152000000002, 2023-11-05T08:33:23.584 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:23.584 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:23.584 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:23.584 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.585 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:23.585 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130806+0000", 2023-11-05T08:33:23.585 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.585 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.585 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.586 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:23.586 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130806+0000", 2023-11-05T08:33:23.586 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.586 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.586 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.587 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:23.587 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130807+0000", 2023-11-05T08:33:23.587 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1000000000000001e-06 2023-11-05T08:33:23.587 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.587 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.588 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:23.588 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130852+0000", 2023-11-05T08:33:23.588 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4428999999999998e-05 2023-11-05T08:33:23.588 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.588 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.588 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:23.589 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130876+0000", 2023-11-05T08:33:23.589 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3975999999999999e-05 2023-11-05T08:33:23.589 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.589 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.589 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:23.590 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130896+0000", 2023-11-05T08:33:23.590 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9905999999999999e-05 2023-11-05T08:33:23.590 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.590 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.591 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:23.591 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.143384+0000", 2023-11-05T08:33:23.591 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012488727999999999 2023-11-05T08:33:23.591 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.591 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.592 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:23.592 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.143412+0000", 2023-11-05T08:33:23.592 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7988999999999999e-05 2023-11-05T08:33:23.592 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.592 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.593 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:23.593 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.166186+0000", 2023-11-05T08:33:23.593 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022773787 2023-11-05T08:33:23.593 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.593 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.594 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:23.594 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.166273+0000", 2023-11-05T08:33:23.594 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.7237000000000005e-05 2023-11-05T08:33:23.594 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.594 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:23.595 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.595 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.595 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.595 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es2 41/30 ECSubWrite(tid=41, reqid=osd.2.0:40, at_version=0'0, trim_to=0'0, roll_forward_to=41'27))", 2023-11-05T08:33:23.595 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.148654+0000", 2023-11-05T08:33:23.595 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 29.408664181999999, 2023-11-05T08:33:23.596 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017394370999999999, 2023-11-05T08:33:23.596 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:23.596 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:23.596 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:23.596 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.597 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:23.597 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148654+0000", 2023-11-05T08:33:23.597 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.597 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.597 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.598 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:23.598 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148654+0000", 2023-11-05T08:33:23.598 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.598 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.598 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.599 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:23.599 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148656+0000", 2023-11-05T08:33:23.599 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.677e-06 2023-11-05T08:33:23.599 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.599 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.599 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:23.600 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148683+0000", 2023-11-05T08:33:23.600 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7246e-05 2023-11-05T08:33:23.600 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.600 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.600 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:23.601 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148715+0000", 2023-11-05T08:33:23.601 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2518999999999998e-05 2023-11-05T08:33:23.601 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.601 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.601 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:23.602 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148742+0000", 2023-11-05T08:33:23.602 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6876000000000002e-05 2023-11-05T08:33:23.602 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.602 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.602 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:23.602 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148890+0000", 2023-11-05T08:33:23.603 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014771599999999999 2023-11-05T08:33:23.603 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.603 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.603 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:23.603 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148915+0000", 2023-11-05T08:33:23.604 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5099000000000001e-05 2023-11-05T08:33:23.604 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.604 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.604 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:23.604 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.165914+0000", 2023-11-05T08:33:23.605 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.016999018000000001 2023-11-05T08:33:23.605 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.605 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.605 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:23.605 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.166048+0000", 2023-11-05T08:33:23.606 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013422 2023-11-05T08:33:23.606 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.606 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:23.606 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.606 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.607 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.607 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.2s5 41/40 ECSubWrite(tid=187, reqid=client.4691.0:292, at_version=41'9, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:23.607 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.162642+0000", 2023-11-05T08:33:23.607 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 29.394675783, 2023-11-05T08:33:23.607 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010859549, 2023-11-05T08:33:23.608 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:23.608 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:23.608 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:23.608 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.608 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:23.608 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.162642+0000", 2023-11-05T08:33:23.608 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.609 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.609 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.609 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:23.609 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.162642+0000", 2023-11-05T08:33:23.609 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.610 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.610 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.610 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:23.610 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.162644+0000", 2023-11-05T08:33:23.610 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.823e-06 2023-11-05T08:33:23.611 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.611 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.611 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:23.611 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.162700+0000", 2023-11-05T08:33:23.611 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5863999999999999e-05 2023-11-05T08:33:23.611 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.612 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.612 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:23.612 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.162731+0000", 2023-11-05T08:33:23.612 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0731000000000001e-05 2023-11-05T08:33:23.612 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.613 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.613 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:23.613 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.162754+0000", 2023-11-05T08:33:23.613 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3234000000000002e-05 2023-11-05T08:33:23.613 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.613 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.614 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:23.614 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.162988+0000", 2023-11-05T08:33:23.614 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00023393399999999999 2023-11-05T08:33:23.614 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.614 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.615 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:23.615 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.163016+0000", 2023-11-05T08:33:23.615 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8309e-05 2023-11-05T08:33:23.615 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.615 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.615 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:23.616 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.173420+0000", 2023-11-05T08:33:23.616 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010403802 2023-11-05T08:33:23.616 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.616 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.616 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:23.617 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.173502+0000", 2023-11-05T08:33:23.617 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.1852000000000002e-05 2023-11-05T08:33:23.617 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.617 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:23.617 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.618 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.618 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.618 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as2 41/33 ECSubWrite(tid=37, reqid=osd.7.0:36, at_version=0'0, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:33:23.618 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.167511+0000", 2023-11-05T08:33:23.618 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 29.389807345000001, 2023-11-05T08:33:23.619 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011732691, 2023-11-05T08:33:23.619 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:23.619 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:23.619 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:23.619 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.619 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:23.620 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167511+0000", 2023-11-05T08:33:23.620 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.620 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.620 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.620 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:23.621 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167511+0000", 2023-11-05T08:33:23.621 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.621 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.621 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.621 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:23.622 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167513+0000", 2023-11-05T08:33:23.622 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.0829999999999998e-06 2023-11-05T08:33:23.622 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.622 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.622 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:23.622 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167540+0000", 2023-11-05T08:33:23.623 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7095999999999999e-05 2023-11-05T08:33:23.623 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.623 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.623 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:23.623 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167572+0000", 2023-11-05T08:33:23.624 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2462000000000002e-05 2023-11-05T08:33:23.624 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.624 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.624 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:23.624 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167594+0000", 2023-11-05T08:33:23.625 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1316999999999999e-05 2023-11-05T08:33:23.625 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.625 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.625 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:23.625 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167791+0000", 2023-11-05T08:33:23.625 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00019748199999999999 2023-11-05T08:33:23.626 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.626 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.626 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:23.626 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167815+0000", 2023-11-05T08:33:23.626 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3536999999999998e-05 2023-11-05T08:33:23.627 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.627 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.627 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:23.627 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.179154+0000", 2023-11-05T08:33:23.627 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011339373999999999 2023-11-05T08:33:23.627 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.628 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.628 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:23.628 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.179244+0000", 2023-11-05T08:33:23.628 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9339999999999995e-05 2023-11-05T08:33:23.628 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.629 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:23.629 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.629 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.629 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.629 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.2s5 41/40 ECSubWrite(tid=188, reqid=osd.13.0:188, at_version=41'11, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:23.629 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.171651+0000", 2023-11-05T08:33:23.630 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 29.385666757999999, 2023-11-05T08:33:23.630 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0076644390000000003, 2023-11-05T08:33:23.630 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:23.630 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:23.630 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:23.631 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.631 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:23.631 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171651+0000", 2023-11-05T08:33:23.631 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.631 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.631 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.632 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:23.632 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171651+0000", 2023-11-05T08:33:23.632 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.632 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.632 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.633 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:23.633 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171653+0000", 2023-11-05T08:33:23.633 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5039999999999999e-06 2023-11-05T08:33:23.633 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.633 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.634 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:23.634 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171700+0000", 2023-11-05T08:33:23.634 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.6989e-05 2023-11-05T08:33:23.634 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.634 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.634 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:23.635 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171731+0000", 2023-11-05T08:33:23.635 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1310999999999999e-05 2023-11-05T08:33:23.635 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.635 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.635 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:23.636 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171750+0000", 2023-11-05T08:33:23.636 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9188e-05 2023-11-05T08:33:23.636 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.636 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.636 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:23.637 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171900+0000", 2023-11-05T08:33:23.637 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00015010699999999999 2023-11-05T08:33:23.637 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.637 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.637 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:23.638 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171920+0000", 2023-11-05T08:33:23.638 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9626000000000001e-05 2023-11-05T08:33:23.638 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.638 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.638 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:23.638 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.179250+0000", 2023-11-05T08:33:23.639 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0073298900000000004 2023-11-05T08:33:23.639 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.639 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.639 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:23.639 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.179316+0000", 2023-11-05T08:33:23.640 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5823999999999999e-05 2023-11-05T08:33:23.640 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.640 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:23.640 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.640 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.641 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.641 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as2 42/33 ECSubWrite(tid=38, reqid=osd.7.0:38, at_version=42'16, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:33:23.641 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:55.121755+0000", 2023-11-05T08:33:23.641 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 28.435562758, 2023-11-05T08:33:23.641 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011609566, 2023-11-05T08:33:23.642 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:23.642 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:23.642 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:23.642 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.642 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:23.642 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.121755+0000", 2023-11-05T08:33:23.643 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.643 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.643 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.643 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:23.643 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.121755+0000", 2023-11-05T08:33:23.644 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.644 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.644 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.644 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:23.644 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.121756+0000", 2023-11-05T08:33:23.645 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.0299999999999998e-07 2023-11-05T08:33:23.645 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.645 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.645 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:23.645 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.121810+0000", 2023-11-05T08:33:23.645 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3918000000000001e-05 2023-11-05T08:33:23.646 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.646 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.646 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:23.646 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.121861+0000", 2023-11-05T08:33:23.646 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.0846000000000001e-05 2023-11-05T08:33:23.647 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.647 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.647 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:23.647 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.121889+0000", 2023-11-05T08:33:23.647 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8359000000000001e-05 2023-11-05T08:33:23.647 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.648 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.648 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:23.648 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.122158+0000", 2023-11-05T08:33:23.648 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00026898800000000001 2023-11-05T08:33:23.648 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.649 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.649 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:23.649 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.122192+0000", 2023-11-05T08:33:23.649 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3259000000000001e-05 2023-11-05T08:33:23.649 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.649 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.650 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:23.650 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.133269+0000", 2023-11-05T08:33:23.650 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011076979000000001 2023-11-05T08:33:23.650 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.650 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.651 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:23.651 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.133365+0000", 2023-11-05T08:33:23.651 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6414000000000002e-05 2023-11-05T08:33:23.651 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.651 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:23.651 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.652 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.652 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.652 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as2 42/33 ECSubWrite(tid=39, reqid=osd.7.0:39, at_version=42'18, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:33:23.652 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:55.124648+0000", 2023-11-05T08:33:23.652 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 28.432669898, 2023-11-05T08:33:23.653 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.025426339999999999, 2023-11-05T08:33:23.653 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:23.653 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:23.653 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:23.653 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.653 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:23.654 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.124648+0000", 2023-11-05T08:33:23.654 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.654 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.654 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.654 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:23.655 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.124648+0000", 2023-11-05T08:33:23.655 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.655 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.655 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.655 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:23.656 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.124650+0000", 2023-11-05T08:33:23.656 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.463e-06 2023-11-05T08:33:23.656 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.656 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.656 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:23.657 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.124710+0000", 2023-11-05T08:33:23.657 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0001999999999999e-05 2023-11-05T08:33:23.657 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.657 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.657 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:23.657 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.124739+0000", 2023-11-05T08:33:23.658 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9278999999999999e-05 2023-11-05T08:33:23.658 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.658 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.658 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:23.658 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.124763+0000", 2023-11-05T08:33:23.659 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4036000000000002e-05 2023-11-05T08:33:23.659 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.659 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.659 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:23.659 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.125122+0000", 2023-11-05T08:33:23.660 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00035926900000000001 2023-11-05T08:33:23.660 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.660 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.660 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:23.660 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.125149+0000", 2023-11-05T08:33:23.660 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7192999999999999e-05 2023-11-05T08:33:23.661 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.661 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.661 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:23.661 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.149969+0000", 2023-11-05T08:33:23.661 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.024819074 2023-11-05T08:33:23.661 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.662 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.662 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:23.662 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.150075+0000", 2023-11-05T08:33:23.662 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000106024 2023-11-05T08:33:23.662 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.663 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:23.663 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.663 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.663 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.663 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es2 42/30 ECSubWrite(tid=43, reqid=osd.2.0:43, at_version=42'29, trim_to=0'0, roll_forward_to=41'27))", 2023-11-05T08:33:23.664 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:55.134544+0000", 2023-11-05T08:33:23.664 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 28.422773977999999, 2023-11-05T08:33:23.664 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.02091324, 2023-11-05T08:33:23.664 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:23.664 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:23.664 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:23.665 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.665 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:23.665 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134544+0000", 2023-11-05T08:33:23.665 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.665 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.666 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.666 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:23.666 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134544+0000", 2023-11-05T08:33:23.666 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.666 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.667 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.667 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:23.667 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134545+0000", 2023-11-05T08:33:23.667 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3499999999999996e-07 2023-11-05T08:33:23.667 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.667 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.668 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:23.668 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134593+0000", 2023-11-05T08:33:23.668 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.8538999999999997e-05 2023-11-05T08:33:23.668 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.668 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.669 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:23.669 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134621+0000", 2023-11-05T08:33:23.669 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7939000000000001e-05 2023-11-05T08:33:23.669 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.669 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.670 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:23.670 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134641+0000", 2023-11-05T08:33:23.670 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9352999999999999e-05 2023-11-05T08:33:23.670 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.670 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.671 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:23.671 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134788+0000", 2023-11-05T08:33:23.671 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000147806 2023-11-05T08:33:23.671 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.671 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.672 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:23.672 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134850+0000", 2023-11-05T08:33:23.672 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1614999999999999e-05 2023-11-05T08:33:23.672 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.672 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.673 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:23.673 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.155400+0000", 2023-11-05T08:33:23.673 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.020550023000000001 2023-11-05T08:33:23.673 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.674 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.674 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:23.674 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.155457+0000", 2023-11-05T08:33:23.674 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7330000000000002e-05 2023-11-05T08:33:23.675 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.675 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:23.675 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.675 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.675 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.675 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.2s5 42/40 ECSubWrite(tid=192, reqid=osd.13.0:192, at_version=42'13, trim_to=0'0, roll_forward_to=41'11))", 2023-11-05T08:33:23.676 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:55.150316+0000", 2023-11-05T08:33:23.676 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 28.407002481999999, 2023-11-05T08:33:23.676 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0088926939999999996, 2023-11-05T08:33:23.676 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:23.677 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:23.677 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:23.677 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.677 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:23.677 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.150316+0000", 2023-11-05T08:33:23.678 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.678 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.678 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.678 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:23.678 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.150316+0000", 2023-11-05T08:33:23.678 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.679 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.679 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.679 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:23.679 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.150318+0000", 2023-11-05T08:33:23.679 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.0140000000000001e-06 2023-11-05T08:33:23.680 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.680 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.680 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:23.680 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.150377+0000", 2023-11-05T08:33:23.680 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9231000000000003e-05 2023-11-05T08:33:23.681 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.681 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.681 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:23.681 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.150411+0000", 2023-11-05T08:33:23.681 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4279000000000002e-05 2023-11-05T08:33:23.682 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.682 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.682 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:23.682 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.150435+0000", 2023-11-05T08:33:23.682 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4165999999999999e-05 2023-11-05T08:33:23.683 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.683 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.683 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:23.683 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.152989+0000", 2023-11-05T08:33:23.683 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0025534889999999999 2023-11-05T08:33:23.683 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.684 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.684 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:23.684 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.153035+0000", 2023-11-05T08:33:23.684 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.6078000000000002e-05 2023-11-05T08:33:23.684 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.685 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.685 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:23.685 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.159134+0000", 2023-11-05T08:33:23.685 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0060986069999999998 2023-11-05T08:33:23.685 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.686 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.686 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:23.686 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.159208+0000", 2023-11-05T08:33:23.686 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.483e-05 2023-11-05T08:33:23.686 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.686 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:23.687 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.687 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.687 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.687 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as2 42/33 ECSubWrite(tid=40, reqid=osd.7.0:39, at_version=0'0, trim_to=0'0, roll_forward_to=42'18))", 2023-11-05T08:33:23.688 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:55.151263+0000", 2023-11-05T08:33:23.688 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 28.406055048999999, 2023-11-05T08:33:23.688 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0080059020000000005, 2023-11-05T08:33:23.688 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:23.688 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:23.688 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:23.689 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.689 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:23.689 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.151263+0000", 2023-11-05T08:33:23.689 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.689 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.690 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.690 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:23.690 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.151263+0000", 2023-11-05T08:33:23.690 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.690 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.691 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.691 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:23.691 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.151265+0000", 2023-11-05T08:33:23.691 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9939999999999999e-06 2023-11-05T08:33:23.691 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.692 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.692 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:23.692 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.151295+0000", 2023-11-05T08:33:23.692 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0355e-05 2023-11-05T08:33:23.692 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.693 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.693 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:23.693 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.151328+0000", 2023-11-05T08:33:23.693 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2246000000000003e-05 2023-11-05T08:33:23.693 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.693 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.694 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:23.694 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.151359+0000", 2023-11-05T08:33:23.694 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1458999999999998e-05 2023-11-05T08:33:23.694 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.694 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.695 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:23.695 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.153854+0000", 2023-11-05T08:33:23.695 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002495115 2023-11-05T08:33:23.695 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.696 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.696 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:23.696 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.153879+0000", 2023-11-05T08:33:23.696 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5065000000000001e-05 2023-11-05T08:33:23.696 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.697 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.697 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:23.697 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.159215+0000", 2023-11-05T08:33:23.697 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0053359690000000003 2023-11-05T08:33:23.697 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.698 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.698 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:23.698 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.159269+0000", 2023-11-05T08:33:23.698 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3699000000000002e-05 2023-11-05T08:33:23.698 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.698 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:23.699 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.699 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.699 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.699 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as2 52/45 ECSubWrite(tid=45, reqid=client.4691.0:349, at_version=52'21, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:23.699 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:05.262984+0000", 2023-11-05T08:33:23.700 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 18.294334406000001, 2023-11-05T08:33:23.700 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0076861730000000001, 2023-11-05T08:33:23.700 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:23.700 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:23.700 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:23.701 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.701 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:23.701 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.262984+0000", 2023-11-05T08:33:23.701 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.701 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.702 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.702 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:23.702 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.262984+0000", 2023-11-05T08:33:23.702 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.702 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.703 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.703 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:23.703 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.262985+0000", 2023-11-05T08:33:23.703 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.8400000000000002e-07 2023-11-05T08:33:23.703 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.704 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.704 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:23.704 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.263110+0000", 2023-11-05T08:33:23.704 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012501999999999999 2023-11-05T08:33:23.704 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.704 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.705 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:23.705 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.263149+0000", 2023-11-05T08:33:23.705 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8899000000000001e-05 2023-11-05T08:33:23.705 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.705 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.706 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:23.706 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.263173+0000", 2023-11-05T08:33:23.706 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4243000000000002e-05 2023-11-05T08:33:23.706 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.706 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.707 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:23.707 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.263816+0000", 2023-11-05T08:33:23.707 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00064262900000000003 2023-11-05T08:33:23.707 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.707 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.708 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:23.708 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.263830+0000", 2023-11-05T08:33:23.708 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4676e-05 2023-11-05T08:33:23.708 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.708 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.709 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:23.709 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.270569+0000", 2023-11-05T08:33:23.709 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0067386939999999999 2023-11-05T08:33:23.709 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.709 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.710 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:23.710 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.270670+0000", 2023-11-05T08:33:23.710 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000101028 2023-11-05T08:33:23.710 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.710 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:23.711 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.711 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.711 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.711 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.14s3 56/48 ECSubWrite(tid=32, reqid=client.4691.0:374, at_version=56'19, trim_to=0'0, roll_forward_to=49'18))", 2023-11-05T08:33:23.711 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:09.317760+0000", 2023-11-05T08:33:23.712 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 14.239557983999999, 2023-11-05T08:33:23.712 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0071132290000000004, 2023-11-05T08:33:23.712 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:23.712 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:23.712 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:23.713 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.713 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:23.713 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:09.317760+0000", 2023-11-05T08:33:23.713 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.714 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.714 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.714 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:23.714 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:09.317760+0000", 2023-11-05T08:33:23.714 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.715 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.715 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.715 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:23.715 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:09.317761+0000", 2023-11-05T08:33:23.715 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3300000000000003e-07 2023-11-05T08:33:23.716 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.716 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.716 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:23.716 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:09.318127+0000", 2023-11-05T08:33:23.716 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00036675999999999999 2023-11-05T08:33:23.717 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.717 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.717 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:23.717 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:09.318169+0000", 2023-11-05T08:33:23.718 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1888999999999998e-05 2023-11-05T08:33:23.718 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.718 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.718 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:23.718 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:09.318194+0000", 2023-11-05T08:33:23.719 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4256e-05 2023-11-05T08:33:23.719 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.719 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.719 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:23.719 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:09.318338+0000", 2023-11-05T08:33:23.720 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000144363 2023-11-05T08:33:23.720 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.720 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.720 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:23.720 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:09.318360+0000", 2023-11-05T08:33:23.721 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.2022999999999999e-05 2023-11-05T08:33:23.721 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.721 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.721 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:23.721 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:09.324756+0000", 2023-11-05T08:33:23.722 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0063961809999999999 2023-11-05T08:33:23.722 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.722 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.722 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:23.722 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:09.324873+0000", 2023-11-05T08:33:23.723 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000117324 2023-11-05T08:33:23.723 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.723 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:23.723 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.723 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.724 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.724 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.bs2 66/37 ECSubWrite(tid=108, reqid=client.4691.0:496, at_version=66'23, trim_to=0'0, roll_forward_to=65'21))", 2023-11-05T08:33:23.724 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.398619+0000", 2023-11-05T08:33:23.724 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4.1586995030000002, 2023-11-05T08:33:23.724 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.016555743000000001, 2023-11-05T08:33:23.724 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:23.725 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:23.725 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:23.725 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.725 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:23.725 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.398619+0000", 2023-11-05T08:33:23.726 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.726 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.726 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.726 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:23.726 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.398619+0000", 2023-11-05T08:33:23.727 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.727 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.727 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.727 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:23.727 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.398629+0000", 2023-11-05T08:33:23.727 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.9529999999999994e-06 2023-11-05T08:33:23.728 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.728 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.728 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:23.728 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.399216+0000", 2023-11-05T08:33:23.728 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00058774000000000001 2023-11-05T08:33:23.729 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.729 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.729 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:23.729 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.399255+0000", 2023-11-05T08:33:23.729 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8902000000000001e-05 2023-11-05T08:33:23.730 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.730 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.730 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:23.730 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.399280+0000", 2023-11-05T08:33:23.730 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4827000000000001e-05 2023-11-05T08:33:23.731 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.731 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.731 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:23.731 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.399530+0000", 2023-11-05T08:33:23.731 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00024994500000000001 2023-11-05T08:33:23.732 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.732 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.732 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:23.732 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.399558+0000", 2023-11-05T08:33:23.732 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8370000000000001e-05 2023-11-05T08:33:23.732 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.733 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.733 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:23.733 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.415065+0000", 2023-11-05T08:33:23.733 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.015506565 2023-11-05T08:33:23.733 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.734 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.734 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:23.734 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.415174+0000", 2023-11-05T08:33:23.734 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000109441 2023-11-05T08:33:23.734 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.735 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:23.735 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.735 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.735 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.735 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.bs2 66/37 ECSubWrite(tid=109, reqid=client.4691.0:497, at_version=66'24, trim_to=0'0, roll_forward_to=65'21))", 2023-11-05T08:33:23.736 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.403242+0000", 2023-11-05T08:33:23.736 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4.154075755, 2023-11-05T08:33:23.736 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.018077743, 2023-11-05T08:33:23.736 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:23.736 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:23.736 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:23.737 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.737 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:23.737 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.403242+0000", 2023-11-05T08:33:23.737 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.737 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.738 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.738 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:23.738 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.403242+0000", 2023-11-05T08:33:23.738 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.739 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.739 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.739 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:23.739 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.403245+0000", 2023-11-05T08:33:23.739 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6000000000000001e-06 2023-11-05T08:33:23.740 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.740 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.740 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:23.740 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.403863+0000", 2023-11-05T08:33:23.740 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000617955 2023-11-05T08:33:23.741 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.741 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.741 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:23.741 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.403904+0000", 2023-11-05T08:33:23.742 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1542999999999998e-05 2023-11-05T08:33:23.742 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.742 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.742 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:23.742 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.403931+0000", 2023-11-05T08:33:23.743 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6016e-05 2023-11-05T08:33:23.743 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.743 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.743 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:23.743 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.404129+0000", 2023-11-05T08:33:23.744 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000198397 2023-11-05T08:33:23.744 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.744 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.744 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:23.744 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.404161+0000", 2023-11-05T08:33:23.745 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2039000000000003e-05 2023-11-05T08:33:23.745 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.745 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.745 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:23.745 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.421200+0000", 2023-11-05T08:33:23.746 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017038604999999998 2023-11-05T08:33:23.746 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.746 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.746 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:23.747 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.421320+0000", 2023-11-05T08:33:23.747 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000120588 2023-11-05T08:33:23.747 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.747 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:23.747 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.748 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.748 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.748 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.bs2 66/37 ECSubWrite(tid=110, reqid=client.4691.0:498, at_version=66'25, trim_to=0'0, roll_forward_to=65'21))", 2023-11-05T08:33:23.748 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.408787+0000", 2023-11-05T08:33:23.748 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4.1485307469999997, 2023-11-05T08:33:23.749 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012658930000000001, 2023-11-05T08:33:23.749 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:23.749 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:23.749 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:23.750 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.750 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:23.750 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.408787+0000", 2023-11-05T08:33:23.750 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.750 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.751 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.751 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:23.751 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.408787+0000", 2023-11-05T08:33:23.751 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.751 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.752 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.752 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:23.752 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.408789+0000", 2023-11-05T08:33:23.752 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.0169999999999999e-06 2023-11-05T08:33:23.752 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.753 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.753 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:23.753 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.408927+0000", 2023-11-05T08:33:23.753 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000137811 2023-11-05T08:33:23.753 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.754 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.754 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:23.754 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.408961+0000", 2023-11-05T08:33:23.754 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4173e-05 2023-11-05T08:33:23.754 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.755 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.755 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:23.755 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.408985+0000", 2023-11-05T08:33:23.755 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3961999999999999e-05 2023-11-05T08:33:23.755 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.756 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.756 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:23.756 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.410719+0000", 2023-11-05T08:33:23.756 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0017334729999999999 2023-11-05T08:33:23.756 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.756 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.757 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:23.757 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.410744+0000", 2023-11-05T08:33:23.757 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4956999999999998e-05 2023-11-05T08:33:23.757 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.757 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.758 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:23.758 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.421379+0000", 2023-11-05T08:33:23.758 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010635627999999999 2023-11-05T08:33:23.758 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.758 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.759 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:23.759 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.421446+0000", 2023-11-05T08:33:23.759 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.6909000000000006e-05 2023-11-05T08:33:23.759 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.759 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:23.759 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.760 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.760 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.760 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.bs2 66/37 ECSubWrite(tid=111, reqid=client.4691.0:499, at_version=66'26, trim_to=0'0, roll_forward_to=65'21))", 2023-11-05T08:33:23.760 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.409025+0000", 2023-11-05T08:33:23.760 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4.1482935870000004, 2023-11-05T08:33:23.761 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012479762, 2023-11-05T08:33:23.761 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:23.761 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:23.761 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:23.761 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.762 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:23.762 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.409025+0000", 2023-11-05T08:33:23.762 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.762 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.762 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.762 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:23.763 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.409025+0000", 2023-11-05T08:33:23.763 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.763 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.763 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.763 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:23.764 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.409026+0000", 2023-11-05T08:33:23.764 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1599999999999999e-06 2023-11-05T08:33:23.764 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.764 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.764 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:23.765 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.409067+0000", 2023-11-05T08:33:23.765 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1189000000000002e-05 2023-11-05T08:33:23.765 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.765 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.765 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:23.766 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.409087+0000", 2023-11-05T08:33:23.766 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9905999999999999e-05 2023-11-05T08:33:23.766 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.766 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.766 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:23.766 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.409098+0000", 2023-11-05T08:33:23.767 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1643e-05 2023-11-05T08:33:23.767 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.767 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.767 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:23.767 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.412893+0000", 2023-11-05T08:33:23.768 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0037942829999999999 2023-11-05T08:33:23.768 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.768 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.768 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:23.768 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.412919+0000", 2023-11-05T08:33:23.769 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6083000000000001e-05 2023-11-05T08:33:23.769 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.769 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.769 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:23.769 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.421452+0000", 2023-11-05T08:33:23.770 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0085328890000000001 2023-11-05T08:33:23.770 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.770 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.770 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:23.770 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.421504+0000", 2023-11-05T08:33:23.770 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2608999999999997e-05 2023-11-05T08:33:23.771 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.771 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:23.771 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.771 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.771 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.772 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es2 66/30 ECSubWrite(tid=87, reqid=osd.2.0:87, at_version=66'65, trim_to=0'0, roll_forward_to=65'63))", 2023-11-05T08:33:23.772 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.411058+0000", 2023-11-05T08:33:23.772 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4.1462596600000001, 2023-11-05T08:33:23.772 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010528946000000001, 2023-11-05T08:33:23.772 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:23.773 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:23.773 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:23.773 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.773 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:23.773 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.411058+0000", 2023-11-05T08:33:23.774 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.774 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.774 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.774 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:23.774 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.411058+0000", 2023-11-05T08:33:23.774 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.775 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.775 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.775 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:23.775 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.411059+0000", 2023-11-05T08:33:23.775 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5300000000000004e-07 2023-11-05T08:33:23.776 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.776 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.776 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:23.776 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.411112+0000", 2023-11-05T08:33:23.776 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2676000000000002e-05 2023-11-05T08:33:23.777 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.777 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.777 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:23.777 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.411142+0000", 2023-11-05T08:33:23.777 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9796000000000001e-05 2023-11-05T08:33:23.777 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.778 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.778 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:23.778 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.411164+0000", 2023-11-05T08:33:23.778 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.2249999999999999e-05 2023-11-05T08:33:23.778 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.779 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.779 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:23.779 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.411322+0000", 2023-11-05T08:33:23.779 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00015797700000000001 2023-11-05T08:33:23.779 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.780 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.780 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:23.780 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.411353+0000", 2023-11-05T08:33:23.780 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0682999999999997e-05 2023-11-05T08:33:23.780 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.780 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.781 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:23.781 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.421482+0000", 2023-11-05T08:33:23.781 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.01012939 2023-11-05T08:33:23.781 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.781 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.782 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:23.782 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.421587+0000", 2023-11-05T08:33:23.782 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010552099999999999 2023-11-05T08:33:23.782 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.782 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:23.783 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.783 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.783 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.783 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es2 66/30 ECSubWrite(tid=88, reqid=osd.2.0:88, at_version=66'67, trim_to=0'0, roll_forward_to=65'63))", 2023-11-05T08:33:23.783 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.414905+0000", 2023-11-05T08:33:23.783 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4.1424128820000004, 2023-11-05T08:33:23.784 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0083023899999999998, 2023-11-05T08:33:23.784 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:23.784 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:23.784 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:23.784 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.785 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:23.785 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.414905+0000", 2023-11-05T08:33:23.785 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.785 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.785 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.786 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:23.786 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.414905+0000", 2023-11-05T08:33:23.786 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:23.786 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.786 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.787 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:23.787 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.414907+0000", 2023-11-05T08:33:23.787 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.81e-06 2023-11-05T08:33:23.787 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.787 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.788 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:23.788 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.414960+0000", 2023-11-05T08:33:23.788 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3386e-05 2023-11-05T08:33:23.788 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.788 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.789 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:23.789 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.414992+0000", 2023-11-05T08:33:23.789 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1115999999999998e-05 2023-11-05T08:33:23.789 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.789 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.789 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:23.790 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.415012+0000", 2023-11-05T08:33:23.790 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.0843e-05 2023-11-05T08:33:23.790 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.790 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.790 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:23.791 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.415228+0000", 2023-11-05T08:33:23.791 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00021531600000000001 2023-11-05T08:33:23.791 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.791 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.791 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:23.792 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.415259+0000", 2023-11-05T08:33:23.792 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1399000000000002e-05 2023-11-05T08:33:23.792 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.792 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.792 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:23.793 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.423080+0000", 2023-11-05T08:33:23.793 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0078204220000000005 2023-11-05T08:33:23.793 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:23.793 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:23.793 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:23.793 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.423208+0000", 2023-11-05T08:33:23.794 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012809799999999999 2023-11-05T08:33:23.794 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.794 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:23.794 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.794 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:23.795 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:23.795 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:33:23.796 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8: {} 2023-11-05T08:33:23.797 INFO:teuthology.orchestra.run.smithi002.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-11-05T08:33:23.797 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9: {} 2023-11-05T08:33:23.798 INFO:teuthology.orchestra.run.smithi002.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-11-05T08:33:23.798 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-11-05T08:33:23.799 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10: {} 2023-11-05T08:33:23.799 INFO:teuthology.orchestra.run.smithi002.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-11-05T08:33:23.800 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11: {} 2023-11-05T08:33:23.800 INFO:teuthology.orchestra.run.smithi002.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-11-05T08:33:23.800 INFO:teuthology.orchestra.run.smithi002.stdout:osd.12: {} 2023-11-05T08:33:23.801 INFO:teuthology.orchestra.run.smithi002.stderr:osd.12: osd_enable_op_tracker = 'true' 2023-11-05T08:33:23.801 INFO:teuthology.orchestra.run.smithi002.stdout:osd.13: {} 2023-11-05T08:33:23.802 INFO:teuthology.orchestra.run.smithi002.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-11-05T08:33:23.802 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:33:23.791+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:33:23.803 INFO:teuthology.orchestra.run.smithi002.stdout:osd.14: {} 2023-11-05T08:33:23.803 INFO:teuthology.orchestra.run.smithi002.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-11-05T08:33:23.804 INFO:teuthology.orchestra.run.smithi002.stdout:osd.15: {} 2023-11-05T08:33:23.804 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-05T08:33:23.805 INFO:teuthology.orchestra.run.smithi002.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-11-05T08:33:23.805 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-05T08:33:23.898 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:23.892+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:23.937 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:33:23.937 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2023-11-05T08:33:23.938 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 0 2023-11-05T08:33:23.938 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:33:23.952 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-11-05T08:33:23.998 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:33:23.998+0000 7fdc5cd58640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:33:24.099 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:33:24.096+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:33:24.120 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:33:24.121 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2023-11-05T08:33:24.121 INFO:teuthology.orchestra.run.smithi053.stdout: "complaint_time": 30, 2023-11-05T08:33:24.121 INFO:teuthology.orchestra.run.smithi053.stdout: "num_blocked_ops": 0 2023-11-05T08:33:24.121 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:33:24.135 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-11-05T08:33:24.200 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:33:24.198+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:33:24.302 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:33:24.298+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:33:24.312 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:33:24.312 INFO:teuthology.orchestra.run.smithi053.stdout: "size": 20, 2023-11-05T08:33:24.313 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 600, 2023-11-05T08:33:24.313 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2023-11-05T08:33:24.313 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.313 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:271 3.18s0 3:1aa38538:::smithi09039252-11:5 [read 0~1097728,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:33:24.313 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:53.109072+0000", 2023-11-05T08:33:24.314 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 31.199867778000002, 2023-11-05T08:33:24.314 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.069197392, 2023-11-05T08:33:24.314 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:24.314 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2023-11-05T08:33:24.314 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:24.315 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:24.315 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:24.315 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 271 2023-11-05T08:33:24.315 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.315 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:24.316 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.316 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:24.316 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109072+0000", 2023-11-05T08:33:24.316 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.316 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.316 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.317 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:24.317 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109072+0000", 2023-11-05T08:33:24.317 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.317 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.317 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.318 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:24.318 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109076+0000", 2023-11-05T08:33:24.318 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.9639999999999997e-06 2023-11-05T08:33:24.318 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.318 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.319 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:24.319 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109094+0000", 2023-11-05T08:33:24.319 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7995e-05 2023-11-05T08:33:24.319 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.319 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.320 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:24.320 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109134+0000", 2023-11-05T08:33:24.320 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.0729000000000003e-05 2023-11-05T08:33:24.320 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.320 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.320 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:24.321 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109160+0000", 2023-11-05T08:33:24.321 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5276e-05 2023-11-05T08:33:24.321 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.321 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.321 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:24.322 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.134337+0000", 2023-11-05T08:33:24.322 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.025177609 2023-11-05T08:33:24.322 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.322 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.322 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2023-11-05T08:33:24.323 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.134667+0000", 2023-11-05T08:33:24.323 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00032950600000000003 2023-11-05T08:33:24.323 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.323 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.323 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:24.323 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.166070+0000", 2023-11-05T08:33:24.324 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0314031800000001 2023-11-05T08:33:24.324 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.324 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.324 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:24.324 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.166695+0000", 2023-11-05T08:33:24.325 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000625137 2023-11-05T08:33:24.325 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.325 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.325 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:24.325 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.178269+0000", 2023-11-05T08:33:24.325 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011573996 2023-11-05T08:33:24.326 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.326 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:24.326 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.326 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.326 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.327 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:272 3.18s0 3:1aa38538:::smithi09039252-11:5 [read 0~1097728,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:33:24.327 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:53.109221+0000", 2023-11-05T08:33:24.327 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 31.199718166, 2023-11-05T08:33:24.327 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0781554369999999, 2023-11-05T08:33:24.327 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:24.328 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2023-11-05T08:33:24.328 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:24.328 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:24.328 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:24.328 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 272 2023-11-05T08:33:24.329 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.329 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:24.329 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.329 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:24.329 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109221+0000", 2023-11-05T08:33:24.330 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.330 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.330 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.330 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:24.330 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109221+0000", 2023-11-05T08:33:24.331 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.331 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.331 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.331 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:24.331 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109223+0000", 2023-11-05T08:33:24.332 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4330000000000001e-06 2023-11-05T08:33:24.332 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.332 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.332 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:24.332 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109232+0000", 2023-11-05T08:33:24.332 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.234e-06 2023-11-05T08:33:24.333 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.333 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.333 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:24.333 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109254+0000", 2023-11-05T08:33:24.333 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1576000000000001e-05 2023-11-05T08:33:24.334 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.334 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.334 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:24.334 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109270+0000", 2023-11-05T08:33:24.334 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6396e-05 2023-11-05T08:33:24.335 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.335 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.335 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:24.335 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.134760+0000", 2023-11-05T08:33:24.335 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.025490292000000001 2023-11-05T08:33:24.336 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.336 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.336 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2023-11-05T08:33:24.336 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.134764+0000", 2023-11-05T08:33:24.336 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5470000000000001e-06 2023-11-05T08:33:24.337 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.337 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.337 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:24.337 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.178397+0000", 2023-11-05T08:33:24.337 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.04363341 2023-11-05T08:33:24.337 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.338 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.338 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:24.338 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.178606+0000", 2023-11-05T08:33:24.338 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00020828200000000001 2023-11-05T08:33:24.338 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.339 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.339 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:24.339 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.187377+0000", 2023-11-05T08:33:24.339 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0087712669999999993 2023-11-05T08:33:24.339 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.340 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:24.340 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.340 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.340 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.340 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:273 3.18s0 3:1aa38538:::smithi09039252-11:5 [read 0~1097728,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:33:24.340 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:53.109291+0000", 2023-11-05T08:33:24.341 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 31.199648395000001, 2023-11-05T08:33:24.341 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.08037483, 2023-11-05T08:33:24.341 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:24.341 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2023-11-05T08:33:24.341 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:24.342 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:24.342 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:24.342 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 273 2023-11-05T08:33:24.342 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.342 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:24.342 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.343 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:24.343 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109291+0000", 2023-11-05T08:33:24.343 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.343 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.343 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.344 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:24.344 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109291+0000", 2023-11-05T08:33:24.344 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.344 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.344 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.344 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:24.345 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109292+0000", 2023-11-05T08:33:24.345 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.054e-06 2023-11-05T08:33:24.345 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.345 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.345 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:24.346 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109299+0000", 2023-11-05T08:33:24.346 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.7700000000000004e-06 2023-11-05T08:33:24.346 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.346 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.346 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:24.347 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109317+0000", 2023-11-05T08:33:24.347 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.8529000000000002e-05 2023-11-05T08:33:24.347 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.347 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.347 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:24.347 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109332+0000", 2023-11-05T08:33:24.348 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4905999999999999e-05 2023-11-05T08:33:24.348 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.348 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.348 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:24.348 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.134830+0000", 2023-11-05T08:33:24.349 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.025497912000000001 2023-11-05T08:33:24.349 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.349 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.349 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2023-11-05T08:33:24.349 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.134832+0000", 2023-11-05T08:33:24.350 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1399999999999998e-06 2023-11-05T08:33:24.350 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.350 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.350 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:24.350 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.179990+0000", 2023-11-05T08:33:24.350 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0451579209999999 2023-11-05T08:33:24.351 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.351 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.351 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:24.351 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.180250+0000", 2023-11-05T08:33:24.351 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000259595 2023-11-05T08:33:24.352 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.352 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.352 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:24.352 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.189666+0000", 2023-11-05T08:33:24.352 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0094160029999999992 2023-11-05T08:33:24.353 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.353 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:24.353 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.353 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.353 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.354 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpReadReply(3.18s0 41/40 ECSubReadReply(tid=25, attrs_read=0))", 2023-11-05T08:33:24.354 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.173059+0000", 2023-11-05T08:33:24.354 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 30.135880438000001, 2023-11-05T08:33:24.354 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0069229169999999998, 2023-11-05T08:33:24.354 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:24.354 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:24.355 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:24.355 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.355 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:24.355 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.173059+0000", 2023-11-05T08:33:24.355 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.356 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.356 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.356 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:24.356 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.173059+0000", 2023-11-05T08:33:24.356 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.357 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.357 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.357 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:24.357 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.173061+0000", 2023-11-05T08:33:24.357 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4720000000000001e-06 2023-11-05T08:33:24.358 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.358 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.358 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:24.358 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.174176+0000", 2023-11-05T08:33:24.359 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001115904 2023-11-05T08:33:24.359 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.359 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.359 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:24.359 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.174201+0000", 2023-11-05T08:33:24.360 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4281e-05 2023-11-05T08:33:24.360 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.360 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.360 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:24.360 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.174215+0000", 2023-11-05T08:33:24.361 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4575e-05 2023-11-05T08:33:24.361 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.361 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.361 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:24.361 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.174738+0000", 2023-11-05T08:33:24.362 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000523122 2023-11-05T08:33:24.362 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.362 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.362 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:24.363 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.179982+0000", 2023-11-05T08:33:24.363 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0052435629999999997 2023-11-05T08:33:24.363 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.363 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:24.363 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.363 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.364 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.364 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.11s3 41/40 ECSubWrite(tid=65, reqid=client.4691.0:287, at_version=41'13, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:24.364 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.173108+0000", 2023-11-05T08:33:24.364 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 30.135831105000001, 2023-11-05T08:33:24.364 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012254233999999999, 2023-11-05T08:33:24.365 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:24.365 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:24.365 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:24.365 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.365 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:24.366 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.173108+0000", 2023-11-05T08:33:24.366 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.366 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.366 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.366 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:24.367 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.173108+0000", 2023-11-05T08:33:24.367 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.367 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.367 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.367 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:24.368 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.173109+0000", 2023-11-05T08:33:24.368 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.7700000000000004e-07 2023-11-05T08:33:24.368 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.368 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.368 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:24.368 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.174570+0000", 2023-11-05T08:33:24.369 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00146053 2023-11-05T08:33:24.369 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.369 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.369 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:24.369 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.174598+0000", 2023-11-05T08:33:24.370 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8404000000000001e-05 2023-11-05T08:33:24.370 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.370 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.370 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:24.370 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.174614+0000", 2023-11-05T08:33:24.371 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6436999999999999e-05 2023-11-05T08:33:24.371 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.371 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.371 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:24.371 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.174752+0000", 2023-11-05T08:33:24.371 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013757799999999999 2023-11-05T08:33:24.372 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.372 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.372 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:24.372 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.174775+0000", 2023-11-05T08:33:24.372 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.287e-05 2023-11-05T08:33:24.373 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.373 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.373 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:24.373 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.185293+0000", 2023-11-05T08:33:24.373 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010518192000000001 2023-11-05T08:33:24.374 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.374 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.374 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:24.374 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.185363+0000", 2023-11-05T08:33:24.374 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9546000000000001e-05 2023-11-05T08:33:24.375 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.375 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:24.375 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.375 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.375 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.375 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.11s3 41/40 ECSubWrite(tid=66, reqid=client.4691.0:288, at_version=41'14, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:24.376 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.177594+0000", 2023-11-05T08:33:24.376 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 30.131345909, 2023-11-05T08:33:24.376 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0090366969999999998, 2023-11-05T08:33:24.376 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:24.376 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:24.377 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:24.377 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.377 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:24.377 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.177594+0000", 2023-11-05T08:33:24.377 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.378 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.378 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.378 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:24.378 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.177594+0000", 2023-11-05T08:33:24.378 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.379 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.379 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.379 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:24.379 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.177595+0000", 2023-11-05T08:33:24.379 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6599999999999994e-07 2023-11-05T08:33:24.380 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.380 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.380 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:24.380 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.180867+0000", 2023-11-05T08:33:24.380 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.003272264 2023-11-05T08:33:24.381 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.381 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.381 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:24.381 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.180893+0000", 2023-11-05T08:33:24.381 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5956e-05 2023-11-05T08:33:24.381 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.382 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.382 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:24.382 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.180915+0000", 2023-11-05T08:33:24.382 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1756999999999999e-05 2023-11-05T08:33:24.382 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.383 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.383 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:24.383 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.181066+0000", 2023-11-05T08:33:24.383 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000151213 2023-11-05T08:33:24.383 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.384 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.384 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:24.384 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.181085+0000", 2023-11-05T08:33:24.384 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9389999999999999e-05 2023-11-05T08:33:24.384 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.384 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.385 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:24.385 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.186564+0000", 2023-11-05T08:33:24.385 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0054784569999999999 2023-11-05T08:33:24.385 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.385 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.386 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:24.386 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.186630+0000", 2023-11-05T08:33:24.386 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.6693999999999995e-05 2023-11-05T08:33:24.386 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.386 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:24.387 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.387 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.387 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.387 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:317 3.17s0 3:e93e8c6a:::smithi09039252-17:head [delete] snapc b=[7,5] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e45)", 2023-11-05T08:33:24.387 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:58.238377+0000", 2023-11-05T08:33:24.388 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 26.070562704, 2023-11-05T08:33:24.388 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9937814299999999, 2023-11-05T08:33:24.388 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:24.388 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:24.388 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:24.388 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:24.389 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:24.389 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 317 2023-11-05T08:33:24.389 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.389 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:24.389 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.390 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:24.390 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:58.238377+0000", 2023-11-05T08:33:24.390 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.390 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.390 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.391 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:24.391 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:58.238377+0000", 2023-11-05T08:33:24.391 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.391 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.391 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.392 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:24.392 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:58.238379+0000", 2023-11-05T08:33:24.392 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9460000000000001e-06 2023-11-05T08:33:24.392 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.392 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.392 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:24.393 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:58.238390+0000", 2023-11-05T08:33:24.393 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1065e-05 2023-11-05T08:33:24.393 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.393 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.393 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:24.394 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:58.238412+0000", 2023-11-05T08:33:24.394 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.2529999999999999e-05 2023-11-05T08:33:24.394 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.394 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.394 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:24.395 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:58.238425+0000", 2023-11-05T08:33:24.395 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3077e-05 2023-11-05T08:33:24.395 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.395 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.395 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:24.396 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:58.238536+0000", 2023-11-05T08:33:24.396 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00011045899999999999 2023-11-05T08:33:24.396 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.396 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.396 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2023-11-05T08:33:24.396 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:58.238541+0000", 2023-11-05T08:33:24.397 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9710000000000003e-06 2023-11-05T08:33:24.397 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.397 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.397 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:24.397 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:59.210323+0000", 2023-11-05T08:33:24.398 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.97178187500000002 2023-11-05T08:33:24.398 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.398 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.398 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2023-11-05T08:33:24.398 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:59.210329+0000", 2023-11-05T08:33:24.399 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5100000000000004e-06 2023-11-05T08:33:24.399 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.399 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.399 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:24.399 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:00.215041+0000", 2023-11-05T08:33:24.400 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.004711436 2023-11-05T08:33:24.400 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.400 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.400 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for missing object", 2023-11-05T08:33:24.400 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:00.216219+0000", 2023-11-05T08:33:24.401 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0011779609999999999 2023-11-05T08:33:24.401 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.401 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.401 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:24.401 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:00.226127+0000", 2023-11-05T08:33:24.401 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0099082130000000008 2023-11-05T08:33:24.402 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.402 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.402 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:24.402 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:00.226269+0000", 2023-11-05T08:33:24.403 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000142032 2023-11-05T08:33:24.403 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.403 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.403 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:24.403 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:00.227010+0000", 2023-11-05T08:33:24.404 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00074093 2023-11-05T08:33:24.404 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.404 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.404 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:24.404 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:00.231106+0000", 2023-11-05T08:33:24.404 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0040965589999999996 2023-11-05T08:33:24.405 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.405 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.405 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:24.405 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:00.231612+0000", 2023-11-05T08:33:24.405 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00050578000000000005 2023-11-05T08:33:24.406 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.406 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.406 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:24.406 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:00.232158+0000", 2023-11-05T08:33:24.406 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00054608599999999999 2023-11-05T08:33:24.407 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.407 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:24.407 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.407 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.407 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.408 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.16s5 51/45 ECSubWrite(tid=58, reqid=osd.11.0:58, at_version=51'19, trim_to=0'0, roll_forward_to=49'17))", 2023-11-05T08:33:24.408 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:04.226078+0000", 2023-11-05T08:33:24.408 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 20.082861824999998, 2023-11-05T08:33:24.408 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.019914247999999999, 2023-11-05T08:33:24.408 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:24.409 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:24.409 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:24.409 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.409 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:24.409 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.226078+0000", 2023-11-05T08:33:24.410 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.410 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.410 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.410 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:24.410 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.226078+0000", 2023-11-05T08:33:24.410 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.411 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.411 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.411 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:24.411 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.226079+0000", 2023-11-05T08:33:24.411 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.133e-06 2023-11-05T08:33:24.412 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.412 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.412 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:24.412 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.226140+0000", 2023-11-05T08:33:24.412 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0809e-05 2023-11-05T08:33:24.413 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.413 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.413 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:24.413 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.226171+0000", 2023-11-05T08:33:24.413 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0889000000000002e-05 2023-11-05T08:33:24.414 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.414 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.414 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:24.414 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.226196+0000", 2023-11-05T08:33:24.415 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5945999999999999e-05 2023-11-05T08:33:24.415 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.415 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.415 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:24.415 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.226413+0000", 2023-11-05T08:33:24.416 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000216379 2023-11-05T08:33:24.416 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.416 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.416 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:24.416 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.226442+0000", 2023-11-05T08:33:24.417 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9323e-05 2023-11-05T08:33:24.417 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.417 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.417 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:24.417 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.245889+0000", 2023-11-05T08:33:24.417 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.019446938 2023-11-05T08:33:24.418 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.418 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.418 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:24.418 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.245992+0000", 2023-11-05T08:33:24.418 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010283100000000001 2023-11-05T08:33:24.419 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.419 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:24.419 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.419 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.419 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.421 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:338 3.17s0 3:e93e8c6a:::smithi09039252-17:head [read 0~0,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:24.421 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:05.250110+0000", 2023-11-05T08:33:24.421 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 19.058829464999999, 2023-11-05T08:33:24.421 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0080018280000000008, 2023-11-05T08:33:24.422 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:24.422 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:24.422 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:24.422 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:24.422 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:24.423 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 338 2023-11-05T08:33:24.423 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.423 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:24.423 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.423 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:24.423 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250110+0000", 2023-11-05T08:33:24.424 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.424 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.424 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.424 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:24.424 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250110+0000", 2023-11-05T08:33:24.425 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.425 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.425 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.425 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:24.425 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250113+0000", 2023-11-05T08:33:24.425 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3230000000000002e-06 2023-11-05T08:33:24.426 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.426 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.426 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:24.426 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250128+0000", 2023-11-05T08:33:24.426 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4749999999999999e-05 2023-11-05T08:33:24.427 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.427 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.427 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:24.427 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250163+0000", 2023-11-05T08:33:24.427 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5079000000000001e-05 2023-11-05T08:33:24.428 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.428 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.428 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:24.428 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250186+0000", 2023-11-05T08:33:24.428 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.2566e-05 2023-11-05T08:33:24.429 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.429 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.429 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:24.429 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250320+0000", 2023-11-05T08:33:24.429 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000134544 2023-11-05T08:33:24.430 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.430 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.430 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op must wait for map", 2023-11-05T08:33:24.430 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250338+0000", 2023-11-05T08:33:24.430 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7813e-05 2023-11-05T08:33:24.431 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.431 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.431 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:24.431 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.257738+0000", 2023-11-05T08:33:24.431 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0074003050000000003 2023-11-05T08:33:24.431 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.432 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.432 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:24.432 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258112+0000", 2023-11-05T08:33:24.432 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00037344800000000002 2023-11-05T08:33:24.432 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.433 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:24.433 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.433 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.433 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.433 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:339 3.17s0 3:e93e8c6a:::smithi09039252-17:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:24.434 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:05.250225+0000", 2023-11-05T08:33:24.434 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 19.058714517999999, 2023-11-05T08:33:24.434 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0084669749999999998, 2023-11-05T08:33:24.434 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:24.434 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:24.435 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:24.435 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:24.435 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:24.435 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 339 2023-11-05T08:33:24.435 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.436 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:24.436 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.436 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:24.436 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250225+0000", 2023-11-05T08:33:24.436 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.437 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.437 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.437 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:24.437 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250225+0000", 2023-11-05T08:33:24.437 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.437 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.438 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.438 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:24.438 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250226+0000", 2023-11-05T08:33:24.438 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.8400000000000003e-07 2023-11-05T08:33:24.438 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.439 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.439 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:24.439 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250232+0000", 2023-11-05T08:33:24.439 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.2099999999999998e-06 2023-11-05T08:33:24.439 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.440 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.440 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:24.440 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250248+0000", 2023-11-05T08:33:24.440 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5639000000000001e-05 2023-11-05T08:33:24.440 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.440 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.441 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:24.441 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250257+0000", 2023-11-05T08:33:24.441 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.8459999999999997e-06 2023-11-05T08:33:24.441 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.441 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.442 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:24.442 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250524+0000", 2023-11-05T08:33:24.442 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00026785099999999999 2023-11-05T08:33:24.442 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.442 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.443 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:33:24.443 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250553+0000", 2023-11-05T08:33:24.443 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8649999999999998e-05 2023-11-05T08:33:24.443 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.443 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.444 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:24.444 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258255+0000", 2023-11-05T08:33:24.444 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0077024119999999996 2023-11-05T08:33:24.444 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.444 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.445 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:24.445 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258692+0000", 2023-11-05T08:33:24.445 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00043648299999999999 2023-11-05T08:33:24.445 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.445 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:24.446 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.446 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.446 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.446 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:340 3.17s0 3:e93e8c6a:::smithi09039252-17:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:24.446 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:05.250298+0000", 2023-11-05T08:33:24.447 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 19.058641779999999, 2023-11-05T08:33:24.447 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0088381339999999992, 2023-11-05T08:33:24.447 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:24.447 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:24.447 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:24.448 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:24.448 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:24.448 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 340 2023-11-05T08:33:24.448 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.448 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:24.448 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.449 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:24.449 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250298+0000", 2023-11-05T08:33:24.449 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.449 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.449 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.450 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:24.450 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250298+0000", 2023-11-05T08:33:24.450 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.450 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.450 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.451 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:24.451 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250298+0000", 2023-11-05T08:33:24.451 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.3e-07 2023-11-05T08:33:24.451 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.451 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.452 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:24.452 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250306+0000", 2023-11-05T08:33:24.452 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.0400000000000004e-06 2023-11-05T08:33:24.452 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.452 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.453 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:24.453 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250317+0000", 2023-11-05T08:33:24.453 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1914e-05 2023-11-05T08:33:24.453 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.453 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.453 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:24.454 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250325+0000", 2023-11-05T08:33:24.454 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.7200000000000006e-06 2023-11-05T08:33:24.454 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.454 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.454 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:24.455 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250639+0000", 2023-11-05T08:33:24.455 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00031349200000000001 2023-11-05T08:33:24.455 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.455 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.455 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:33:24.456 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250664+0000", 2023-11-05T08:33:24.456 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5786e-05 2023-11-05T08:33:24.456 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.456 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.456 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:24.456 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258785+0000", 2023-11-05T08:33:24.457 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0081202930000000006 2023-11-05T08:33:24.457 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.457 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.457 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:24.457 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.259136+0000", 2023-11-05T08:33:24.458 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00035115899999999998 2023-11-05T08:33:24.458 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.458 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:24.458 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.458 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.459 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.459 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:342 3.8s0 3:11e42f5e:::smithi09039252-35:head [copy-from ver 10] snapc 10=[7] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:24.459 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:05.254724+0000", 2023-11-05T08:33:24.459 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 19.054215042999999, 2023-11-05T08:33:24.459 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.043497013000000001, 2023-11-05T08:33:24.460 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:24.460 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:24.460 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:24.460 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:24.460 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:24.460 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 342 2023-11-05T08:33:24.461 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.461 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:24.461 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.461 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:24.462 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.254724+0000", 2023-11-05T08:33:24.462 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.462 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.462 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.462 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:24.463 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.254724+0000", 2023-11-05T08:33:24.463 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.463 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.463 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.463 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:24.464 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.254727+0000", 2023-11-05T08:33:24.464 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.2570000000000002e-06 2023-11-05T08:33:24.464 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.464 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.464 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:24.464 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.254741+0000", 2023-11-05T08:33:24.465 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4746000000000001e-05 2023-11-05T08:33:24.465 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.465 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.465 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:24.465 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.254771+0000", 2023-11-05T08:33:24.466 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0015999999999999e-05 2023-11-05T08:33:24.466 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.466 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.466 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:24.466 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.254796+0000", 2023-11-05T08:33:24.467 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4536e-05 2023-11-05T08:33:24.467 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.467 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.467 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:24.467 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.256414+0000", 2023-11-05T08:33:24.468 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0016180369999999999 2023-11-05T08:33:24.468 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.468 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.468 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:24.468 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.256671+0000", 2023-11-05T08:33:24.469 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00025643699999999998 2023-11-05T08:33:24.469 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.469 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.469 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:24.469 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.291461+0000", 2023-11-05T08:33:24.470 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.034790465999999999 2023-11-05T08:33:24.470 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.470 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.470 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:24.470 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.297059+0000", 2023-11-05T08:33:24.471 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0055981729999999997 2023-11-05T08:33:24.471 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.471 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.471 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:24.471 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.297172+0000", 2023-11-05T08:33:24.472 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00011328900000000001 2023-11-05T08:33:24.472 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.472 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.472 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:24.472 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.298221+0000", 2023-11-05T08:33:24.473 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001049056 2023-11-05T08:33:24.473 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.473 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:24.473 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.473 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.474 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.474 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:343 3.8s0 3:11e42f5e:::smithi09039252-35:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:24.474 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:05.254831+0000", 2023-11-05T08:33:24.474 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 19.054108121999999, 2023-11-05T08:33:24.474 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.041641958, 2023-11-05T08:33:24.475 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:24.475 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2023-11-05T08:33:24.475 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:24.475 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:24.475 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:24.475 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 343 2023-11-05T08:33:24.476 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.476 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:24.476 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.476 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:24.476 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.254831+0000", 2023-11-05T08:33:24.477 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.477 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.477 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.477 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:24.477 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.254831+0000", 2023-11-05T08:33:24.478 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.478 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.478 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.478 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:24.478 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.254833+0000", 2023-11-05T08:33:24.479 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9069999999999999e-06 2023-11-05T08:33:24.479 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.479 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.479 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:24.480 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.254841+0000", 2023-11-05T08:33:24.480 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.5959999999999998e-06 2023-11-05T08:33:24.480 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.480 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.480 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:24.480 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.254864+0000", 2023-11-05T08:33:24.481 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3499999999999999e-05 2023-11-05T08:33:24.481 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.481 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.481 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:24.481 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.254875+0000", 2023-11-05T08:33:24.482 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1032999999999999e-05 2023-11-05T08:33:24.482 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.482 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.482 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:24.482 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.257178+0000", 2023-11-05T08:33:24.483 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002302784 2023-11-05T08:33:24.483 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.483 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.483 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for blocked object", 2023-11-05T08:33:24.483 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.257359+0000", 2023-11-05T08:33:24.484 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00018044600000000001 2023-11-05T08:33:24.484 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.484 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.484 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:24.484 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.295697+0000", 2023-11-05T08:33:24.485 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.038338192 2023-11-05T08:33:24.485 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.485 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.485 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:24.485 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.295953+0000", 2023-11-05T08:33:24.486 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00025650899999999998 2023-11-05T08:33:24.486 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.486 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.486 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:24.486 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.296473+0000", 2023-11-05T08:33:24.487 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00051999100000000003 2023-11-05T08:33:24.487 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.487 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:24.487 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.487 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.488 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.488 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.18s4 52/45 ECSubWrite(tid=205, reqid=client.4691.0:351, at_version=52'24, trim_to=0'0, roll_forward_to=51'23))", 2023-11-05T08:33:24.488 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:05.281620+0000", 2023-11-05T08:33:24.488 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 19.027319780999999, 2023-11-05T08:33:24.488 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0073326679999999997, 2023-11-05T08:33:24.489 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:24.489 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:24.489 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:24.489 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.489 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:24.489 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.281620+0000", 2023-11-05T08:33:24.490 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.490 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.490 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.490 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:24.490 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.281620+0000", 2023-11-05T08:33:24.491 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.491 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.491 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.491 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:24.491 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.281621+0000", 2023-11-05T08:33:24.491 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.7400000000000002e-07 2023-11-05T08:33:24.492 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.492 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.492 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:24.492 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.282585+0000", 2023-11-05T08:33:24.492 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00096495399999999996 2023-11-05T08:33:24.493 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.493 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.493 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:24.493 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.282639+0000", 2023-11-05T08:33:24.493 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3282000000000002e-05 2023-11-05T08:33:24.494 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.494 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.494 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:24.494 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.282670+0000", 2023-11-05T08:33:24.494 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0825999999999999e-05 2023-11-05T08:33:24.495 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.495 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.495 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:24.495 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.282933+0000", 2023-11-05T08:33:24.495 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000263758 2023-11-05T08:33:24.495 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.496 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.496 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:24.496 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.282966+0000", 2023-11-05T08:33:24.496 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3121999999999999e-05 2023-11-05T08:33:24.496 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.497 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.497 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:24.497 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.288887+0000", 2023-11-05T08:33:24.497 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0059202129999999997 2023-11-05T08:33:24.497 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.498 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.498 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:24.498 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.288952+0000", 2023-11-05T08:33:24.498 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5739000000000002e-05 2023-11-05T08:33:24.498 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.499 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:24.499 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.499 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.499 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.499 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:412 3.8s0 3:11e42f5e:::smithi09039252-35:head [append 0~0] snapc 1b=[1b,1a,17,7] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2023-11-05T08:33:24.500 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:14.389862+0000", 2023-11-05T08:33:24.500 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 9.9190772910000007, 2023-11-05T08:33:24.500 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010619417000000001, 2023-11-05T08:33:24.500 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:24.500 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:24.500 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:24.501 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:24.501 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:24.501 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 412 2023-11-05T08:33:24.501 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.501 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:24.502 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.502 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:24.502 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.389862+0000", 2023-11-05T08:33:24.502 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.502 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.503 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.503 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:24.503 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.389862+0000", 2023-11-05T08:33:24.503 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.503 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.504 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.504 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:24.504 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.389865+0000", 2023-11-05T08:33:24.504 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3599999999999999e-06 2023-11-05T08:33:24.504 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.505 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.505 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:24.505 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.389875+0000", 2023-11-05T08:33:24.505 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0489000000000001e-05 2023-11-05T08:33:24.505 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.505 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.506 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:24.506 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.389905+0000", 2023-11-05T08:33:24.506 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0020000000000001e-05 2023-11-05T08:33:24.506 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.506 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.507 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:24.507 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.389921+0000", 2023-11-05T08:33:24.507 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5682999999999999e-05 2023-11-05T08:33:24.507 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.507 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.508 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:24.508 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.390092+0000", 2023-11-05T08:33:24.508 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00017081600000000001 2023-11-05T08:33:24.508 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.508 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.509 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:24.509 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.390428+0000", 2023-11-05T08:33:24.509 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00033592099999999998 2023-11-05T08:33:24.509 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.509 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.509 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:24.510 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.392907+0000", 2023-11-05T08:33:24.510 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0024797439999999999 2023-11-05T08:33:24.510 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.510 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.510 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:24.511 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.399481+0000", 2023-11-05T08:33:24.511 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0065741369999999999 2023-11-05T08:33:24.511 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.511 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.511 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:24.512 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.400140+0000", 2023-11-05T08:33:24.512 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00065843399999999995 2023-11-05T08:33:24.512 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.512 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.512 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:24.513 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.400482+0000", 2023-11-05T08:33:24.513 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000341813 2023-11-05T08:33:24.513 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.513 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:24.513 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.513 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.514 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.514 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:413 3.8s0 3:11e42f5e:::smithi09039252-35:head [rollback 7] snapc 1b=[1b,1a,17,7] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2023-11-05T08:33:24.514 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:14.389956+0000", 2023-11-05T08:33:24.514 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 9.9189838600000009, 2023-11-05T08:33:24.515 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011423634, 2023-11-05T08:33:24.515 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:24.515 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:24.515 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:24.515 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:24.515 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:24.516 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 413 2023-11-05T08:33:24.516 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.516 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:24.516 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.516 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:24.517 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.389956+0000", 2023-11-05T08:33:24.517 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.517 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.517 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.517 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:24.518 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.389956+0000", 2023-11-05T08:33:24.518 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.518 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.518 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.518 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:24.519 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.389957+0000", 2023-11-05T08:33:24.519 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.22e-06 2023-11-05T08:33:24.519 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.519 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.519 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:24.519 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.389964+0000", 2023-11-05T08:33:24.520 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.3000000000000004e-06 2023-11-05T08:33:24.520 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.520 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.520 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:24.520 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.390000+0000", 2023-11-05T08:33:24.521 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5982999999999998e-05 2023-11-05T08:33:24.521 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.521 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.521 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:24.521 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.390011+0000", 2023-11-05T08:33:24.522 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.099e-05 2023-11-05T08:33:24.522 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.522 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.522 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:24.522 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.394506+0000", 2023-11-05T08:33:24.523 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0044950939999999998 2023-11-05T08:33:24.523 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.523 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.523 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:24.523 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.394702+0000", 2023-11-05T08:33:24.523 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00019568 2023-11-05T08:33:24.524 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.524 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.524 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:24.524 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.395966+0000", 2023-11-05T08:33:24.524 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0012637379999999999 2023-11-05T08:33:24.525 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.525 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.525 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:24.525 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.399485+0000", 2023-11-05T08:33:24.525 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.003518864 2023-11-05T08:33:24.526 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.526 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.526 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:24.526 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.401053+0000", 2023-11-05T08:33:24.526 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0015684620000000001 2023-11-05T08:33:24.526 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.527 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.527 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:24.527 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.401379+0000", 2023-11-05T08:33:24.527 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00032630300000000001 2023-11-05T08:33:24.527 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.528 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:24.528 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.528 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.528 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.528 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:414 3.8s0 3:11e42f5e:::smithi09039252-35:head [append 0~0] snapc 1b=[1b,1a,17,7] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2023-11-05T08:33:24.529 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:14.390035+0000", 2023-11-05T08:33:24.529 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 9.9189045319999991, 2023-11-05T08:33:24.529 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.01234273, 2023-11-05T08:33:24.529 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:24.529 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:24.529 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:24.530 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:24.530 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:24.530 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 414 2023-11-05T08:33:24.530 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.530 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:24.531 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.531 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:24.531 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.390035+0000", 2023-11-05T08:33:24.531 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.531 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.531 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.532 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:24.532 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.390035+0000", 2023-11-05T08:33:24.532 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.532 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.532 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.533 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:24.533 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.390036+0000", 2023-11-05T08:33:24.533 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2500000000000001e-06 2023-11-05T08:33:24.533 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.533 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.534 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:24.534 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.390043+0000", 2023-11-05T08:33:24.534 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3640000000000004e-06 2023-11-05T08:33:24.534 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.534 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.535 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:24.535 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.390063+0000", 2023-11-05T08:33:24.535 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9902999999999999e-05 2023-11-05T08:33:24.535 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.535 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.535 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:24.536 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.390081+0000", 2023-11-05T08:33:24.536 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.8369e-05 2023-11-05T08:33:24.536 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.536 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.536 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:24.537 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.396827+0000", 2023-11-05T08:33:24.537 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0067464500000000002 2023-11-05T08:33:24.537 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.537 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.537 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:24.538 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.397166+0000", 2023-11-05T08:33:24.538 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00033854000000000002 2023-11-05T08:33:24.538 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.538 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.538 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:24.539 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.398388+0000", 2023-11-05T08:33:24.539 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0012222859999999999 2023-11-05T08:33:24.539 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.539 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.539 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:24.540 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.400877+0000", 2023-11-05T08:33:24.540 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002489144 2023-11-05T08:33:24.540 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.540 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.540 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:24.540 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.401956+0000", 2023-11-05T08:33:24.541 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0010789389999999999 2023-11-05T08:33:24.541 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.541 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.541 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:24.541 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.402378+0000", 2023-11-05T08:33:24.542 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00042148499999999998 2023-11-05T08:33:24.542 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.542 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:24.542 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.542 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.543 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.543 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.16s5 66/45 ECSubWrite(tid=112, reqid=osd.11.0:112, at_version=66'34, trim_to=0'0, roll_forward_to=65'32))", 2023-11-05T08:33:24.543 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.400318+0000", 2023-11-05T08:33:24.543 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4.9086218749999997, 2023-11-05T08:33:24.543 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.015622313000000001, 2023-11-05T08:33:24.543 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:24.544 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:24.544 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:24.544 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.544 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:24.544 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.400318+0000", 2023-11-05T08:33:24.545 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.545 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.545 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.545 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:24.545 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.400318+0000", 2023-11-05T08:33:24.546 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.546 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.546 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.546 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:24.546 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.400318+0000", 2023-11-05T08:33:24.546 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8599999999999998e-07 2023-11-05T08:33:24.547 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.547 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.547 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:24.547 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.400371+0000", 2023-11-05T08:33:24.547 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2636000000000002e-05 2023-11-05T08:33:24.548 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.548 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.548 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:24.548 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.400408+0000", 2023-11-05T08:33:24.548 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.7015999999999999e-05 2023-11-05T08:33:24.549 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.549 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.549 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:24.549 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.400436+0000", 2023-11-05T08:33:24.549 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8113e-05 2023-11-05T08:33:24.549 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.550 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.550 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:24.550 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.400633+0000", 2023-11-05T08:33:24.550 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000197147 2023-11-05T08:33:24.550 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.551 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.551 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:24.551 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.400665+0000", 2023-11-05T08:33:24.551 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1498999999999998e-05 2023-11-05T08:33:24.551 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.552 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.552 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:24.552 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.415850+0000", 2023-11-05T08:33:24.552 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.015185067999999999 2023-11-05T08:33:24.552 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.552 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.553 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:24.553 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.415940+0000", 2023-11-05T08:33:24.553 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.0247999999999994e-05 2023-11-05T08:33:24.553 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.553 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:24.554 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.554 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.554 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.554 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.16s5 66/45 ECSubWrite(tid=115, reqid=osd.11.0:112, at_version=0'0, trim_to=0'0, roll_forward_to=66'34))", 2023-11-05T08:33:24.554 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.416614+0000", 2023-11-05T08:33:24.555 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4.8923254939999996, 2023-11-05T08:33:24.555 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0087502250000000004, 2023-11-05T08:33:24.555 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:24.555 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:24.555 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:24.556 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.556 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:24.556 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.416614+0000", 2023-11-05T08:33:24.556 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.556 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.556 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.557 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:24.557 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.416614+0000", 2023-11-05T08:33:24.557 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.557 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.557 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.558 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:24.558 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.416616+0000", 2023-11-05T08:33:24.558 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7e-06 2023-11-05T08:33:24.558 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.558 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.559 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:24.559 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.416644+0000", 2023-11-05T08:33:24.559 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8549000000000001e-05 2023-11-05T08:33:24.559 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.559 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.560 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:24.560 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.416678+0000", 2023-11-05T08:33:24.560 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3266000000000003e-05 2023-11-05T08:33:24.560 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.560 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.561 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:24.561 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.416701+0000", 2023-11-05T08:33:24.561 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3632999999999999e-05 2023-11-05T08:33:24.561 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.561 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.561 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:24.562 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.416913+0000", 2023-11-05T08:33:24.562 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00021194700000000001 2023-11-05T08:33:24.562 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.562 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.562 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:24.563 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.416946+0000", 2023-11-05T08:33:24.563 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2869e-05 2023-11-05T08:33:24.563 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.563 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.563 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:24.564 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.425257+0000", 2023-11-05T08:33:24.564 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0083110960000000005 2023-11-05T08:33:24.564 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.564 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.564 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:24.565 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.425364+0000", 2023-11-05T08:33:24.565 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010716499999999999 2023-11-05T08:33:24.565 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.565 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:24.565 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.565 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.566 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.566 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.18s4 66/45 ECSubWrite(tid=250, reqid=osd.13.0:250, at_version=66'38, trim_to=0'0, roll_forward_to=65'36))", 2023-11-05T08:33:24.566 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.418433+0000", 2023-11-05T08:33:24.566 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4.8905067830000002, 2023-11-05T08:33:24.566 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0069943790000000002, 2023-11-05T08:33:24.567 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:24.567 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:24.567 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:24.567 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.567 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:24.568 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.418433+0000", 2023-11-05T08:33:24.568 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.568 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.568 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.568 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:24.568 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.418433+0000", 2023-11-05T08:33:24.569 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:24.569 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.569 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.569 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:24.569 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.418433+0000", 2023-11-05T08:33:24.570 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.6300000000000005e-07 2023-11-05T08:33:24.570 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.570 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.570 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:24.570 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.418511+0000", 2023-11-05T08:33:24.571 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.7712000000000004e-05 2023-11-05T08:33:24.571 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.571 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.571 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:24.571 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.418542+0000", 2023-11-05T08:33:24.571 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0436000000000001e-05 2023-11-05T08:33:24.572 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.572 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.572 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:24.572 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.418564+0000", 2023-11-05T08:33:24.572 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.2175999999999999e-05 2023-11-05T08:33:24.573 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.573 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.573 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:24.573 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.418777+0000", 2023-11-05T08:33:24.573 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00021367700000000001 2023-11-05T08:33:24.574 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.574 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.574 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:24.574 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.418823+0000", 2023-11-05T08:33:24.574 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5791999999999998e-05 2023-11-05T08:33:24.575 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.575 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.575 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:24.575 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.425287+0000", 2023-11-05T08:33:24.575 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0064638330000000004 2023-11-05T08:33:24.576 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:24.576 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.576 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:24.576 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.425427+0000", 2023-11-05T08:33:24.576 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014009 2023-11-05T08:33:24.577 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.577 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:24.577 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.577 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:24.577 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:24.577 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:33:24.578 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-05T08:33:24.579 INFO:tasks.rados.rados.0.smithi090.stdout:228: done (2 left) 2023-11-05T08:33:24.579 INFO:tasks.rados.rados.0.smithi090.stdout:229: done (1 left) 2023-11-05T08:33:24.579 INFO:tasks.rados.rados.0.smithi090.stdout:230: snap_create 2023-11-05T08:33:24.580 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2023-11-05T08:33:24.681 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:33:24.678+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:33:24.714 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:33:24.715 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2023-11-05T08:33:24.715 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 0 2023-11-05T08:33:24.715 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:33:24.731 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2023-11-05T08:33:24.780 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-05T08:33:24.850 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:33:24.851 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2023-11-05T08:33:24.851 INFO:teuthology.orchestra.run.smithi053.stdout: "complaint_time": 30, 2023-11-05T08:33:24.851 INFO:teuthology.orchestra.run.smithi053.stdout: "num_blocked_ops": 0 2023-11-05T08:33:24.851 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:33:24.866 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2023-11-05T08:33:24.883 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:24.879+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:24.983 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~29s 2023-11-05T08:33:24.984 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:33:24.982+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:33:24.998 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:33:24.998 INFO:teuthology.orchestra.run.smithi053.stdout: "size": 20, 2023-11-05T08:33:24.998 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 600, 2023-11-05T08:33:24.999 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2023-11-05T08:33:24.999 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:24.999 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as3 41/33 ECSubWrite(tid=35, reqid=osd.7.0:35, at_version=41'12, trim_to=0'0, roll_forward_to=40'10))", 2023-11-05T08:33:24.999 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.129446+0000", 2023-11-05T08:33:24.999 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 30.865235495, 2023-11-05T08:33:25.000 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010453017, 2023-11-05T08:33:25.000 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.000 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:25.000 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.000 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.001 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.001 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.129446+0000", 2023-11-05T08:33:25.001 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.001 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.001 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.002 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.002 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.129446+0000", 2023-11-05T08:33:25.002 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.002 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.002 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.002 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.003 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.129446+0000", 2023-11-05T08:33:25.003 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.5000000000000002e-07 2023-11-05T08:33:25.003 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.003 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.003 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.004 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.129503+0000", 2023-11-05T08:33:25.004 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6286000000000003e-05 2023-11-05T08:33:25.004 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.004 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.004 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.004 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.129536+0000", 2023-11-05T08:33:25.005 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3036e-05 2023-11-05T08:33:25.005 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.005 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.005 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.005 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.129559+0000", 2023-11-05T08:33:25.006 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3091999999999998e-05 2023-11-05T08:33:25.006 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.006 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.006 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.006 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.129756+0000", 2023-11-05T08:33:25.007 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000196918 2023-11-05T08:33:25.007 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.007 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.007 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:25.007 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.129782+0000", 2023-11-05T08:33:25.008 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6013e-05 2023-11-05T08:33:25.008 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.008 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.008 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:25.008 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.139822+0000", 2023-11-05T08:33:25.009 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010040261 2023-11-05T08:33:25.009 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.009 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.009 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:25.009 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.139899+0000", 2023-11-05T08:33:25.010 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.6661000000000003e-05 2023-11-05T08:33:25.010 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.010 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.010 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.010 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.010 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.011 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as3 41/33 ECSubWrite(tid=36, reqid=osd.7.0:36, at_version=41'14, trim_to=0'0, roll_forward_to=40'10))", 2023-11-05T08:33:25.011 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.131534+0000", 2023-11-05T08:33:25.011 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 30.863147071, 2023-11-05T08:33:25.011 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.016827443000000001, 2023-11-05T08:33:25.011 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.012 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:25.012 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.012 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.012 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.012 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.131534+0000", 2023-11-05T08:33:25.013 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.013 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.013 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.013 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.013 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.131534+0000", 2023-11-05T08:33:25.014 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.014 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.014 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.014 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.014 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.131536+0000", 2023-11-05T08:33:25.015 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3769999999999999e-06 2023-11-05T08:33:25.015 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.015 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.015 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.015 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.131586+0000", 2023-11-05T08:33:25.016 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.0778000000000001e-05 2023-11-05T08:33:25.016 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.016 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.016 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.016 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.131619+0000", 2023-11-05T08:33:25.016 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2793000000000003e-05 2023-11-05T08:33:25.017 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.017 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.017 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.017 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.131642+0000", 2023-11-05T08:33:25.017 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3241999999999999e-05 2023-11-05T08:33:25.018 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.018 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.018 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.018 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.131806+0000", 2023-11-05T08:33:25.018 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000163843 2023-11-05T08:33:25.019 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.019 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.019 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:25.019 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.131832+0000", 2023-11-05T08:33:25.019 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5689000000000001e-05 2023-11-05T08:33:25.020 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.020 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.020 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:25.020 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148268+0000", 2023-11-05T08:33:25.020 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.016435924000000001 2023-11-05T08:33:25.021 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.021 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.021 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:25.021 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148362+0000", 2023-11-05T08:33:25.021 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.3796999999999997e-05 2023-11-05T08:33:25.021 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.022 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.022 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.022 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.022 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.022 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es1 41/30 ECSubWrite(tid=41, reqid=osd.2.0:40, at_version=0'0, trim_to=0'0, roll_forward_to=41'27))", 2023-11-05T08:33:25.023 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.148780+0000", 2023-11-05T08:33:25.023 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 30.845901466000001, 2023-11-05T08:33:25.023 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0090092869999999995, 2023-11-05T08:33:25.023 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.023 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:25.024 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.024 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.024 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.024 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148780+0000", 2023-11-05T08:33:25.024 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.025 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.025 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.025 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.025 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148780+0000", 2023-11-05T08:33:25.025 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.025 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.026 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.026 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.026 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148789+0000", 2023-11-05T08:33:25.026 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.2029999999999998e-06 2023-11-05T08:33:25.026 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.027 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.027 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.027 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148816+0000", 2023-11-05T08:33:25.027 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7291999999999999e-05 2023-11-05T08:33:25.027 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.028 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.028 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.028 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148847+0000", 2023-11-05T08:33:25.028 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1118999999999998e-05 2023-11-05T08:33:25.028 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.029 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.029 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.029 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148871+0000", 2023-11-05T08:33:25.029 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3448999999999999e-05 2023-11-05T08:33:25.029 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.029 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.030 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.030 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.149090+0000", 2023-11-05T08:33:25.030 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00021928799999999999 2023-11-05T08:33:25.030 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.030 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.031 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:25.031 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.149116+0000", 2023-11-05T08:33:25.031 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6239000000000001e-05 2023-11-05T08:33:25.031 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.031 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.031 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:25.032 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.157693+0000", 2023-11-05T08:33:25.032 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0085768340000000002 2023-11-05T08:33:25.032 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.032 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.032 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:25.032 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.157789+0000", 2023-11-05T08:33:25.033 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.5863e-05 2023-11-05T08:33:25.033 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.033 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.033 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.033 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.033 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.034 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.11s2 41/40 ECSubWrite(tid=65, reqid=client.4691.0:287, at_version=41'13, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:25.034 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.173568+0000", 2023-11-05T08:33:25.034 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 30.821113368999999, 2023-11-05T08:33:25.034 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0078777859999999995, 2023-11-05T08:33:25.034 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.035 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:25.035 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.035 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.035 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.035 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.173568+0000", 2023-11-05T08:33:25.036 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.036 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.036 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.036 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.036 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.173568+0000", 2023-11-05T08:33:25.037 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.037 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.037 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.037 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.037 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.173570+0000", 2023-11-05T08:33:25.037 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9460000000000001e-06 2023-11-05T08:33:25.038 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.038 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.038 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.038 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.174495+0000", 2023-11-05T08:33:25.038 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00092541300000000004 2023-11-05T08:33:25.039 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.039 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.039 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.039 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.174540+0000", 2023-11-05T08:33:25.039 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5170999999999998e-05 2023-11-05T08:33:25.039 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.040 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.040 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.040 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.174564+0000", 2023-11-05T08:33:25.040 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3159e-05 2023-11-05T08:33:25.040 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.041 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.041 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.041 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.175041+0000", 2023-11-05T08:33:25.041 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00047766399999999999 2023-11-05T08:33:25.041 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.042 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.042 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:25.042 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.175062+0000", 2023-11-05T08:33:25.042 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1064999999999999e-05 2023-11-05T08:33:25.042 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.043 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.043 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:25.043 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.181334+0000", 2023-11-05T08:33:25.043 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0062722159999999997 2023-11-05T08:33:25.043 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.043 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.044 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:25.044 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.181446+0000", 2023-11-05T08:33:25.044 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000111152 2023-11-05T08:33:25.044 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.044 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.045 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.045 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.045 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.045 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as3 42/33 ECSubWrite(tid=38, reqid=osd.7.0:38, at_version=42'16, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:33:25.045 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:55.126927+0000", 2023-11-05T08:33:25.046 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 29.867754483999999, 2023-11-05T08:33:25.046 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.013340393000000001, 2023-11-05T08:33:25.046 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.046 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:25.046 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.046 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.047 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.047 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.126927+0000", 2023-11-05T08:33:25.047 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.047 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.047 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.048 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.048 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.126927+0000", 2023-11-05T08:33:25.048 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.048 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.048 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.049 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.049 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.126927+0000", 2023-11-05T08:33:25.049 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7e-07 2023-11-05T08:33:25.049 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.049 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.050 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.050 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.127040+0000", 2023-11-05T08:33:25.050 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00011244299999999999 2023-11-05T08:33:25.050 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.050 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.051 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.051 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.127069+0000", 2023-11-05T08:33:25.051 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9255999999999999e-05 2023-11-05T08:33:25.051 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.051 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.051 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.052 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.127093+0000", 2023-11-05T08:33:25.052 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3839e-05 2023-11-05T08:33:25.052 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.052 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.052 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.052 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.129369+0000", 2023-11-05T08:33:25.053 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0022765250000000002 2023-11-05T08:33:25.053 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.053 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.053 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:25.053 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.129395+0000", 2023-11-05T08:33:25.054 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5755999999999999e-05 2023-11-05T08:33:25.054 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.054 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.054 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:25.054 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.140178+0000", 2023-11-05T08:33:25.055 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010782523 2023-11-05T08:33:25.055 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.055 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.055 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:25.055 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.140267+0000", 2023-11-05T08:33:25.055 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9580999999999995e-05 2023-11-05T08:33:25.056 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.056 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.056 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.056 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.056 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.057 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as3 42/33 ECSubWrite(tid=39, reqid=osd.7.0:39, at_version=42'18, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:33:25.057 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:55.128574+0000", 2023-11-05T08:33:25.057 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 29.866107251999999, 2023-11-05T08:33:25.057 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.013189111, 2023-11-05T08:33:25.057 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.058 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:25.058 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.058 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.058 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.058 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.128574+0000", 2023-11-05T08:33:25.059 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.059 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.059 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.059 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.059 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.128574+0000", 2023-11-05T08:33:25.060 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.060 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.060 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.060 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.060 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.128575+0000", 2023-11-05T08:33:25.060 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1370000000000001e-06 2023-11-05T08:33:25.061 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.061 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.061 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.061 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.128665+0000", 2023-11-05T08:33:25.061 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9571e-05 2023-11-05T08:33:25.062 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.062 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.062 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.062 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.128693+0000", 2023-11-05T08:33:25.062 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8822e-05 2023-11-05T08:33:25.063 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.063 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.063 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.063 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.128737+0000", 2023-11-05T08:33:25.063 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3469e-05 2023-11-05T08:33:25.064 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.064 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.064 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.064 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.130720+0000", 2023-11-05T08:33:25.064 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0019833229999999999 2023-11-05T08:33:25.064 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.065 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.065 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:25.065 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.130746+0000", 2023-11-05T08:33:25.065 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5326000000000001e-05 2023-11-05T08:33:25.065 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.066 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.066 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:25.066 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.141675+0000", 2023-11-05T08:33:25.066 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010929589 2023-11-05T08:33:25.066 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.067 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.067 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:25.067 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.141763+0000", 2023-11-05T08:33:25.067 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.7874000000000006e-05 2023-11-05T08:33:25.067 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.068 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.068 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.068 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.068 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.068 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es1 42/30 ECSubWrite(tid=43, reqid=osd.2.0:43, at_version=42'29, trim_to=0'0, roll_forward_to=41'27))", 2023-11-05T08:33:25.069 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:55.134325+0000", 2023-11-05T08:33:25.069 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 29.86035605, 2023-11-05T08:33:25.069 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.008930136, 2023-11-05T08:33:25.069 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.069 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:25.069 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.070 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.070 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.070 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134325+0000", 2023-11-05T08:33:25.070 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.070 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.071 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.071 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.071 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134325+0000", 2023-11-05T08:33:25.071 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.071 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.072 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.072 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.072 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134327+0000", 2023-11-05T08:33:25.072 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6899999999999999e-06 2023-11-05T08:33:25.072 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.073 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.073 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.073 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134380+0000", 2023-11-05T08:33:25.073 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3458000000000002e-05 2023-11-05T08:33:25.073 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.073 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.074 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.074 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134410+0000", 2023-11-05T08:33:25.074 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9405999999999999e-05 2023-11-05T08:33:25.074 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.074 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.075 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.075 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134433+0000", 2023-11-05T08:33:25.075 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3745999999999999e-05 2023-11-05T08:33:25.075 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.075 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.076 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.076 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134900+0000", 2023-11-05T08:33:25.076 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00046645999999999998 2023-11-05T08:33:25.076 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.076 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.077 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:25.077 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134925+0000", 2023-11-05T08:33:25.077 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5196000000000001e-05 2023-11-05T08:33:25.077 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.077 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.077 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:25.078 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.143157+0000", 2023-11-05T08:33:25.078 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0082321760000000008 2023-11-05T08:33:25.078 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.078 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.078 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:25.079 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.143255+0000", 2023-11-05T08:33:25.079 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.8004000000000006e-05 2023-11-05T08:33:25.079 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.079 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.079 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.080 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.080 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.080 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.16s4 51/45 ECSubWrite(tid=58, reqid=osd.11.0:58, at_version=51'19, trim_to=0'0, roll_forward_to=49'17))", 2023-11-05T08:33:25.080 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:04.226129+0000", 2023-11-05T08:33:25.080 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 20.768552167999999, 2023-11-05T08:33:25.081 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0055450669999999999, 2023-11-05T08:33:25.081 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.081 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:25.081 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.081 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.081 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.082 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.226129+0000", 2023-11-05T08:33:25.082 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.082 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.082 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.082 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.083 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.226129+0000", 2023-11-05T08:33:25.083 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.083 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.083 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.083 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.084 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.226130+0000", 2023-11-05T08:33:25.084 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0299999999999999e-07 2023-11-05T08:33:25.084 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.084 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.084 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.085 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.226185+0000", 2023-11-05T08:33:25.085 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5145999999999997e-05 2023-11-05T08:33:25.085 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.085 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.085 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.086 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.226220+0000", 2023-11-05T08:33:25.086 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5441999999999997e-05 2023-11-05T08:33:25.086 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.086 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.086 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.086 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.226243+0000", 2023-11-05T08:33:25.087 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.2379e-05 2023-11-05T08:33:25.087 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.087 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.087 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.087 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.226413+0000", 2023-11-05T08:33:25.088 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00016998 2023-11-05T08:33:25.088 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.088 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.088 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:25.088 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.226437+0000", 2023-11-05T08:33:25.089 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3909000000000001e-05 2023-11-05T08:33:25.089 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.089 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.089 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:25.089 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.231571+0000", 2023-11-05T08:33:25.089 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0051343170000000002 2023-11-05T08:33:25.090 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.090 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.090 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:25.090 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.231674+0000", 2023-11-05T08:33:25.090 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000103291 2023-11-05T08:33:25.091 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.091 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.091 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.091 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.091 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.092 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as3 52/45 ECSubWrite(tid=45, reqid=client.4691.0:349, at_version=52'21, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:25.092 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:05.263635+0000", 2023-11-05T08:33:25.092 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 19.731045835, 2023-11-05T08:33:25.092 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0081552580000000003, 2023-11-05T08:33:25.092 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.093 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:25.093 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.093 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.093 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.093 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.263635+0000", 2023-11-05T08:33:25.094 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.094 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.094 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.094 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.094 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.263635+0000", 2023-11-05T08:33:25.095 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.095 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.095 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.095 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.095 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.263637+0000", 2023-11-05T08:33:25.095 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.897e-06 2023-11-05T08:33:25.096 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.096 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.096 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.096 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.264062+0000", 2023-11-05T08:33:25.096 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00042438099999999999 2023-11-05T08:33:25.097 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.097 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.097 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.097 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.264102+0000", 2023-11-05T08:33:25.097 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.0058999999999997e-05 2023-11-05T08:33:25.098 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.098 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.098 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.098 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.264138+0000", 2023-11-05T08:33:25.098 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.6208999999999998e-05 2023-11-05T08:33:25.099 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.099 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.099 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.099 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.264324+0000", 2023-11-05T08:33:25.099 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00018639199999999999 2023-11-05T08:33:25.100 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.100 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.100 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:25.100 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.264351+0000", 2023-11-05T08:33:25.100 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6673e-05 2023-11-05T08:33:25.100 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.101 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.101 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:25.101 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.271652+0000", 2023-11-05T08:33:25.101 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0073011170000000002 2023-11-05T08:33:25.101 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.102 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.102 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:25.102 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.271791+0000", 2023-11-05T08:33:25.102 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013852999999999999 2023-11-05T08:33:25.102 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.103 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.103 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.103 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.103 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.103 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as3 61/45 ECSubWrite(tid=57, reqid=client.4691.0:405, at_version=61'23, trim_to=0'0, roll_forward_to=52'21))", 2023-11-05T08:33:25.103 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:14.365006+0000", 2023-11-05T08:33:25.104 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 10.629675461, 2023-11-05T08:33:25.104 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0064071809999999996, 2023-11-05T08:33:25.104 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.104 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:25.104 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.105 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.105 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.105 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.365006+0000", 2023-11-05T08:33:25.105 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.105 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.106 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.106 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.106 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.365006+0000", 2023-11-05T08:33:25.106 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.106 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.107 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.107 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.107 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.365007+0000", 2023-11-05T08:33:25.107 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.72e-06 2023-11-05T08:33:25.107 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.107 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.108 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.108 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.365501+0000", 2023-11-05T08:33:25.108 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00049378099999999999 2023-11-05T08:33:25.108 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.108 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.109 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.109 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.365533+0000", 2023-11-05T08:33:25.109 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1835999999999998e-05 2023-11-05T08:33:25.109 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.109 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.109 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.110 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.365550+0000", 2023-11-05T08:33:25.110 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6813e-05 2023-11-05T08:33:25.110 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.110 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.110 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.111 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.365689+0000", 2023-11-05T08:33:25.111 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000138784 2023-11-05T08:33:25.111 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.111 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.111 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:25.112 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.365722+0000", 2023-11-05T08:33:25.112 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3509000000000001e-05 2023-11-05T08:33:25.112 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.112 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.112 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:25.113 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.371344+0000", 2023-11-05T08:33:25.113 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.005621366 2023-11-05T08:33:25.113 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.113 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.113 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:25.114 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.371413+0000", 2023-11-05T08:33:25.114 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9371999999999999e-05 2023-11-05T08:33:25.114 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.114 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.114 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.115 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.115 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.115 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as3 61/45 ECSubWrite(tid=58, reqid=client.4691.0:406, at_version=61'24, trim_to=0'0, roll_forward_to=52'21))", 2023-11-05T08:33:25.115 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:14.371923+0000", 2023-11-05T08:33:25.115 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 10.62275863, 2023-11-05T08:33:25.116 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0074408499999999997, 2023-11-05T08:33:25.116 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.116 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:25.116 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.116 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.117 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.117 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.371923+0000", 2023-11-05T08:33:25.117 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.117 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.117 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.117 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.118 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.371923+0000", 2023-11-05T08:33:25.118 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.118 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.118 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.119 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.119 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.371925+0000", 2023-11-05T08:33:25.119 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.153e-06 2023-11-05T08:33:25.119 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.119 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.119 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.120 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.372289+0000", 2023-11-05T08:33:25.120 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00036441300000000001 2023-11-05T08:33:25.120 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.120 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.120 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.120 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.372329+0000", 2023-11-05T08:33:25.121 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.9906e-05 2023-11-05T08:33:25.121 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.121 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.121 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.121 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.372353+0000", 2023-11-05T08:33:25.122 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3856e-05 2023-11-05T08:33:25.122 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.122 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.122 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.122 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.372522+0000", 2023-11-05T08:33:25.123 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00016944 2023-11-05T08:33:25.123 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.123 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.123 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:25.123 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.372551+0000", 2023-11-05T08:33:25.123 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8856e-05 2023-11-05T08:33:25.124 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.124 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.124 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:25.124 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.379263+0000", 2023-11-05T08:33:25.124 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0067113049999999999 2023-11-05T08:33:25.125 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.125 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.125 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:25.125 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.379363+0000", 2023-11-05T08:33:25.125 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000100921 2023-11-05T08:33:25.125 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.126 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.126 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.126 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.126 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.126 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.bs6 66/37 ECSubWrite(tid=108, reqid=client.4691.0:496, at_version=66'23, trim_to=0'0, roll_forward_to=65'21))", 2023-11-05T08:33:25.127 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.399096+0000", 2023-11-05T08:33:25.127 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 5.5955852659999996, 2023-11-05T08:33:25.127 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010759829, 2023-11-05T08:33:25.127 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.127 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:25.128 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.128 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.128 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.128 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.399096+0000", 2023-11-05T08:33:25.128 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.129 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.129 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.129 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.129 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.399096+0000", 2023-11-05T08:33:25.129 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.130 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.130 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.130 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.130 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.399098+0000", 2023-11-05T08:33:25.130 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.0899999999999999e-06 2023-11-05T08:33:25.130 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.131 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.131 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.131 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.399777+0000", 2023-11-05T08:33:25.131 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000678468 2023-11-05T08:33:25.131 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.132 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.132 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.132 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.399810+0000", 2023-11-05T08:33:25.132 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3646000000000003e-05 2023-11-05T08:33:25.132 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.133 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.133 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.133 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.399834+0000", 2023-11-05T08:33:25.133 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3782999999999999e-05 2023-11-05T08:33:25.133 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.133 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.134 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.134 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.399991+0000", 2023-11-05T08:33:25.134 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000157317 2023-11-05T08:33:25.134 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.134 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.135 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:25.135 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.400017+0000", 2023-11-05T08:33:25.135 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5996999999999999e-05 2023-11-05T08:33:25.135 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.135 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.136 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:25.136 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.409770+0000", 2023-11-05T08:33:25.136 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0097524240000000009 2023-11-05T08:33:25.136 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.136 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.136 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:25.137 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.409856+0000", 2023-11-05T08:33:25.137 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.6104000000000001e-05 2023-11-05T08:33:25.137 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.137 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.137 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.138 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.138 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.138 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.16s4 66/45 ECSubWrite(tid=112, reqid=osd.11.0:112, at_version=66'34, trim_to=0'0, roll_forward_to=65'32))", 2023-11-05T08:33:25.138 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.401373+0000", 2023-11-05T08:33:25.138 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 5.5933081199999997, 2023-11-05T08:33:25.139 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0064814720000000003, 2023-11-05T08:33:25.139 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.139 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:25.139 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.139 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.140 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.140 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.401373+0000", 2023-11-05T08:33:25.140 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.140 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.140 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.140 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.141 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.401373+0000", 2023-11-05T08:33:25.141 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.141 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.141 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.141 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.142 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.401374+0000", 2023-11-05T08:33:25.142 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9699999999999996e-07 2023-11-05T08:33:25.142 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.142 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.142 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.143 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.401420+0000", 2023-11-05T08:33:25.143 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.6683e-05 2023-11-05T08:33:25.143 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.143 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.143 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.143 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.401450+0000", 2023-11-05T08:33:25.144 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9788999999999999e-05 2023-11-05T08:33:25.144 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.144 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.144 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.145 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.401473+0000", 2023-11-05T08:33:25.145 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.2682999999999999e-05 2023-11-05T08:33:25.145 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.145 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.145 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.146 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.401644+0000", 2023-11-05T08:33:25.146 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00017150999999999999 2023-11-05T08:33:25.146 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.146 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.146 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:25.146 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.401674+0000", 2023-11-05T08:33:25.147 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9723e-05 2023-11-05T08:33:25.147 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.147 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.147 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:25.147 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.407763+0000", 2023-11-05T08:33:25.148 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0060884950000000002 2023-11-05T08:33:25.148 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.148 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.148 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:25.148 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.407855+0000", 2023-11-05T08:33:25.149 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1991999999999996e-05 2023-11-05T08:33:25.149 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.149 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.149 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.149 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.149 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.150 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.bs6 66/37 ECSubWrite(tid=109, reqid=client.4691.0:497, at_version=66'24, trim_to=0'0, roll_forward_to=65'21))", 2023-11-05T08:33:25.150 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.404475+0000", 2023-11-05T08:33:25.150 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 5.5902060149999997, 2023-11-05T08:33:25.150 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.006663576, 2023-11-05T08:33:25.150 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.151 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:25.151 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.151 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.151 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.151 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.404475+0000", 2023-11-05T08:33:25.151 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.152 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.152 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.152 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.152 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.404475+0000", 2023-11-05T08:33:25.152 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.153 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.153 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.153 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.153 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.404477+0000", 2023-11-05T08:33:25.153 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.0030000000000001e-06 2023-11-05T08:33:25.153 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.154 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.154 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.154 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.404894+0000", 2023-11-05T08:33:25.154 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000416697 2023-11-05T08:33:25.154 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.155 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.155 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.155 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.404928+0000", 2023-11-05T08:33:25.155 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4329000000000003e-05 2023-11-05T08:33:25.155 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.156 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.156 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.156 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.404989+0000", 2023-11-05T08:33:25.156 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0424999999999998e-05 2023-11-05T08:33:25.156 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.156 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.157 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.157 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.405674+0000", 2023-11-05T08:33:25.157 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000685678 2023-11-05T08:33:25.157 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.157 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.157 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:25.158 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.405700+0000", 2023-11-05T08:33:25.158 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5313e-05 2023-11-05T08:33:25.158 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.158 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.158 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:25.159 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.411060+0000", 2023-11-05T08:33:25.159 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0053600990000000001 2023-11-05T08:33:25.159 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.159 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.159 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:25.160 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.411139+0000", 2023-11-05T08:33:25.160 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.9031999999999998e-05 2023-11-05T08:33:25.160 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.160 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.160 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.161 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.161 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.161 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.bs6 66/37 ECSubWrite(tid=110, reqid=client.4691.0:498, at_version=66'25, trim_to=0'0, roll_forward_to=65'21))", 2023-11-05T08:33:25.161 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.411793+0000", 2023-11-05T08:33:25.161 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 5.5828883310000004, 2023-11-05T08:33:25.162 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0078269380000000003, 2023-11-05T08:33:25.162 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.162 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:25.162 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.162 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.162 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.163 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.411793+0000", 2023-11-05T08:33:25.163 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.163 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.163 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.163 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.164 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.411793+0000", 2023-11-05T08:33:25.164 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.164 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.164 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.164 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.164 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.411795+0000", 2023-11-05T08:33:25.165 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.827e-06 2023-11-05T08:33:25.165 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.165 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.165 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.165 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.411946+0000", 2023-11-05T08:33:25.166 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00015118999999999999 2023-11-05T08:33:25.166 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.166 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.166 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.166 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.411995+0000", 2023-11-05T08:33:25.167 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9175999999999998e-05 2023-11-05T08:33:25.167 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.167 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.167 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.167 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.412024+0000", 2023-11-05T08:33:25.168 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8739999999999999e-05 2023-11-05T08:33:25.168 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.168 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.168 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.168 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.412505+0000", 2023-11-05T08:33:25.168 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00048081399999999998 2023-11-05T08:33:25.169 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.169 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.169 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:25.169 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.412530+0000", 2023-11-05T08:33:25.169 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5374000000000001e-05 2023-11-05T08:33:25.170 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.170 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.170 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:25.170 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.419518+0000", 2023-11-05T08:33:25.170 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0069876119999999998 2023-11-05T08:33:25.171 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.171 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.171 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:25.171 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.419620+0000", 2023-11-05T08:33:25.171 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000102205 2023-11-05T08:33:25.172 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.172 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.172 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.172 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.172 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.173 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.bs6 66/37 ECSubWrite(tid=111, reqid=client.4691.0:499, at_version=66'26, trim_to=0'0, roll_forward_to=65'21))", 2023-11-05T08:33:25.173 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.412085+0000", 2023-11-05T08:33:25.173 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 5.5825965389999999, 2023-11-05T08:33:25.173 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0076042080000000003, 2023-11-05T08:33:25.173 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.174 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:25.174 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.174 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.174 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.174 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.412085+0000", 2023-11-05T08:33:25.175 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.175 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.175 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.175 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.175 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.412085+0000", 2023-11-05T08:33:25.176 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.176 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.176 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.176 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.176 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.412086+0000", 2023-11-05T08:33:25.176 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0359999999999999e-06 2023-11-05T08:33:25.177 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.177 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.177 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.177 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.412133+0000", 2023-11-05T08:33:25.177 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7076000000000001e-05 2023-11-05T08:33:25.178 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.178 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.178 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.178 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.412161+0000", 2023-11-05T08:33:25.178 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8518999999999999e-05 2023-11-05T08:33:25.179 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.179 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.179 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.179 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.412181+0000", 2023-11-05T08:33:25.179 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9347e-05 2023-11-05T08:33:25.179 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.180 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.180 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.180 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.413827+0000", 2023-11-05T08:33:25.180 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001646034 2023-11-05T08:33:25.180 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.181 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.181 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:25.181 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.413851+0000", 2023-11-05T08:33:25.181 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4332999999999999e-05 2023-11-05T08:33:25.181 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.182 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.182 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:25.182 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.419630+0000", 2023-11-05T08:33:25.182 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0057790309999999996 2023-11-05T08:33:25.182 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.183 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.183 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:25.183 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.419689+0000", 2023-11-05T08:33:25.183 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8832000000000002e-05 2023-11-05T08:33:25.183 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.184 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.184 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.184 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.184 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.184 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es1 66/30 ECSubWrite(tid=87, reqid=osd.2.0:87, at_version=66'65, trim_to=0'0, roll_forward_to=65'63))", 2023-11-05T08:33:25.184 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.413863+0000", 2023-11-05T08:33:25.185 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 5.5808183280000003, 2023-11-05T08:33:25.185 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0091784380000000006, 2023-11-05T08:33:25.185 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.185 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:25.185 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.186 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.186 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.186 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.413863+0000", 2023-11-05T08:33:25.186 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.187 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.187 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.187 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.187 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.413863+0000", 2023-11-05T08:33:25.187 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.187 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.188 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.188 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.188 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.413864+0000", 2023-11-05T08:33:25.188 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.407e-06 2023-11-05T08:33:25.188 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.189 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.189 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.189 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.413917+0000", 2023-11-05T08:33:25.189 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2785999999999999e-05 2023-11-05T08:33:25.189 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.190 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.190 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.190 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.413947+0000", 2023-11-05T08:33:25.190 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9422999999999999e-05 2023-11-05T08:33:25.190 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.191 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.191 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.191 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.413969+0000", 2023-11-05T08:33:25.191 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.2582000000000002e-05 2023-11-05T08:33:25.191 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.191 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.192 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.192 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.414151+0000", 2023-11-05T08:33:25.192 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00018211099999999999 2023-11-05T08:33:25.192 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.192 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.193 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:25.193 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.414182+0000", 2023-11-05T08:33:25.193 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1072000000000003e-05 2023-11-05T08:33:25.193 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.193 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.193 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:25.194 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.422947+0000", 2023-11-05T08:33:25.194 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0087647520000000007 2023-11-05T08:33:25.194 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.194 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.194 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:25.195 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.423041+0000", 2023-11-05T08:33:25.195 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.4304999999999999e-05 2023-11-05T08:33:25.195 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.195 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.195 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.196 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.196 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.196 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.1s2 66/40 ECSubWrite(tid=112, reqid=osd.4.0:107, at_version=0'0, trim_to=0'0, roll_forward_to=66'26))", 2023-11-05T08:33:25.196 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.415009+0000", 2023-11-05T08:33:25.196 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 5.5796724409999996, 2023-11-05T08:33:25.197 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0079745690000000008, 2023-11-05T08:33:25.197 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.197 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:25.197 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.197 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.198 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.198 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.415009+0000", 2023-11-05T08:33:25.198 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.198 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.198 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.198 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.199 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.415009+0000", 2023-11-05T08:33:25.199 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.199 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.199 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.199 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.200 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.415010+0000", 2023-11-05T08:33:25.200 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.68e-06 2023-11-05T08:33:25.200 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.200 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.200 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.201 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.415039+0000", 2023-11-05T08:33:25.201 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8076e-05 2023-11-05T08:33:25.201 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.201 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.201 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.202 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.415262+0000", 2023-11-05T08:33:25.202 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000223019 2023-11-05T08:33:25.202 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.202 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.202 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.203 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.415286+0000", 2023-11-05T08:33:25.203 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4162999999999999e-05 2023-11-05T08:33:25.203 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.203 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.203 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.204 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.415463+0000", 2023-11-05T08:33:25.204 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00017772400000000001 2023-11-05T08:33:25.204 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.204 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.204 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:25.204 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.415494+0000", 2023-11-05T08:33:25.205 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1025999999999997e-05 2023-11-05T08:33:25.205 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.205 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.205 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:25.205 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.422878+0000", 2023-11-05T08:33:25.206 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0073837720000000003 2023-11-05T08:33:25.206 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.206 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.206 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:25.206 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.422983+0000", 2023-11-05T08:33:25.207 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000105109 2023-11-05T08:33:25.207 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.207 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.207 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.207 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.208 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.208 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es1 66/30 ECSubWrite(tid=88, reqid=osd.2.0:88, at_version=66'67, trim_to=0'0, roll_forward_to=65'63))", 2023-11-05T08:33:25.208 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.416186+0000", 2023-11-05T08:33:25.208 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 5.5784956990000003, 2023-11-05T08:33:25.208 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.008119823, 2023-11-05T08:33:25.209 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.209 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:25.209 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.209 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.209 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.210 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.416186+0000", 2023-11-05T08:33:25.210 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.210 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.210 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.210 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.211 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.416186+0000", 2023-11-05T08:33:25.211 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.211 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.211 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.211 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.212 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.416187+0000", 2023-11-05T08:33:25.212 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.77e-06 2023-11-05T08:33:25.212 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.212 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.212 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.212 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.416239+0000", 2023-11-05T08:33:25.213 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1229e-05 2023-11-05T08:33:25.213 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.213 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.213 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.213 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.416297+0000", 2023-11-05T08:33:25.213 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8856e-05 2023-11-05T08:33:25.214 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.214 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.214 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.214 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.416324+0000", 2023-11-05T08:33:25.214 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6202000000000001e-05 2023-11-05T08:33:25.215 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.215 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.215 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.215 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.416570+0000", 2023-11-05T08:33:25.215 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00024649999999999997 2023-11-05T08:33:25.215 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.216 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.216 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:25.216 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.416602+0000", 2023-11-05T08:33:25.216 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1829000000000003e-05 2023-11-05T08:33:25.216 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.217 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.217 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:25.217 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.424215+0000", 2023-11-05T08:33:25.217 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0076131990000000002 2023-11-05T08:33:25.217 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.217 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.218 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:25.218 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.424305+0000", 2023-11-05T08:33:25.218 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.0237999999999999e-05 2023-11-05T08:33:25.218 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.218 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.219 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.219 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.219 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.219 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.16s4 66/45 ECSubWrite(tid=115, reqid=osd.11.0:112, at_version=0'0, trim_to=0'0, roll_forward_to=66'34))", 2023-11-05T08:33:25.219 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.416982+0000", 2023-11-05T08:33:25.219 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 5.5776992429999996, 2023-11-05T08:33:25.220 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0061497449999999999, 2023-11-05T08:33:25.220 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.220 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:25.220 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.220 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.220 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.221 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.416982+0000", 2023-11-05T08:33:25.221 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.221 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.221 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.221 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.221 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.416982+0000", 2023-11-05T08:33:25.222 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.222 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.222 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.222 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.222 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.416984+0000", 2023-11-05T08:33:25.223 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.8959999999999999e-06 2023-11-05T08:33:25.223 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.223 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.223 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.223 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.417010+0000", 2023-11-05T08:33:25.224 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6587000000000001e-05 2023-11-05T08:33:25.224 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.224 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.224 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.224 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.417042+0000", 2023-11-05T08:33:25.224 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1816000000000002e-05 2023-11-05T08:33:25.225 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.225 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.225 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.225 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.417065+0000", 2023-11-05T08:33:25.225 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.2753000000000001e-05 2023-11-05T08:33:25.226 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.226 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.226 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.226 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.417225+0000", 2023-11-05T08:33:25.226 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00015948700000000001 2023-11-05T08:33:25.227 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.227 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.227 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:25.227 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.417256+0000", 2023-11-05T08:33:25.228 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1563000000000003e-05 2023-11-05T08:33:25.228 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.228 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.228 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:25.228 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.423049+0000", 2023-11-05T08:33:25.228 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0057928370000000003 2023-11-05T08:33:25.229 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.229 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.229 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:25.229 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.423132+0000", 2023-11-05T08:33:25.229 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.2806000000000006e-05 2023-11-05T08:33:25.230 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.230 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.230 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.230 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.230 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.231 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:33:25.232 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2023-11-05T08:33:25.233 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:33:25.230+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:33:25.333 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:33:25.334+0000 7fdc5cd58640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:33:25.364 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:33:25.365 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2023-11-05T08:33:25.365 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 0 2023-11-05T08:33:25.365 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:33:25.380 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2023-11-05T08:33:25.426 INFO:tasks.rados.rados.0.smithi090.stdout:230: done (1 left) 2023-11-05T08:33:25.427 INFO:tasks.rados.rados.0.smithi090.stdout:231: snap_create 2023-11-05T08:33:25.434 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:33:25.428+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:33:25.435 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-11-05T08:33:25.507 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:33:25.508 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2023-11-05T08:33:25.508 INFO:teuthology.orchestra.run.smithi053.stdout: "complaint_time": 30, 2023-11-05T08:33:25.508 INFO:teuthology.orchestra.run.smithi053.stdout: "num_blocked_ops": 0 2023-11-05T08:33:25.508 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:33:25.521 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2023-11-05T08:33:25.535 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-05T08:33:25.637 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:33:25.634+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:33:25.691 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:33:25.692 INFO:teuthology.orchestra.run.smithi053.stdout: "size": 20, 2023-11-05T08:33:25.692 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 600, 2023-11-05T08:33:25.692 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2023-11-05T08:33:25.692 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.692 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:25.693 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:28.710073+0000", 2023-11-05T08:33:25.693 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 56.977312370999996, 2023-11-05T08:33:25.693 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.052834791999999998, 2023-11-05T08:33:25.693 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.693 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:25.693 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:25.694 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:25.694 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:25.694 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 1 2023-11-05T08:33:25.694 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.695 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.695 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.695 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.695 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710073+0000", 2023-11-05T08:33:25.695 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.696 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.696 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.696 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.696 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710073+0000", 2023-11-05T08:33:25.696 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.697 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.697 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.697 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.697 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710076+0000", 2023-11-05T08:33:25.697 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1729999999999999e-06 2023-11-05T08:33:25.697 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.698 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.698 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.698 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710108+0000", 2023-11-05T08:33:25.698 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1458000000000003e-05 2023-11-05T08:33:25.698 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.699 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.699 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.699 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710140+0000", 2023-11-05T08:33:25.699 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2532e-05 2023-11-05T08:33:25.699 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.700 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.700 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.700 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710165+0000", 2023-11-05T08:33:25.700 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5205e-05 2023-11-05T08:33:25.700 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.701 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.701 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.701 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710311+0000", 2023-11-05T08:33:25.701 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014532799999999999 2023-11-05T08:33:25.701 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.702 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.702 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2023-11-05T08:33:25.702 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710327+0000", 2023-11-05T08:33:25.702 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6059e-05 2023-11-05T08:33:25.702 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.703 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.703 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.703 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.759348+0000", 2023-11-05T08:33:25.703 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.049021481999999998 2023-11-05T08:33:25.703 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.703 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.704 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:25.704 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.759880+0000", 2023-11-05T08:33:25.704 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00053197400000000005 2023-11-05T08:33:25.704 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.704 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.705 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:25.705 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.760478+0000", 2023-11-05T08:33:25.705 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00059839300000000002 2023-11-05T08:33:25.705 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.705 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.706 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:25.706 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.761955+0000", 2023-11-05T08:33:25.706 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001476991 2023-11-05T08:33:25.706 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.706 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.707 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:25.707 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.762611+0000", 2023-11-05T08:33:25.707 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00065598700000000002 2023-11-05T08:33:25.707 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.707 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.708 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:25.708 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.762782+0000", 2023-11-05T08:33:25.708 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00017064000000000001 2023-11-05T08:33:25.708 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.708 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.709 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:25.709 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.762908+0000", 2023-11-05T08:33:25.709 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012557000000000001 2023-11-05T08:33:25.709 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.709 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.710 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.710 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.710 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.710 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:25.710 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.151546+0000", 2023-11-05T08:33:25.711 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 56.535839023000001, 2023-11-05T08:33:25.711 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.018837386000000001, 2023-11-05T08:33:25.711 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.711 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:25.711 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:25.712 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:25.712 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:25.712 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 113 2023-11-05T08:33:25.712 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.712 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.713 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.713 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.713 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151546+0000", 2023-11-05T08:33:25.713 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.713 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.714 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.714 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.714 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151546+0000", 2023-11-05T08:33:25.714 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.714 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.715 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.715 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.715 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151547+0000", 2023-11-05T08:33:25.715 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9699999999999995e-07 2023-11-05T08:33:25.715 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.716 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.716 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.716 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151553+0000", 2023-11-05T08:33:25.716 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9830000000000001e-06 2023-11-05T08:33:25.716 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.717 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.717 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.717 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151562+0000", 2023-11-05T08:33:25.717 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.5060000000000001e-06 2023-11-05T08:33:25.717 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.718 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.718 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.718 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151569+0000", 2023-11-05T08:33:25.718 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9160000000000004e-06 2023-11-05T08:33:25.718 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.719 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.719 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.719 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.163944+0000", 2023-11-05T08:33:25.719 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012374703000000001 2023-11-05T08:33:25.719 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.720 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.720 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:25.720 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.164091+0000", 2023-11-05T08:33:25.720 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014739199999999999 2023-11-05T08:33:25.720 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.720 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.721 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:25.721 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.164295+0000", 2023-11-05T08:33:25.721 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00020334899999999999 2023-11-05T08:33:25.721 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.721 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.722 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:25.722 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.167412+0000", 2023-11-05T08:33:25.722 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.003117529 2023-11-05T08:33:25.722 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.722 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.723 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:25.723 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.170241+0000", 2023-11-05T08:33:25.723 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0028283890000000002 2023-11-05T08:33:25.723 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.723 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.723 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:25.724 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.170326+0000", 2023-11-05T08:33:25.724 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.5581999999999995e-05 2023-11-05T08:33:25.724 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.724 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.724 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:25.725 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.170384+0000", 2023-11-05T08:33:25.725 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7340000000000003e-05 2023-11-05T08:33:25.725 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.725 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.725 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.725 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.726 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.726 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:25.726 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.151583+0000", 2023-11-05T08:33:25.726 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 56.535802578000002, 2023-11-05T08:33:25.726 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.020286660000000001, 2023-11-05T08:33:25.727 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.727 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:25.727 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:25.727 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:25.727 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:25.728 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 114 2023-11-05T08:33:25.728 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.728 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.728 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.728 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.729 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151583+0000", 2023-11-05T08:33:25.729 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.729 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.729 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.729 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.729 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151583+0000", 2023-11-05T08:33:25.730 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.730 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.730 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.730 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.730 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151583+0000", 2023-11-05T08:33:25.731 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.2600000000000002e-07 2023-11-05T08:33:25.731 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.731 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.731 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.731 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151589+0000", 2023-11-05T08:33:25.732 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5099999999999998e-06 2023-11-05T08:33:25.732 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.732 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.732 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.732 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151598+0000", 2023-11-05T08:33:25.732 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1730000000000001e-06 2023-11-05T08:33:25.733 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.733 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.733 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.733 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151608+0000", 2023-11-05T08:33:25.733 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0059e-05 2023-11-05T08:33:25.734 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.734 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.734 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.734 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.165015+0000", 2023-11-05T08:33:25.734 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.013407207000000001 2023-11-05T08:33:25.735 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.735 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.735 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:25.735 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.165268+0000", 2023-11-05T08:33:25.735 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00025284999999999999 2023-11-05T08:33:25.736 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.736 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.736 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:25.736 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.165635+0000", 2023-11-05T08:33:25.736 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00036695899999999998 2023-11-05T08:33:25.737 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.737 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.737 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:25.737 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.170105+0000", 2023-11-05T08:33:25.737 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0044696960000000004 2023-11-05T08:33:25.738 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.738 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.738 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:25.738 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.171719+0000", 2023-11-05T08:33:25.738 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001613855 2023-11-05T08:33:25.739 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.739 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.739 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:25.739 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.171810+0000", 2023-11-05T08:33:25.739 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1625000000000005e-05 2023-11-05T08:33:25.739 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.740 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.740 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:25.740 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.171869+0000", 2023-11-05T08:33:25.740 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.91e-05 2023-11-05T08:33:25.740 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.741 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.741 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.741 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.741 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.741 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:25.742 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.151660+0000", 2023-11-05T08:33:25.742 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 56.535724778999999, 2023-11-05T08:33:25.742 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022553719, 2023-11-05T08:33:25.742 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.742 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:25.743 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:25.743 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:25.743 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:25.743 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 115 2023-11-05T08:33:25.743 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.744 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.744 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.744 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.744 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151660+0000", 2023-11-05T08:33:25.744 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.745 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.745 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.745 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.745 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151660+0000", 2023-11-05T08:33:25.746 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.746 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.746 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.747 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.747 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151661+0000", 2023-11-05T08:33:25.747 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.5600000000000004e-07 2023-11-05T08:33:25.747 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.748 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.748 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.748 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151799+0000", 2023-11-05T08:33:25.748 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000137207 2023-11-05T08:33:25.748 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.749 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.749 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.749 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151814+0000", 2023-11-05T08:33:25.749 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4885000000000001e-05 2023-11-05T08:33:25.750 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.750 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.750 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.750 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151827+0000", 2023-11-05T08:33:25.750 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3413e-05 2023-11-05T08:33:25.751 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.751 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.751 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.751 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.167513+0000", 2023-11-05T08:33:25.751 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.015686091999999999 2023-11-05T08:33:25.752 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.752 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.752 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:25.752 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.167652+0000", 2023-11-05T08:33:25.753 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013890500000000001 2023-11-05T08:33:25.753 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.753 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.753 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:25.753 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.167860+0000", 2023-11-05T08:33:25.754 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00020838599999999999 2023-11-05T08:33:25.754 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.754 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.754 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:25.754 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.171602+0000", 2023-11-05T08:33:25.755 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0037417449999999999 2023-11-05T08:33:25.755 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.755 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.755 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:25.755 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174068+0000", 2023-11-05T08:33:25.755 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0024658010000000001 2023-11-05T08:33:25.756 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.756 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.756 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:25.756 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174152+0000", 2023-11-05T08:33:25.756 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.3752999999999996e-05 2023-11-05T08:33:25.757 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.757 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.757 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:25.757 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174214+0000", 2023-11-05T08:33:25.757 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.2576000000000005e-05 2023-11-05T08:33:25.758 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.758 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.758 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.758 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.758 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.759 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:25.759 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.151849+0000", 2023-11-05T08:33:25.759 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 56.535536423000003, 2023-11-05T08:33:25.759 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022629176000000001, 2023-11-05T08:33:25.759 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.759 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:25.760 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:25.760 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:25.760 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:25.760 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 116 2023-11-05T08:33:25.760 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.760 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.761 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.761 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.761 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151849+0000", 2023-11-05T08:33:25.761 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.761 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.762 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.762 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.762 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151849+0000", 2023-11-05T08:33:25.762 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.762 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.762 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.763 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.763 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151850+0000", 2023-11-05T08:33:25.763 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.037e-06 2023-11-05T08:33:25.763 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.763 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.764 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.764 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151858+0000", 2023-11-05T08:33:25.764 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.3389999999999993e-06 2023-11-05T08:33:25.764 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.764 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.764 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.765 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151868+0000", 2023-11-05T08:33:25.765 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6670000000000008e-06 2023-11-05T08:33:25.765 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.765 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.765 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.766 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151875+0000", 2023-11-05T08:33:25.766 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9229999999999996e-06 2023-11-05T08:33:25.766 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.766 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.766 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.767 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.168777+0000", 2023-11-05T08:33:25.767 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.016901884999999998 2023-11-05T08:33:25.767 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.767 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.767 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:25.768 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.169009+0000", 2023-11-05T08:33:25.768 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00023236400000000001 2023-11-05T08:33:25.768 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.768 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.768 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:25.769 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.169365+0000", 2023-11-05T08:33:25.769 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00035625300000000001 2023-11-05T08:33:25.769 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.769 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.769 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:25.770 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.171624+0000", 2023-11-05T08:33:25.770 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0022587169999999999 2023-11-05T08:33:25.770 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.770 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.770 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:25.771 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174314+0000", 2023-11-05T08:33:25.771 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002690263 2023-11-05T08:33:25.771 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.771 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.771 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:25.772 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174405+0000", 2023-11-05T08:33:25.772 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1064999999999996e-05 2023-11-05T08:33:25.772 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.772 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.772 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:25.772 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174478+0000", 2023-11-05T08:33:25.773 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.2663000000000005e-05 2023-11-05T08:33:25.773 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.773 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.773 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.773 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.774 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.774 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:25.774 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.151886+0000", 2023-11-05T08:33:25.774 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 56.535499008000002, 2023-11-05T08:33:25.774 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.023174162000000002, 2023-11-05T08:33:25.775 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.775 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:25.775 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:25.775 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:25.775 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:25.776 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 117 2023-11-05T08:33:25.776 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.776 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.776 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.776 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.777 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151886+0000", 2023-11-05T08:33:25.777 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.777 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.777 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.777 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.778 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151886+0000", 2023-11-05T08:33:25.778 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.778 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.778 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.778 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.779 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151887+0000", 2023-11-05T08:33:25.779 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.2300000000000001e-07 2023-11-05T08:33:25.779 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.779 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.779 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.779 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151893+0000", 2023-11-05T08:33:25.780 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.19e-06 2023-11-05T08:33:25.780 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.780 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.780 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.780 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151905+0000", 2023-11-05T08:33:25.781 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1883e-05 2023-11-05T08:33:25.781 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.781 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.781 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.781 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151912+0000", 2023-11-05T08:33:25.781 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.0690000000000004e-06 2023-11-05T08:33:25.782 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.782 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.782 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.782 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.170509+0000", 2023-11-05T08:33:25.782 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.018596686000000001 2023-11-05T08:33:25.783 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.783 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.783 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:25.783 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.170791+0000", 2023-11-05T08:33:25.783 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00028221799999999999 2023-11-05T08:33:25.784 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.784 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.784 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:25.784 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.171137+0000", 2023-11-05T08:33:25.784 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000346454 2023-11-05T08:33:25.785 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.785 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.785 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:25.785 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.172413+0000", 2023-11-05T08:33:25.785 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0012752340000000001 2023-11-05T08:33:25.786 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.786 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.786 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:25.786 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174926+0000", 2023-11-05T08:33:25.786 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0025137050000000002 2023-11-05T08:33:25.787 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.787 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.787 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:25.787 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.175007+0000", 2023-11-05T08:33:25.787 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.0413000000000004e-05 2023-11-05T08:33:25.788 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.788 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.788 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:25.788 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.175060+0000", 2023-11-05T08:33:25.788 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3687000000000003e-05 2023-11-05T08:33:25.788 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.789 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.789 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.789 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.789 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.789 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:163 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:25.790 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.670890+0000", 2023-11-05T08:33:25.790 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 56.016494778999999, 2023-11-05T08:33:25.790 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.02132307, 2023-11-05T08:33:25.790 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.790 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:25.791 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:25.791 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:25.791 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:25.791 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 163 2023-11-05T08:33:25.791 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.791 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.792 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.792 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.792 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670890+0000", 2023-11-05T08:33:25.792 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.792 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.793 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.793 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.793 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670890+0000", 2023-11-05T08:33:25.793 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.793 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.794 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.794 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.794 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670891+0000", 2023-11-05T08:33:25.794 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3300000000000002e-07 2023-11-05T08:33:25.794 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.795 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.795 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.795 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670897+0000", 2023-11-05T08:33:25.795 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.4600000000000002e-06 2023-11-05T08:33:25.795 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.795 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.796 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.796 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670906+0000", 2023-11-05T08:33:25.796 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.3130000000000004e-06 2023-11-05T08:33:25.796 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.796 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.797 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.797 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670913+0000", 2023-11-05T08:33:25.797 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9859999999999997e-06 2023-11-05T08:33:25.797 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.797 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.798 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.798 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.687360+0000", 2023-11-05T08:33:25.798 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.016447244 2023-11-05T08:33:25.798 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.798 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.799 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:25.799 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.687588+0000", 2023-11-05T08:33:25.799 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00022757699999999999 2023-11-05T08:33:25.799 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.799 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.800 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:25.800 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.687946+0000", 2023-11-05T08:33:25.800 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00035789399999999999 2023-11-05T08:33:25.800 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.800 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.801 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:25.801 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.691929+0000", 2023-11-05T08:33:25.801 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0039835690000000002 2023-11-05T08:33:25.801 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.801 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.802 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:25.802 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.691970+0000", 2023-11-05T08:33:25.802 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.0351e-05 2023-11-05T08:33:25.802 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.803 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.803 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:25.803 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692111+0000", 2023-11-05T08:33:25.803 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014114499999999999 2023-11-05T08:33:25.803 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.803 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.804 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:25.804 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692214+0000", 2023-11-05T08:33:25.804 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000102898 2023-11-05T08:33:25.804 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.804 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.805 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.805 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.805 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.805 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:25.805 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.670924+0000", 2023-11-05T08:33:25.806 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 56.016461370999998, 2023-11-05T08:33:25.806 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.021621676999999999, 2023-11-05T08:33:25.806 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.806 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:25.806 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:25.807 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:25.807 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:25.807 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 164 2023-11-05T08:33:25.807 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.808 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.808 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.808 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.808 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670924+0000", 2023-11-05T08:33:25.808 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.808 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.809 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.809 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.809 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670924+0000", 2023-11-05T08:33:25.809 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.809 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.810 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.810 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.810 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670925+0000", 2023-11-05T08:33:25.810 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.8299999999999996e-07 2023-11-05T08:33:25.810 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.811 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.811 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.811 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671022+0000", 2023-11-05T08:33:25.811 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.7445e-05 2023-11-05T08:33:25.811 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.812 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.812 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.812 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671033+0000", 2023-11-05T08:33:25.812 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1206000000000001e-05 2023-11-05T08:33:25.812 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.812 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.813 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.813 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671045+0000", 2023-11-05T08:33:25.813 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1797000000000001e-05 2023-11-05T08:33:25.813 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.813 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.813 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.814 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.688874+0000", 2023-11-05T08:33:25.814 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017829424 2023-11-05T08:33:25.814 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.814 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.814 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:25.815 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.689097+0000", 2023-11-05T08:33:25.815 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000222828 2023-11-05T08:33:25.815 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.815 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.815 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:25.815 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.689444+0000", 2023-11-05T08:33:25.816 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00034672099999999998 2023-11-05T08:33:25.816 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.816 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.816 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:25.816 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692240+0000", 2023-11-05T08:33:25.816 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0027959030000000002 2023-11-05T08:33:25.817 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.817 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.817 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:25.817 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692339+0000", 2023-11-05T08:33:25.817 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.8801000000000005e-05 2023-11-05T08:33:25.818 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.818 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.818 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:25.818 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692435+0000", 2023-11-05T08:33:25.818 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6311999999999995e-05 2023-11-05T08:33:25.819 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.819 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.819 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:25.819 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692546+0000", 2023-11-05T08:33:25.819 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00011055700000000001 2023-11-05T08:33:25.819 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.820 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.820 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.820 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.820 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.820 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:25.821 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.671060+0000", 2023-11-05T08:33:25.821 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 56.016325705, 2023-11-05T08:33:25.821 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022080292000000001, 2023-11-05T08:33:25.821 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.821 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:25.821 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:25.822 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:25.822 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:25.822 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 165 2023-11-05T08:33:25.822 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.822 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.823 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.823 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.823 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671060+0000", 2023-11-05T08:33:25.823 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.823 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.823 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.824 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.824 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671060+0000", 2023-11-05T08:33:25.824 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.824 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.824 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.824 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.825 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671060+0000", 2023-11-05T08:33:25.825 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.1699999999999997e-07 2023-11-05T08:33:25.825 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.825 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.825 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.826 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671067+0000", 2023-11-05T08:33:25.826 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5690000000000001e-06 2023-11-05T08:33:25.826 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.826 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.826 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.826 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671076+0000", 2023-11-05T08:33:25.827 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.5270000000000004e-06 2023-11-05T08:33:25.827 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.827 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.827 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.827 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671084+0000", 2023-11-05T08:33:25.828 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.3459999999999996e-06 2023-11-05T08:33:25.828 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.828 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.828 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.828 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.690832+0000", 2023-11-05T08:33:25.829 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.019747793 2023-11-05T08:33:25.829 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.829 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.829 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:25.829 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.690971+0000", 2023-11-05T08:33:25.830 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000139245 2023-11-05T08:33:25.830 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.830 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.830 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:25.830 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.691178+0000", 2023-11-05T08:33:25.831 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000206892 2023-11-05T08:33:25.831 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.831 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.831 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:25.831 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692704+0000", 2023-11-05T08:33:25.831 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00152655 2023-11-05T08:33:25.832 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.832 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.832 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:25.832 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692904+0000", 2023-11-05T08:33:25.832 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000199376 2023-11-05T08:33:25.833 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.833 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.833 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:25.833 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.693044+0000", 2023-11-05T08:33:25.833 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013989199999999999 2023-11-05T08:33:25.834 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.834 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.834 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:25.834 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.693140+0000", 2023-11-05T08:33:25.834 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6285000000000004e-05 2023-11-05T08:33:25.835 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.835 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.835 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.835 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.835 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.836 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:184 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:25.836 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.866152+0000", 2023-11-05T08:33:25.836 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 55.821233429999999, 2023-11-05T08:33:25.836 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.019116587000000001, 2023-11-05T08:33:25.836 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.837 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:25.837 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:25.837 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:25.837 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:25.837 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 184 2023-11-05T08:33:25.837 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.838 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.838 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.838 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.838 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866152+0000", 2023-11-05T08:33:25.838 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.839 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.839 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.839 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.839 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866152+0000", 2023-11-05T08:33:25.839 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.840 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.840 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.840 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.840 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866152+0000", 2023-11-05T08:33:25.840 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4600000000000004e-07 2023-11-05T08:33:25.841 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.841 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.841 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.841 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866158+0000", 2023-11-05T08:33:25.841 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5629999999999999e-06 2023-11-05T08:33:25.841 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.842 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.842 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.842 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866167+0000", 2023-11-05T08:33:25.842 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.2229999999999996e-06 2023-11-05T08:33:25.843 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.843 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.843 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.843 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866175+0000", 2023-11-05T08:33:25.843 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.3699999999999997e-06 2023-11-05T08:33:25.844 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.844 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.844 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.844 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.878285+0000", 2023-11-05T08:33:25.844 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012110164 2023-11-05T08:33:25.845 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.845 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.845 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:25.845 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.878584+0000", 2023-11-05T08:33:25.845 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00029961300000000001 2023-11-05T08:33:25.846 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.846 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.846 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:25.846 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.878942+0000", 2023-11-05T08:33:25.846 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00035795100000000002 2023-11-05T08:33:25.847 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.847 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.847 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:25.847 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.883845+0000", 2023-11-05T08:33:25.847 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0049021450000000001 2023-11-05T08:33:25.848 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.848 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.848 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:25.848 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885018+0000", 2023-11-05T08:33:25.848 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0011738160000000001 2023-11-05T08:33:25.848 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.849 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.849 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:25.849 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885167+0000", 2023-11-05T08:33:25.849 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014895800000000001 2023-11-05T08:33:25.849 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.850 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.850 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:25.850 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885268+0000", 2023-11-05T08:33:25.850 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010113800000000001 2023-11-05T08:33:25.850 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.851 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.851 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.851 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.851 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.851 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:185 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:25.851 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.866212+0000", 2023-11-05T08:33:25.852 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 55.821173709999997, 2023-11-05T08:33:25.852 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.019448929, 2023-11-05T08:33:25.852 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.852 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:25.852 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:25.853 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:25.853 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:25.853 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 185 2023-11-05T08:33:25.853 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.853 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.854 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.854 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.854 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866212+0000", 2023-11-05T08:33:25.854 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.854 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.855 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.855 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.855 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866212+0000", 2023-11-05T08:33:25.855 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.856 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.856 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.856 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.856 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866213+0000", 2023-11-05T08:33:25.856 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0559999999999999e-06 2023-11-05T08:33:25.857 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.857 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.857 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.857 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866308+0000", 2023-11-05T08:33:25.857 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.4984999999999999e-05 2023-11-05T08:33:25.858 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.858 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.858 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.858 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866320+0000", 2023-11-05T08:33:25.858 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2639e-05 2023-11-05T08:33:25.859 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.859 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.859 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.859 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866328+0000", 2023-11-05T08:33:25.859 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.1240000000000005e-06 2023-11-05T08:33:25.860 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.860 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.860 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.860 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.879899+0000", 2023-11-05T08:33:25.860 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.013571059999999999 2023-11-05T08:33:25.861 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.861 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.861 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:25.861 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.880034+0000", 2023-11-05T08:33:25.861 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000134676 2023-11-05T08:33:25.861 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.862 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.862 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:25.862 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.880258+0000", 2023-11-05T08:33:25.862 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00022358100000000001 2023-11-05T08:33:25.862 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.863 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.863 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:25.863 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885299+0000", 2023-11-05T08:33:25.863 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.005041678 2023-11-05T08:33:25.863 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.864 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.864 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:25.864 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885436+0000", 2023-11-05T08:33:25.864 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000136472 2023-11-05T08:33:25.864 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.864 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.865 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:25.865 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885557+0000", 2023-11-05T08:33:25.865 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000121644 2023-11-05T08:33:25.865 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.865 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.866 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:25.866 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885660+0000", 2023-11-05T08:33:25.866 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000103014 2023-11-05T08:33:25.866 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.866 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.867 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.867 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.867 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.867 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:186 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:25.867 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.866343+0000", 2023-11-05T08:33:25.867 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 55.821041766999997, 2023-11-05T08:33:25.868 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.021691589000000001, 2023-11-05T08:33:25.868 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.868 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:25.868 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:25.868 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:25.869 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:25.869 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 186 2023-11-05T08:33:25.869 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.869 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.869 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.870 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.870 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866343+0000", 2023-11-05T08:33:25.870 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.870 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.870 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.871 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.871 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866343+0000", 2023-11-05T08:33:25.871 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.871 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.871 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.871 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.872 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866344+0000", 2023-11-05T08:33:25.872 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.6000000000000003e-07 2023-11-05T08:33:25.872 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.872 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.872 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.873 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866351+0000", 2023-11-05T08:33:25.873 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3090000000000001e-06 2023-11-05T08:33:25.873 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.873 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.873 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.873 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866360+0000", 2023-11-05T08:33:25.874 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.2599999999999994e-06 2023-11-05T08:33:25.874 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.874 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.874 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.874 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866367+0000", 2023-11-05T08:33:25.875 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.7599999999999997e-06 2023-11-05T08:33:25.875 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.875 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.875 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.875 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.882269+0000", 2023-11-05T08:33:25.875 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.015902218999999999 2023-11-05T08:33:25.876 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.876 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.876 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:25.876 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.882545+0000", 2023-11-05T08:33:25.876 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00027617199999999998 2023-11-05T08:33:25.876 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.877 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.877 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:25.877 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.882925+0000", 2023-11-05T08:33:25.877 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00037995900000000003 2023-11-05T08:33:25.877 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.878 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.878 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:25.878 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.887718+0000", 2023-11-05T08:33:25.878 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0047935069999999998 2023-11-05T08:33:25.878 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.879 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.879 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:25.879 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.887836+0000", 2023-11-05T08:33:25.879 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000117401 2023-11-05T08:33:25.879 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.880 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.880 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:25.880 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.887931+0000", 2023-11-05T08:33:25.880 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.4802000000000005e-05 2023-11-05T08:33:25.880 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.880 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.881 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:25.881 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888035+0000", 2023-11-05T08:33:25.881 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010444 2023-11-05T08:33:25.881 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.881 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.881 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.882 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.882 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.882 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:187 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:25.882 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.866377+0000", 2023-11-05T08:33:25.882 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 55.821007805999997, 2023-11-05T08:33:25.883 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022216751, 2023-11-05T08:33:25.883 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.883 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:25.883 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:25.883 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:25.883 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:25.884 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 187 2023-11-05T08:33:25.884 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.884 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.884 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.884 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.885 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866377+0000", 2023-11-05T08:33:25.885 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.885 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.885 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.885 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.886 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866377+0000", 2023-11-05T08:33:25.886 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.886 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.886 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.886 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.887 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866378+0000", 2023-11-05T08:33:25.887 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9999999999999997e-07 2023-11-05T08:33:25.887 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.887 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.887 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.888 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866384+0000", 2023-11-05T08:33:25.888 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.4099999999999999e-06 2023-11-05T08:33:25.888 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.888 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.888 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.889 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866393+0000", 2023-11-05T08:33:25.889 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9719999999999998e-06 2023-11-05T08:33:25.889 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.889 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.889 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.890 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866400+0000", 2023-11-05T08:33:25.890 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.2740000000000002e-06 2023-11-05T08:33:25.890 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.890 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.890 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.891 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.883936+0000", 2023-11-05T08:33:25.891 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017536513 2023-11-05T08:33:25.891 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.891 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.891 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:25.892 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.884061+0000", 2023-11-05T08:33:25.892 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012435300000000001 2023-11-05T08:33:25.892 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.892 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.892 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:25.892 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.884277+0000", 2023-11-05T08:33:25.893 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00021665500000000001 2023-11-05T08:33:25.893 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.893 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.893 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:25.894 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888193+0000", 2023-11-05T08:33:25.894 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0039160319999999998 2023-11-05T08:33:25.894 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.894 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.894 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:25.895 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888235+0000", 2023-11-05T08:33:25.895 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1267999999999999e-05 2023-11-05T08:33:25.895 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.895 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.895 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:25.896 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888373+0000", 2023-11-05T08:33:25.896 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000138789 2023-11-05T08:33:25.896 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.896 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.896 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:25.897 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888594+0000", 2023-11-05T08:33:25.897 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00022078500000000001 2023-11-05T08:33:25.897 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.897 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.897 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.898 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.898 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.898 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:188 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:25.898 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.866411+0000", 2023-11-05T08:33:25.898 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 55.820974544999999, 2023-11-05T08:33:25.899 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.023678187999999999, 2023-11-05T08:33:25.899 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.899 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:25.899 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:25.899 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:25.900 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:25.900 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 188 2023-11-05T08:33:25.900 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.900 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.900 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.900 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.901 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866411+0000", 2023-11-05T08:33:25.901 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.901 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.901 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.901 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.902 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866411+0000", 2023-11-05T08:33:25.902 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.902 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.902 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.902 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.903 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866411+0000", 2023-11-05T08:33:25.903 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.44e-07 2023-11-05T08:33:25.903 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.903 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.903 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.904 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866531+0000", 2023-11-05T08:33:25.904 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000119664 2023-11-05T08:33:25.904 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.904 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.904 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.904 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866545+0000", 2023-11-05T08:33:25.905 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3875e-05 2023-11-05T08:33:25.905 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.905 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.905 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.905 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866554+0000", 2023-11-05T08:33:25.906 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9099999999999994e-06 2023-11-05T08:33:25.906 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.906 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.906 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.906 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885784+0000", 2023-11-05T08:33:25.907 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.019230651000000001 2023-11-05T08:33:25.907 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.907 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.907 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:25.907 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.886020+0000", 2023-11-05T08:33:25.907 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00023538300000000001 2023-11-05T08:33:25.908 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.908 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.908 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:25.908 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.886386+0000", 2023-11-05T08:33:25.908 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00036659399999999998 2023-11-05T08:33:25.909 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.909 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.909 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:25.909 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888662+0000", 2023-11-05T08:33:25.909 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0022756019999999998 2023-11-05T08:33:25.909 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.910 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.910 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:25.910 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.889922+0000", 2023-11-05T08:33:25.910 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0012603149999999999 2023-11-05T08:33:25.910 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.911 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.911 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:25.911 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.890034+0000", 2023-11-05T08:33:25.911 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000111283 2023-11-05T08:33:25.911 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.911 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.912 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:25.912 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.890089+0000", 2023-11-05T08:33:25.912 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5266999999999998e-05 2023-11-05T08:33:25.912 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.912 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.913 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.913 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.913 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.913 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:189 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:25.913 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.866572+0000", 2023-11-05T08:33:25.913 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 55.820813587000004, 2023-11-05T08:33:25.914 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.024895937999999999, 2023-11-05T08:33:25.914 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.914 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:25.914 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:25.914 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:25.915 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:25.915 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 189 2023-11-05T08:33:25.915 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.915 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.915 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.915 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.916 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866572+0000", 2023-11-05T08:33:25.916 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.916 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.916 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.916 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.917 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866572+0000", 2023-11-05T08:33:25.917 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.917 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.917 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.917 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.918 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866572+0000", 2023-11-05T08:33:25.918 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.8000000000000005e-07 2023-11-05T08:33:25.918 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.918 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.918 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.919 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866581+0000", 2023-11-05T08:33:25.919 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9800000000000004e-06 2023-11-05T08:33:25.919 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.919 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.919 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.919 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866591+0000", 2023-11-05T08:33:25.920 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6059999999999992e-06 2023-11-05T08:33:25.920 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.920 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.920 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.920 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866598+0000", 2023-11-05T08:33:25.921 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.1029999999999996e-06 2023-11-05T08:33:25.921 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.921 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.921 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.921 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888850+0000", 2023-11-05T08:33:25.922 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022251615999999998 2023-11-05T08:33:25.922 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.922 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.922 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:25.922 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888988+0000", 2023-11-05T08:33:25.923 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000138436 2023-11-05T08:33:25.923 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.923 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.923 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:25.923 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.889206+0000", 2023-11-05T08:33:25.923 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00021818099999999999 2023-11-05T08:33:25.924 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.924 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.924 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:25.924 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.890621+0000", 2023-11-05T08:33:25.924 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001414219 2023-11-05T08:33:25.924 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.925 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.925 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:25.925 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.891316+0000", 2023-11-05T08:33:25.925 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00069570500000000002 2023-11-05T08:33:25.925 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.925 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.926 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:25.926 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.891406+0000", 2023-11-05T08:33:25.926 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9901999999999994e-05 2023-11-05T08:33:25.926 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.926 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.927 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:25.927 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.891468+0000", 2023-11-05T08:33:25.927 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1409999999999996e-05 2023-11-05T08:33:25.927 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.927 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.928 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.928 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.928 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.928 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:292 3.2s0 3:41aa5db9:::smithi09039252-7:head [delete] snapc 8=[8,7,6,5] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e41)", 2023-11-05T08:33:25.928 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.121023+0000", 2023-11-05T08:33:25.928 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 31.566362172000002, 2023-11-05T08:33:25.929 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.061679772000000001, 2023-11-05T08:33:25.929 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.929 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:25.929 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:25.929 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:25.930 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:25.930 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 292 2023-11-05T08:33:25.930 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.930 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.930 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.931 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.931 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.121023+0000", 2023-11-05T08:33:25.931 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.931 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.931 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.932 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.932 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.121023+0000", 2023-11-05T08:33:25.932 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.932 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.932 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.933 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.933 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.121026+0000", 2023-11-05T08:33:25.933 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0369999999999999e-06 2023-11-05T08:33:25.933 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.933 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.934 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.934 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.121039+0000", 2023-11-05T08:33:25.934 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3103e-05 2023-11-05T08:33:25.934 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.935 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.935 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.935 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.121086+0000", 2023-11-05T08:33:25.935 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7095000000000002e-05 2023-11-05T08:33:25.935 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.936 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.936 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.936 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.121131+0000", 2023-11-05T08:33:25.936 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4455e-05 2023-11-05T08:33:25.936 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.937 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.937 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.937 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.129136+0000", 2023-11-05T08:33:25.937 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0080047439999999994 2023-11-05T08:33:25.937 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.938 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.938 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2023-11-05T08:33:25.938 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.129142+0000", 2023-11-05T08:33:25.938 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3759999999999996e-06 2023-11-05T08:33:25.939 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.939 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.939 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.939 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.158095+0000", 2023-11-05T08:33:25.939 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.028953402 2023-11-05T08:33:25.940 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.940 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.940 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:25.940 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.158534+0000", 2023-11-05T08:33:25.940 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00043911800000000001 2023-11-05T08:33:25.940 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.941 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.941 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:25.941 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.160380+0000", 2023-11-05T08:33:25.941 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001845923 2023-11-05T08:33:25.941 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.942 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.942 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:25.942 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.177604+0000", 2023-11-05T08:33:25.942 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017223405000000001 2023-11-05T08:33:25.942 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.943 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.943 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:25.943 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.182556+0000", 2023-11-05T08:33:25.943 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0049522740000000004 2023-11-05T08:33:25.943 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.943 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.944 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:25.944 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.182703+0000", 2023-11-05T08:33:25.944 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014684 2023-11-05T08:33:25.944 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.944 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.944 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.945 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.945 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.945 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:335 3.18s0 3:1aa38538:::smithi09039252-11:head [stat,delete] snapc 10=[10,7] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2023-11-05T08:33:25.945 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:04.225103+0000", 2023-11-05T08:33:25.945 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 21.462282298000002, 2023-11-05T08:33:25.945 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.028941109999999999, 2023-11-05T08:33:25.946 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.946 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:25.946 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:25.946 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:25.946 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:25.946 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 335 2023-11-05T08:33:25.947 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.947 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.947 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.947 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.947 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.225103+0000", 2023-11-05T08:33:25.948 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.948 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.948 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.948 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.948 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.225103+0000", 2023-11-05T08:33:25.949 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.949 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.949 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.949 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.949 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.225106+0000", 2023-11-05T08:33:25.949 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0929999999999999e-06 2023-11-05T08:33:25.950 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.950 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.950 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.950 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.225118+0000", 2023-11-05T08:33:25.950 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2017e-05 2023-11-05T08:33:25.951 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.951 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.951 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.951 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.225155+0000", 2023-11-05T08:33:25.951 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.6711999999999997e-05 2023-11-05T08:33:25.951 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.952 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.952 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.952 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.225172+0000", 2023-11-05T08:33:25.952 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7367000000000002e-05 2023-11-05T08:33:25.952 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.952 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.953 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.953 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.225312+0000", 2023-11-05T08:33:25.953 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000140323 2023-11-05T08:33:25.953 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.953 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.953 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op must wait for map", 2023-11-05T08:33:25.954 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.225355+0000", 2023-11-05T08:33:25.954 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.2746e-05 2023-11-05T08:33:25.954 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.954 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.954 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.955 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.237049+0000", 2023-11-05T08:33:25.955 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011693360999999999 2023-11-05T08:33:25.955 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.955 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.955 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:25.955 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.237652+0000", 2023-11-05T08:33:25.956 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000603749 2023-11-05T08:33:25.956 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.956 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.956 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:25.956 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.245013+0000", 2023-11-05T08:33:25.957 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0073607000000000004 2023-11-05T08:33:25.957 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.957 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.957 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:25.957 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.249533+0000", 2023-11-05T08:33:25.958 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0045203939999999996 2023-11-05T08:33:25.958 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.958 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.958 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:25.958 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.253639+0000", 2023-11-05T08:33:25.959 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0041051730000000002 2023-11-05T08:33:25.959 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.959 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.959 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:25.959 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.254044+0000", 2023-11-05T08:33:25.960 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00040547499999999998 2023-11-05T08:33:25.960 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.960 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.960 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.960 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.961 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.961 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(osd.5.22:1 3.10s0 3:084c43b8:::smithi09039252-5:head [assert-version v10,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:25.961 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:05.258544+0000", 2023-11-05T08:33:25.961 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 20.428841611999999, 2023-11-05T08:33:25.961 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.024845459, 2023-11-05T08:33:25.962 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.962 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2023-11-05T08:33:25.962 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.962 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.963 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.963 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258544+0000", 2023-11-05T08:33:25.963 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.963 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.963 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.963 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.964 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258544+0000", 2023-11-05T08:33:25.964 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.964 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.964 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.964 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.965 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258547+0000", 2023-11-05T08:33:25.965 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0740000000000001e-06 2023-11-05T08:33:25.965 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.965 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.965 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.966 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258561+0000", 2023-11-05T08:33:25.966 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4525999999999999e-05 2023-11-05T08:33:25.966 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.966 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.966 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.967 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258601+0000", 2023-11-05T08:33:25.967 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.9433e-05 2023-11-05T08:33:25.967 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.967 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.967 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.968 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258625+0000", 2023-11-05T08:33:25.968 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3819e-05 2023-11-05T08:33:25.968 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.968 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.968 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.968 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258788+0000", 2023-11-05T08:33:25.969 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00016302600000000001 2023-11-05T08:33:25.969 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.969 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.969 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op must wait for map", 2023-11-05T08:33:25.969 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258812+0000", 2023-11-05T08:33:25.969 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4043e-05 2023-11-05T08:33:25.970 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.970 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.970 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.970 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.275660+0000", 2023-11-05T08:33:25.970 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.016848823999999998 2023-11-05T08:33:25.971 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.971 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.971 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:25.971 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.276035+0000", 2023-11-05T08:33:25.971 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00037483200000000002 2023-11-05T08:33:25.972 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.972 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.972 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:25.972 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.283389+0000", 2023-11-05T08:33:25.972 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.007353882 2023-11-05T08:33:25.973 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.973 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.973 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.973 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.973 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.973 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:351 3.18s0 3:1aa38538:::smithi09039252-11:head [append 0~491520] snapc 10=[7] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:25.974 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:05.264529+0000", 2023-11-05T08:33:25.974 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 20.422856383999999, 2023-11-05T08:33:25.974 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.029015691, 2023-11-05T08:33:25.974 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.974 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:25.975 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:25.975 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:25.975 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:25.975 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 351 2023-11-05T08:33:25.975 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.975 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.976 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.976 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.976 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.264529+0000", 2023-11-05T08:33:25.976 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.976 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.977 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.977 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.977 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.264529+0000", 2023-11-05T08:33:25.977 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.977 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.978 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.978 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.978 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.264532+0000", 2023-11-05T08:33:25.978 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3730000000000001e-06 2023-11-05T08:33:25.978 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.979 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.979 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.979 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.266714+0000", 2023-11-05T08:33:25.979 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0021819880000000002 2023-11-05T08:33:25.979 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.979 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.980 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.980 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.266743+0000", 2023-11-05T08:33:25.980 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9218999999999999e-05 2023-11-05T08:33:25.980 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.980 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.980 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.981 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.266759+0000", 2023-11-05T08:33:25.981 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5102999999999999e-05 2023-11-05T08:33:25.981 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.981 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.981 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.981 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.266928+0000", 2023-11-05T08:33:25.982 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00016909299999999999 2023-11-05T08:33:25.982 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.982 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.982 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op must wait for map", 2023-11-05T08:33:25.982 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.266953+0000", 2023-11-05T08:33:25.983 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5344999999999999e-05 2023-11-05T08:33:25.983 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.983 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.983 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.983 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.276760+0000", 2023-11-05T08:33:25.983 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0098067109999999992 2023-11-05T08:33:25.984 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.984 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.984 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:25.984 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.277064+0000", 2023-11-05T08:33:25.984 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00030468299999999998 2023-11-05T08:33:25.985 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.985 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.985 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:25.985 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.279923+0000", 2023-11-05T08:33:25.985 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0028589119999999999 2023-11-05T08:33:25.986 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.986 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.986 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:25.986 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.292166+0000", 2023-11-05T08:33:25.986 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012242723 2023-11-05T08:33:25.987 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.987 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.987 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:25.987 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.292949+0000", 2023-11-05T08:33:25.988 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00078316699999999998 2023-11-05T08:33:25.988 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.988 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.988 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:25.988 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.293545+0000", 2023-11-05T08:33:25.988 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00059537399999999997 2023-11-05T08:33:25.989 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.989 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:25.989 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:25.989 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.990 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.990 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:354 3.18s0 3:1aa38538:::smithi09039252-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:25.990 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:05.281130+0000", 2023-11-05T08:33:25.990 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 20.406254790999999, 2023-11-05T08:33:25.990 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.021364054, 2023-11-05T08:33:25.991 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:25.991 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2023-11-05T08:33:25.991 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:25.991 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:25.991 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:25.992 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 354 2023-11-05T08:33:25.992 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.992 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:25.992 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.993 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:25.993 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.281130+0000", 2023-11-05T08:33:25.993 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.993 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.993 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.993 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:25.994 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.281130+0000", 2023-11-05T08:33:25.994 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:25.994 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.994 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.994 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:25.995 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.281131+0000", 2023-11-05T08:33:25.995 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.0299999999999998e-07 2023-11-05T08:33:25.995 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.995 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.995 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:25.995 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.281135+0000", 2023-11-05T08:33:25.996 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2799999999999999e-06 2023-11-05T08:33:25.996 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.996 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.996 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:25.996 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.281146+0000", 2023-11-05T08:33:25.997 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.119e-05 2023-11-05T08:33:25.997 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.997 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.997 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:25.997 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.281154+0000", 2023-11-05T08:33:25.997 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.7719999999999994e-06 2023-11-05T08:33:25.998 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.998 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.998 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:25.998 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.298629+0000", 2023-11-05T08:33:25.998 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017475195999999998 2023-11-05T08:33:25.998 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.999 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:25.999 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:25.999 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.298839+0000", 2023-11-05T08:33:25.999 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00020991799999999999 2023-11-05T08:33:25.999 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:25.999 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:26.000 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:26.000 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.302495+0000", 2023-11-05T08:33:26.000 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0036558950000000002 2023-11-05T08:33:26.000 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:26.000 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:26.000 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:26.001 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:26.001 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:26.001 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:33:26.002 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-05T08:33:26.003 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:25.995+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:26.004 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-11-05T08:33:26.103 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:33:26.099+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:33:26.136 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:33:26.136 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2023-11-05T08:33:26.136 INFO:teuthology.orchestra.run.smithi069.stdout: "num_ops": 0 2023-11-05T08:33:26.136 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:33:26.143 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-11-05T08:33:26.151 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-11-05T08:33:26.166 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1: {} 2023-11-05T08:33:26.166 INFO:teuthology.orchestra.run.smithi002.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-11-05T08:33:26.185 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2: {} 2023-11-05T08:33:26.186 INFO:teuthology.orchestra.run.smithi002.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-11-05T08:33:26.186 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-11-05T08:33:26.204 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4: {} 2023-11-05T08:33:26.204 INFO:teuthology.orchestra.run.smithi002.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-11-05T08:33:26.205 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:33:26.202+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:33:26.226 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5: {} 2023-11-05T08:33:26.227 INFO:teuthology.orchestra.run.smithi002.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-11-05T08:33:26.249 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6: {} 2023-11-05T08:33:26.250 INFO:teuthology.orchestra.run.smithi002.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-11-05T08:33:26.267 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:26.267 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:26.272 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7: {} 2023-11-05T08:33:26.273 INFO:teuthology.orchestra.run.smithi002.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-11-05T08:33:26.281 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-11-05T08:33:26.293 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8: {} 2023-11-05T08:33:26.293 INFO:teuthology.orchestra.run.smithi002.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-11-05T08:33:26.305 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:26.300+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:26.317 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9: {} 2023-11-05T08:33:26.317 INFO:teuthology.orchestra.run.smithi002.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-11-05T08:33:26.340 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10: {} 2023-11-05T08:33:26.341 INFO:teuthology.orchestra.run.smithi002.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-11-05T08:33:26.365 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11: {} 2023-11-05T08:33:26.366 INFO:teuthology.orchestra.run.smithi002.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-11-05T08:33:26.388 INFO:teuthology.orchestra.run.smithi002.stdout:osd.12: {} 2023-11-05T08:33:26.389 INFO:teuthology.orchestra.run.smithi002.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-11-05T08:33:26.405 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:26.399+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:26.502 INFO:teuthology.orchestra.run.smithi002.stdout:osd.13: {} 2023-11-05T08:33:26.502 INFO:tasks.rados.rados.0.smithi090.stdout:231: done (1 left) 2023-11-05T08:33:26.503 INFO:tasks.rados.rados.0.smithi090.stdout:232: snap_remove snap 19 2023-11-05T08:33:26.503 INFO:teuthology.orchestra.run.smithi002.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-11-05T08:33:26.503 INFO:teuthology.orchestra.run.smithi002.stderr:nodeep-scrub is unset 2023-11-05T08:33:26.504 INFO:teuthology.orchestra.run.smithi002.stdout:osd.14: {} 2023-11-05T08:33:26.504 INFO:teuthology.orchestra.run.smithi002.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-11-05T08:33:26.505 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:26.505 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:26.506 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-11-05T08:33:26.507 INFO:teuthology.orchestra.run.smithi002.stdout:osd.15: {} 2023-11-05T08:33:26.508 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-05T08:33:26.508 INFO:teuthology.orchestra.run.smithi002.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-11-05T08:33:26.509 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-05T08:33:26.509 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:33:26.505+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:33:26.609 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:33:26.606+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:33:26.632 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:26.632 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:26.646 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-11-05T08:33:26.709 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:33:26.704+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:33:26.810 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:26.810 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:26.811 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:33:26.808+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:33:26.823 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-11-05T08:33:26.911 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:33:26.910+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:33:26.938 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:26.939 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:26.953 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2023-11-05T08:33:27.012 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:27.007+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:27.113 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:27.114 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:27.114 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:33:27.108+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:33:27.127 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2023-11-05T08:33:27.214 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:33:27.210+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:33:27.241 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:27.242 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:27.257 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2023-11-05T08:33:27.314 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:33:27.307+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:33:27.415 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:33:27.414+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:33:27.418 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [3] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 7, 11, 15] 2023-11-05T08:33:27.418 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-11-05T08:33:27.419 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2023-11-05T08:33:27.419 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-11-05T08:33:27.419 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-11-05T08:33:27.419 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3 2023-11-05T08:33:27.421 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:27.421 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:27.435 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2023-11-05T08:33:27.452 INFO:tasks.rados.rados.0.smithi090.stdout:232: done (1 left) 2023-11-05T08:33:27.453 INFO:tasks.rados.rados.0.smithi090.stdout:233: rollback oid 47 current snap is 21 2023-11-05T08:33:27.453 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 47 to 4 2023-11-05T08:33:27.453 INFO:tasks.rados.rados.0.smithi090.stdout:234: rollback oid 5 current snap is 21 2023-11-05T08:33:27.453 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 5 to 18 2023-11-05T08:33:27.454 INFO:tasks.rados.rados.0.smithi090.stdout:235: read oid 33 snap 18 2023-11-05T08:33:27.454 INFO:tasks.rados.rados.0.smithi090.stdout:235: expect (ObjNum 32 snap 0 seq_num 32) 2023-11-05T08:33:27.454 INFO:tasks.rados.rados.0.smithi090.stdout:236: delete oid 20 current snap is 21 2023-11-05T08:33:27.463 INFO:tasks.rados.rados.0.smithi090.stdout:236: done (4 left) 2023-11-05T08:33:27.463 INFO:tasks.rados.rados.0.smithi090.stdout:237: snap_remove snap 20 2023-11-05T08:33:27.523 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:33:27.522+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:33:27.562 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:33:27.563 INFO:teuthology.orchestra.run.smithi053.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2023-11-05T08:33:27.563 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:33:27.596 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:27.597 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:27.610 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2023-11-05T08:33:27.617 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:27.615+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:27.717 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-05T08:33:27.778 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:27.778 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:27.792 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2023-11-05T08:33:27.819 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:33:27.817+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:33:27.912 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:27.912 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:27.920 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:27.916+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:27.926 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-11-05T08:33:28.022 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:33:28.018+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:33:28.056 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:28.057 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:28.070 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-11-05T08:33:28.122 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:28.115+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:28.188 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:28.188 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:28.203 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-11-05T08:33:28.223 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:33:28.222+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:33:28.324 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:33:28.320+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:33:28.370 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:28.371 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:28.384 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2023-11-05T08:33:28.431 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:28.427+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:28.467 INFO:tasks.rados.rados.0.smithi090.stdout:237: done (4 left) 2023-11-05T08:33:28.467 INFO:tasks.rados.rados.0.smithi090.stdout:238: copy_from oid 19 from oid 41 current snap is 21 2023-11-05T08:33:28.467 INFO:tasks.rados.rados.0.smithi090.stdout:234: finishing rollback tid 0 to smithi09039252-5 2023-11-05T08:33:28.468 INFO:tasks.rados.rados.0.smithi090.stdout:234: finishing rollback tid 1 to smithi09039252-5 2023-11-05T08:33:28.468 INFO:tasks.rados.rados.0.smithi090.stdout:233: finishing rollback tid 0 to smithi09039252-47 2023-11-05T08:33:28.468 INFO:tasks.rados.rados.0.smithi090.stdout:233: finishing rollback tid 1 to smithi09039252-47 2023-11-05T08:33:28.468 INFO:tasks.rados.rados.0.smithi090.stdout:234: finishing rollback tid 2 to smithi09039252-5 2023-11-05T08:33:28.468 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 5 v 32 (ObjNum 98 snap 17 seq_num 98) dirty exists 2023-11-05T08:33:28.469 INFO:tasks.rados.rados.0.smithi090.stdout:233: finishing rollback tid 2 to smithi09039252-47 2023-11-05T08:33:28.469 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 47 v 19 (ObjNum 61 snap 2 seq_num 61) dirty exists 2023-11-05T08:33:28.469 INFO:tasks.rados.rados.0.smithi090.stdout:235: expect (ObjNum 32 snap 0 seq_num 32) 2023-11-05T08:33:28.504 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:28.504 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:28.507 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-11-05T08:33:28.509 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-05T08:33:28.513 INFO:tasks.rados.rados.0.smithi090.stdout:238: finishing copy_from racing read to smithi09039252-19 2023-11-05T08:33:28.513 INFO:tasks.rados.rados.0.smithi090.stdout:238: finishing copy_from to smithi09039252-19 2023-11-05T08:33:28.513 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 19 v 24 (ObjNum 40 snap 0 seq_num 40) dirty exists 2023-11-05T08:33:28.513 INFO:tasks.rados.rados.0.smithi090.stdout:233: done (4 left) 2023-11-05T08:33:28.514 INFO:tasks.rados.rados.0.smithi090.stdout:234: done (3 left) 2023-11-05T08:33:28.514 INFO:tasks.rados.rados.0.smithi090.stdout:235: done (2 left) 2023-11-05T08:33:28.514 INFO:tasks.rados.rados.0.smithi090.stdout:238: done (1 left) 2023-11-05T08:33:28.514 INFO:tasks.rados.rados.0.smithi090.stdout:239: snap_remove snap 18 2023-11-05T08:33:28.519 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2023-11-05T08:33:28.525 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:33:28.520+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:33:28.626 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:33:28.624+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:33:28.677 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-11-05T08:33:28.683 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:28.683 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:28.698 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2023-11-05T08:33:28.700 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1: {} 2023-11-05T08:33:28.700 INFO:teuthology.orchestra.run.smithi002.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-11-05T08:33:28.720 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2: {} 2023-11-05T08:33:28.721 INFO:teuthology.orchestra.run.smithi002.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-11-05T08:33:28.721 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-11-05T08:33:28.727 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:28.724+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:28.737 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4: {} 2023-11-05T08:33:28.737 INFO:teuthology.orchestra.run.smithi002.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-11-05T08:33:28.761 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5: {} 2023-11-05T08:33:28.761 INFO:teuthology.orchestra.run.smithi002.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-11-05T08:33:28.781 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6: {} 2023-11-05T08:33:28.782 INFO:teuthology.orchestra.run.smithi002.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-11-05T08:33:28.799 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7: {} 2023-11-05T08:33:28.799 INFO:teuthology.orchestra.run.smithi002.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-11-05T08:33:28.816 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8: {} 2023-11-05T08:33:28.816 INFO:teuthology.orchestra.run.smithi002.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-11-05T08:33:28.820 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:28.820 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:28.828 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:33:28.825+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:33:28.834 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2023-11-05T08:33:28.837 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9: {} 2023-11-05T08:33:28.837 INFO:teuthology.orchestra.run.smithi002.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-11-05T08:33:28.857 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10: {} 2023-11-05T08:33:28.857 INFO:teuthology.orchestra.run.smithi002.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-11-05T08:33:28.877 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11: {} 2023-11-05T08:33:28.877 INFO:teuthology.orchestra.run.smithi002.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-11-05T08:33:28.897 INFO:teuthology.orchestra.run.smithi002.stdout:osd.12: {} 2023-11-05T08:33:28.897 INFO:teuthology.orchestra.run.smithi002.stderr:osd.12: osd_enable_op_tracker = 'true' 2023-11-05T08:33:28.921 INFO:teuthology.orchestra.run.smithi002.stdout:osd.13: {} 2023-11-05T08:33:28.922 INFO:teuthology.orchestra.run.smithi002.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-11-05T08:33:28.929 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:33:28.925+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:33:28.941 INFO:teuthology.orchestra.run.smithi002.stdout:osd.14: {} 2023-11-05T08:33:28.942 INFO:teuthology.orchestra.run.smithi002.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-11-05T08:33:28.955 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:28.956 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:28.963 INFO:teuthology.orchestra.run.smithi002.stdout:osd.15: {} 2023-11-05T08:33:28.963 INFO:teuthology.orchestra.run.smithi002.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-11-05T08:33:28.967 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-05T08:33:28.968 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-05T08:33:28.969 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2023-11-05T08:33:29.029 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:33:29.025+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:33:29.086 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:33:29.087 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-11-05T08:33:29.087 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-11-05T08:33:29.087 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-11-05T08:33:29.087 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:33:29.100 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2023-11-05T08:33:29.143 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:29.139+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:29.224 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:33:29.225 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-11-05T08:33:29.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-11-05T08:33:29.225 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-11-05T08:33:29.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.226 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.3s3 36/30 ECSubWrite(tid=183, reqid=osd.13.0:183, at_version=36'12, trim_to=0'0, roll_forward_to=35'10))", 2023-11-05T08:33:29.226 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.072455+0000", 2023-11-05T08:33:29.226 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.148375690000002, 2023-11-05T08:33:29.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0063400499999999999, 2023-11-05T08:33:29.227 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:29.227 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:29.227 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:29.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:29.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072455+0000", 2023-11-05T08:33:29.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:29.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072455+0000", 2023-11-05T08:33:29.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:29.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072456+0000", 2023-11-05T08:33:29.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9999999999999998e-07 2023-11-05T08:33:29.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:29.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072512+0000", 2023-11-05T08:33:29.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5667000000000001e-05 2023-11-05T08:33:29.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:29.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072537+0000", 2023-11-05T08:33:29.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5833000000000002e-05 2023-11-05T08:33:29.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:29.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072581+0000", 2023-11-05T08:33:29.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3470999999999998e-05 2023-11-05T08:33:29.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:29.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072752+0000", 2023-11-05T08:33:29.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017130100000000001 2023-11-05T08:33:29.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:29.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072775+0000", 2023-11-05T08:33:29.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3379000000000001e-05 2023-11-05T08:33:29.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:29.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.078646+0000", 2023-11-05T08:33:29.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0058709419999999997 2023-11-05T08:33:29.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:29.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.078795+0000", 2023-11-05T08:33:29.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014895699999999999 2023-11-05T08:33:29.237 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.237 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:29.237 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.238 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.8s5 36/30 ECSubWrite(tid=23, reqid=osd.5.0:23, at_version=36'15, trim_to=0'0, roll_forward_to=34'13))", 2023-11-05T08:33:29.238 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.079609+0000", 2023-11-05T08:33:29.238 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.141221987999998, 2023-11-05T08:33:29.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049579109999999997, 2023-11-05T08:33:29.239 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:29.239 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:29.239 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:29.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:29.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079609+0000", 2023-11-05T08:33:29.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:29.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079609+0000", 2023-11-05T08:33:29.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:29.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079609+0000", 2023-11-05T08:33:29.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.27e-07 2023-11-05T08:33:29.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:29.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079645+0000", 2023-11-05T08:33:29.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5522000000000003e-05 2023-11-05T08:33:29.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:29.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079672+0000", 2023-11-05T08:33:29.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6648e-05 2023-11-05T08:33:29.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:29.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079693+0000", 2023-11-05T08:33:29.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1206e-05 2023-11-05T08:33:29.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:29.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079849+0000", 2023-11-05T08:33:29.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156112 2023-11-05T08:33:29.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:29.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079873+0000", 2023-11-05T08:33:29.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3589000000000001e-05 2023-11-05T08:33:29.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:29.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.084477+0000", 2023-11-05T08:33:29.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0046040969999999997 2023-11-05T08:33:29.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:29.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.084567+0000", 2023-11-05T08:33:29.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0309999999999994e-05 2023-11-05T08:33:29.249 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.249 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:29.249 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.250 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.8s5 36/30 ECSubWrite(tid=24, reqid=osd.5.0:23, at_version=0'0, trim_to=0'0, roll_forward_to=36'15))", 2023-11-05T08:33:29.250 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.088806+0000", 2023-11-05T08:33:29.250 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.132024717999997, 2023-11-05T08:33:29.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055977739999999998, 2023-11-05T08:33:29.250 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:29.251 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:29.251 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:29.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:29.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.088806+0000", 2023-11-05T08:33:29.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:29.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.088806+0000", 2023-11-05T08:33:29.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:29.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.092263+0000", 2023-11-05T08:33:29.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034565210000000002 2023-11-05T08:33:29.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:29.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.092287+0000", 2023-11-05T08:33:29.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4505999999999999e-05 2023-11-05T08:33:29.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:29.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.092314+0000", 2023-11-05T08:33:29.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6335000000000001e-05 2023-11-05T08:33:29.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:29.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.092329+0000", 2023-11-05T08:33:29.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5486e-05 2023-11-05T08:33:29.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:29.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.092519+0000", 2023-11-05T08:33:29.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019026700000000001 2023-11-05T08:33:29.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:29.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.092545+0000", 2023-11-05T08:33:29.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5644999999999999e-05 2023-11-05T08:33:29.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:29.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.094327+0000", 2023-11-05T08:33:29.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001781543 2023-11-05T08:33:29.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:29.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.094404+0000", 2023-11-05T08:33:29.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7471000000000004e-05 2023-11-05T08:33:29.260 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.260 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:29.261 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.261 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.2s4 37/30 ECSubWrite(tid=185, reqid=client.4691.0:240, at_version=37'7, trim_to=0'0, roll_forward_to=33'5))", 2023-11-05T08:33:29.261 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:50.088409+0000", 2023-11-05T08:33:29.262 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.132422515999998, 2023-11-05T08:33:29.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036490530000000002, 2023-11-05T08:33:29.262 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:29.262 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:29.262 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:29.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:29.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.088409+0000", 2023-11-05T08:33:29.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:29.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.088409+0000", 2023-11-05T08:33:29.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:29.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.088409+0000", 2023-11-05T08:33:29.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.569999999999999e-07 2023-11-05T08:33:29.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:29.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.088461+0000", 2023-11-05T08:33:29.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1665000000000001e-05 2023-11-05T08:33:29.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:29.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.088486+0000", 2023-11-05T08:33:29.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4757999999999999e-05 2023-11-05T08:33:29.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:29.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.088505+0000", 2023-11-05T08:33:29.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8885999999999998e-05 2023-11-05T08:33:29.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:29.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.088651+0000", 2023-11-05T08:33:29.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014625900000000001 2023-11-05T08:33:29.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:29.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.088678+0000", 2023-11-05T08:33:29.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7229e-05 2023-11-05T08:33:29.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:29.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.091969+0000", 2023-11-05T08:33:29.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032904900000000001 2023-11-05T08:33:29.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:29.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.092058+0000", 2023-11-05T08:33:29.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8808999999999996e-05 2023-11-05T08:33:29.272 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.272 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:29.272 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.273 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.2s4 41/40 ECSubWrite(tid=187, reqid=client.4691.0:292, at_version=41'9, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:29.273 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:54.162954+0000", 2023-11-05T08:33:29.273 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 35.057876594, 2023-11-05T08:33:29.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0045082990000000003, 2023-11-05T08:33:29.274 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:29.274 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:29.274 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:29.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:29.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.162954+0000", 2023-11-05T08:33:29.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:29.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.162954+0000", 2023-11-05T08:33:29.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:29.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.162956+0000", 2023-11-05T08:33:29.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.86e-06 2023-11-05T08:33:29.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:29.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.163012+0000", 2023-11-05T08:33:29.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5744e-05 2023-11-05T08:33:29.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:29.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.163048+0000", 2023-11-05T08:33:29.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6346000000000001e-05 2023-11-05T08:33:29.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:29.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.163075+0000", 2023-11-05T08:33:29.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6622e-05 2023-11-05T08:33:29.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:29.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.163236+0000", 2023-11-05T08:33:29.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016064599999999999 2023-11-05T08:33:29.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:29.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.163262+0000", 2023-11-05T08:33:29.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6356e-05 2023-11-05T08:33:29.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:29.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.167302+0000", 2023-11-05T08:33:29.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040403000000000001 2023-11-05T08:33:29.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:29.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.167463+0000", 2023-11-05T08:33:29.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016042500000000001 2023-11-05T08:33:29.284 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.284 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:29.284 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.285 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.2s4 41/40 ECSubWrite(tid=188, reqid=osd.13.0:188, at_version=41'11, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:29.285 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:54.171988+0000", 2023-11-05T08:33:29.285 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 35.048843064000003, 2023-11-05T08:33:29.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037888499999999999, 2023-11-05T08:33:29.286 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:29.286 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:29.286 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:29.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:29.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.171988+0000", 2023-11-05T08:33:29.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:29.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.171988+0000", 2023-11-05T08:33:29.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:29.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.171988+0000", 2023-11-05T08:33:29.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1400000000000001e-07 2023-11-05T08:33:29.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:29.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.172004+0000", 2023-11-05T08:33:29.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6308e-05 2023-11-05T08:33:29.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:29.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.172011+0000", 2023-11-05T08:33:29.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9430000000000003e-06 2023-11-05T08:33:29.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:29.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.172018+0000", 2023-11-05T08:33:29.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6250000000000001e-06 2023-11-05T08:33:29.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:29.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.172082+0000", 2023-11-05T08:33:29.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4311000000000001e-05 2023-11-05T08:33:29.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:29.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.172093+0000", 2023-11-05T08:33:29.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1045e-05 2023-11-05T08:33:29.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:29.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.175705+0000", 2023-11-05T08:33:29.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036118819999999999 2023-11-05T08:33:29.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:29.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.175777+0000", 2023-11-05T08:33:29.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1521999999999997e-05 2023-11-05T08:33:29.296 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.297 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:29.297 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.297 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.13s3 41/40 ECSubWrite(tid=190, reqid=osd.13.0:190, at_version=41'14, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:29.298 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:54.189971+0000", 2023-11-05T08:33:29.298 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 35.030859861000003, 2023-11-05T08:33:29.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003736746, 2023-11-05T08:33:29.298 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:29.299 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:29.299 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:29.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:29.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.189971+0000", 2023-11-05T08:33:29.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:29.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.189971+0000", 2023-11-05T08:33:29.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:29.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.189972+0000", 2023-11-05T08:33:29.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4900000000000005e-07 2023-11-05T08:33:29.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:29.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.190023+0000", 2023-11-05T08:33:29.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1162999999999998e-05 2023-11-05T08:33:29.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:29.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.190044+0000", 2023-11-05T08:33:29.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0761999999999999e-05 2023-11-05T08:33:29.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:29.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.190062+0000", 2023-11-05T08:33:29.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7977000000000001e-05 2023-11-05T08:33:29.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:29.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.190138+0000", 2023-11-05T08:33:29.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5907000000000004e-05 2023-11-05T08:33:29.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:29.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.190148+0000", 2023-11-05T08:33:29.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0344e-05 2023-11-05T08:33:29.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:29.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.193643+0000", 2023-11-05T08:33:29.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003494718 2023-11-05T08:33:29.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:29.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.193708+0000", 2023-11-05T08:33:29.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5025999999999997e-05 2023-11-05T08:33:29.308 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.309 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:29.309 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.310 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.2s4 42/40 ECSubWrite(tid=192, reqid=osd.13.0:192, at_version=42'13, trim_to=0'0, roll_forward_to=41'11))", 2023-11-05T08:33:29.310 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:55.150758+0000", 2023-11-05T08:33:29.310 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.070073505000003, 2023-11-05T08:33:29.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032901889999999998, 2023-11-05T08:33:29.310 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:29.311 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:29.311 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:29.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:29.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.150758+0000", 2023-11-05T08:33:29.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:29.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.150758+0000", 2023-11-05T08:33:29.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:29.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.150758+0000", 2023-11-05T08:33:29.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9999999999999998e-07 2023-11-05T08:33:29.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:29.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.150792+0000", 2023-11-05T08:33:29.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4369999999999998e-05 2023-11-05T08:33:29.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:29.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.150820+0000", 2023-11-05T08:33:29.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7285999999999999e-05 2023-11-05T08:33:29.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:29.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.150833+0000", 2023-11-05T08:33:29.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3603e-05 2023-11-05T08:33:29.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:29.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.150981+0000", 2023-11-05T08:33:29.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000148028 2023-11-05T08:33:29.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:29.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.151008+0000", 2023-11-05T08:33:29.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7169e-05 2023-11-05T08:33:29.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:29.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.153963+0000", 2023-11-05T08:33:29.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029548700000000001 2023-11-05T08:33:29.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:29.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.154048+0000", 2023-11-05T08:33:29.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4363000000000005e-05 2023-11-05T08:33:29.320 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.320 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:29.320 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.321 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.16s1 46/45 ECSubWrite(tid=45, reqid=osd.11.0:45, at_version=46'11, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:29.321 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:59.219281+0000", 2023-11-05T08:33:29.321 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.001549578999999, 2023-11-05T08:33:29.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.005540456, 2023-11-05T08:33:29.322 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:29.322 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:29.322 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:29.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:29.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219281+0000", 2023-11-05T08:33:29.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:29.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219281+0000", 2023-11-05T08:33:29.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:29.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219283+0000", 2023-11-05T08:33:29.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.637e-06 2023-11-05T08:33:29.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:29.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219344+0000", 2023-11-05T08:33:29.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0831000000000001e-05 2023-11-05T08:33:29.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:29.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219376+0000", 2023-11-05T08:33:29.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1945e-05 2023-11-05T08:33:29.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:29.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219417+0000", 2023-11-05T08:33:29.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1335000000000003e-05 2023-11-05T08:33:29.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:29.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219601+0000", 2023-11-05T08:33:29.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018345699999999999 2023-11-05T08:33:29.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:29.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219627+0000", 2023-11-05T08:33:29.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6732000000000001e-05 2023-11-05T08:33:29.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:29.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.224749+0000", 2023-11-05T08:33:29.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0051213129999999997 2023-11-05T08:33:29.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:29.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.224822+0000", 2023-11-05T08:33:29.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3206000000000003e-05 2023-11-05T08:33:29.331 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.331 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:29.331 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.332 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.11s2 46/45 ECSubWrite(tid=0, reqid=osd.10.0:0, at_version=46'17, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:29.332 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:59.222006+0000", 2023-11-05T08:33:29.332 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.998824887000001, 2023-11-05T08:33:29.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053541609999999996, 2023-11-05T08:33:29.332 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:29.333 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:29.333 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:29.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:29.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222006+0000", 2023-11-05T08:33:29.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:29.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222006+0000", 2023-11-05T08:33:29.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:29.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222008+0000", 2023-11-05T08:33:29.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7430000000000001e-06 2023-11-05T08:33:29.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:29.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222062+0000", 2023-11-05T08:33:29.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3891000000000003e-05 2023-11-05T08:33:29.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:29.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222095+0000", 2023-11-05T08:33:29.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3148999999999997e-05 2023-11-05T08:33:29.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:29.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222118+0000", 2023-11-05T08:33:29.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3258999999999999e-05 2023-11-05T08:33:29.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:29.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222279+0000", 2023-11-05T08:33:29.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016113 2023-11-05T08:33:29.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:29.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222303+0000", 2023-11-05T08:33:29.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4088999999999999e-05 2023-11-05T08:33:29.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:29.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.227275+0000", 2023-11-05T08:33:29.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049715599999999999 2023-11-05T08:33:29.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:29.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.227360+0000", 2023-11-05T08:33:29.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5340000000000006e-05 2023-11-05T08:33:29.342 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.342 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:29.343 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.343 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.11s2 46/45 ECSubWrite(tid=1, reqid=osd.10.0:0, at_version=0'0, trim_to=0'0, roll_forward_to=46'17))", 2023-11-05T08:33:29.343 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:59.232093+0000", 2023-11-05T08:33:29.343 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.988738411, 2023-11-05T08:33:29.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037473570000000002, 2023-11-05T08:33:29.344 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:29.344 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:29.344 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:29.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:29.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232093+0000", 2023-11-05T08:33:29.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:29.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232093+0000", 2023-11-05T08:33:29.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:29.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232094+0000", 2023-11-05T08:33:29.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0380000000000001e-06 2023-11-05T08:33:29.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:29.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232122+0000", 2023-11-05T08:33:29.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8585000000000002e-05 2023-11-05T08:33:29.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:29.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232142+0000", 2023-11-05T08:33:29.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9357000000000001e-05 2023-11-05T08:33:29.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:29.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232157+0000", 2023-11-05T08:33:29.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5141e-05 2023-11-05T08:33:29.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:29.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232272+0000", 2023-11-05T08:33:29.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114945 2023-11-05T08:33:29.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.352 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:29.352 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232285+0000", 2023-11-05T08:33:29.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2917e-05 2023-11-05T08:33:29.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.352 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:29.353 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.235777+0000", 2023-11-05T08:33:29.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003492069 2023-11-05T08:33:29.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.353 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:29.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.235840+0000", 2023-11-05T08:33:29.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3305000000000005e-05 2023-11-05T08:33:29.354 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.354 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:29.354 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.355 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.16s1 46/45 ECSubWrite(tid=47, reqid=osd.11.0:47, at_version=46'13, trim_to=0'0, roll_forward_to=46'11))", 2023-11-05T08:33:29.355 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:59.233948+0000", 2023-11-05T08:33:29.355 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.986882657999999, 2023-11-05T08:33:29.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048084130000000001, 2023-11-05T08:33:29.356 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:29.356 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:29.356 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:29.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:29.356 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.233948+0000", 2023-11-05T08:33:29.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:29.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.233948+0000", 2023-11-05T08:33:29.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:29.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.233950+0000", 2023-11-05T08:33:29.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5969999999999999e-06 2023-11-05T08:33:29.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:29.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.234005+0000", 2023-11-05T08:33:29.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5414999999999997e-05 2023-11-05T08:33:29.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:29.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.234040+0000", 2023-11-05T08:33:29.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.455e-05 2023-11-05T08:33:29.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:29.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.234061+0000", 2023-11-05T08:33:29.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1036e-05 2023-11-05T08:33:29.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:29.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.234178+0000", 2023-11-05T08:33:29.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000116822 2023-11-05T08:33:29.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:29.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.234195+0000", 2023-11-05T08:33:29.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6801999999999999e-05 2023-11-05T08:33:29.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:29.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.238676+0000", 2023-11-05T08:33:29.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004481044 2023-11-05T08:33:29.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:29.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.238757+0000", 2023-11-05T08:33:29.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1147000000000001e-05 2023-11-05T08:33:29.365 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.365 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:29.366 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.366 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.16s1 46/45 ECSubWrite(tid=48, reqid=osd.11.0:47, at_version=0'0, trim_to=0'0, roll_forward_to=46'13))", 2023-11-05T08:33:29.366 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:59.240171+0000", 2023-11-05T08:33:29.367 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.980660051000001, 2023-11-05T08:33:29.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0045632529999999998, 2023-11-05T08:33:29.367 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:29.367 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:29.368 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:29.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:29.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240171+0000", 2023-11-05T08:33:29.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:29.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240171+0000", 2023-11-05T08:33:29.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:29.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240172+0000", 2023-11-05T08:33:29.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.282e-06 2023-11-05T08:33:29.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:29.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240193+0000", 2023-11-05T08:33:29.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0601000000000001e-05 2023-11-05T08:33:29.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:29.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240214+0000", 2023-11-05T08:33:29.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1574999999999999e-05 2023-11-05T08:33:29.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:29.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240230+0000", 2023-11-05T08:33:29.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5656999999999999e-05 2023-11-05T08:33:29.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:29.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240361+0000", 2023-11-05T08:33:29.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000130488 2023-11-05T08:33:29.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:29.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240372+0000", 2023-11-05T08:33:29.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1158999999999999e-05 2023-11-05T08:33:29.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:29.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.244663+0000", 2023-11-05T08:33:29.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004291563 2023-11-05T08:33:29.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:29.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.244734+0000", 2023-11-05T08:33:29.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0927999999999996e-05 2023-11-05T08:33:29.377 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.377 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:29.377 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.378 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.16s1 51/45 ECSubWrite(tid=58, reqid=osd.11.0:58, at_version=51'19, trim_to=0'0, roll_forward_to=49'17))", 2023-11-05T08:33:29.378 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:04.226221+0000", 2023-11-05T08:33:29.378 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 24.994610279, 2023-11-05T08:33:29.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.005713414, 2023-11-05T08:33:29.378 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:29.378 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:29.379 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:29.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:29.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226221+0000", 2023-11-05T08:33:29.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:29.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226221+0000", 2023-11-05T08:33:29.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:29.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226221+0000", 2023-11-05T08:33:29.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.53e-07 2023-11-05T08:33:29.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:29.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226253+0000", 2023-11-05T08:33:29.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1671999999999998e-05 2023-11-05T08:33:29.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:29.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226269+0000", 2023-11-05T08:33:29.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6150999999999998e-05 2023-11-05T08:33:29.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:29.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226282+0000", 2023-11-05T08:33:29.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3465000000000001e-05 2023-11-05T08:33:29.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:29.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226474+0000", 2023-11-05T08:33:29.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019140499999999999 2023-11-05T08:33:29.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:29.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226501+0000", 2023-11-05T08:33:29.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7523000000000001e-05 2023-11-05T08:33:29.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:29.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.231902+0000", 2023-11-05T08:33:29.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0054011019999999996 2023-11-05T08:33:29.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:29.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.231934+0000", 2023-11-05T08:33:29.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1742999999999997e-05 2023-11-05T08:33:29.388 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.388 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:29.388 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.389 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.9s4 52/45 ECSubWrite(tid=75, reqid=client.4691.0:341, at_version=52'24, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:29.389 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:05.257804+0000", 2023-11-05T08:33:29.389 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.963027091000001, 2023-11-05T08:33:29.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010353123000000001, 2023-11-05T08:33:29.390 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:29.390 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:29.390 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:29.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:29.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.257804+0000", 2023-11-05T08:33:29.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:29.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.257804+0000", 2023-11-05T08:33:29.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:29.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.257805+0000", 2023-11-05T08:33:29.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3399999999999999e-07 2023-11-05T08:33:29.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:29.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.258142+0000", 2023-11-05T08:33:29.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033764100000000002 2023-11-05T08:33:29.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:29.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.258157+0000", 2023-11-05T08:33:29.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4625e-05 2023-11-05T08:33:29.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:29.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.258168+0000", 2023-11-05T08:33:29.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0992000000000001e-05 2023-11-05T08:33:29.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:29.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.258371+0000", 2023-11-05T08:33:29.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020296299999999999 2023-11-05T08:33:29.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:29.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.258404+0000", 2023-11-05T08:33:29.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2765999999999998e-05 2023-11-05T08:33:29.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:29.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.268067+0000", 2023-11-05T08:33:29.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0096629479999999993 2023-11-05T08:33:29.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:29.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.268157+0000", 2023-11-05T08:33:29.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0554000000000001e-05 2023-11-05T08:33:29.400 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.400 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:29.401 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.401 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(osd.14.22:3 3.5s0 3:a287d13d:::smithi09039252-33:head [assert-version v4,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:29.401 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:05.264331+0000", 2023-11-05T08:33:29.402 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.956500487, 2023-11-05T08:33:29.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010947936, 2023-11-05T08:33:29.402 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:29.402 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:29.403 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:29.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:29.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.264331+0000", 2023-11-05T08:33:29.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:29.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.264331+0000", 2023-11-05T08:33:29.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:29.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.264334+0000", 2023-11-05T08:33:29.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.467e-06 2023-11-05T08:33:29.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:29.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.264350+0000", 2023-11-05T08:33:29.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6200000000000001e-05 2023-11-05T08:33:29.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.407 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:29.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.264387+0000", 2023-11-05T08:33:29.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6684999999999999e-05 2023-11-05T08:33:29.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:29.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.264489+0000", 2023-11-05T08:33:29.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010218 2023-11-05T08:33:29.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:29.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.265518+0000", 2023-11-05T08:33:29.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0010291899999999999 2023-11-05T08:33:29.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-11-05T08:33:29.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.265535+0000", 2023-11-05T08:33:29.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6697e-05 2023-11-05T08:33:29.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:29.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.269530+0000", 2023-11-05T08:33:29.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039951769999999999 2023-11-05T08:33:29.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:29.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.269646+0000", 2023-11-05T08:33:29.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011579299999999999 2023-11-05T08:33:29.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:29.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.275278+0000", 2023-11-05T08:33:29.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0056325469999999999 2023-11-05T08:33:29.413 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.413 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:29.413 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.414 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpReadReply(3.5s0 52/40 ECSubReadReply(tid=27, attrs_read=0))", 2023-11-05T08:33:29.414 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:05.271241+0000", 2023-11-05T08:33:29.414 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.949590071999999, 2023-11-05T08:33:29.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040795400000000004, 2023-11-05T08:33:29.415 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:29.415 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:29.415 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:29.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:29.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.271241+0000", 2023-11-05T08:33:29.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:29.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.271241+0000", 2023-11-05T08:33:29.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:29.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.271242+0000", 2023-11-05T08:33:29.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1370000000000001e-06 2023-11-05T08:33:29.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:29.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.272078+0000", 2023-11-05T08:33:29.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00083553299999999996 2023-11-05T08:33:29.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:29.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.272107+0000", 2023-11-05T08:33:29.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9195000000000001e-05 2023-11-05T08:33:29.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:29.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.272124+0000", 2023-11-05T08:33:29.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6959000000000001e-05 2023-11-05T08:33:29.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:29.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.272391+0000", 2023-11-05T08:33:29.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026688200000000002 2023-11-05T08:33:29.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:29.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.275320+0000", 2023-11-05T08:33:29.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029298340000000001 2023-11-05T08:33:29.423 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.423 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:29.423 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.424 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.16s1 56/45 ECSubWrite(tid=63, reqid=client.4691.0:369, at_version=56'20, trim_to=0'0, roll_forward_to=51'19))", 2023-11-05T08:33:29.424 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:09.292024+0000", 2023-11-05T08:33:29.424 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 19.928806711, 2023-11-05T08:33:29.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0035291490000000001, 2023-11-05T08:33:29.425 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:29.425 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:29.425 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:29.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:29.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.292024+0000", 2023-11-05T08:33:29.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:29.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.292024+0000", 2023-11-05T08:33:29.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:29.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.292025+0000", 2023-11-05T08:33:29.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.083e-06 2023-11-05T08:33:29.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:29.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.292178+0000", 2023-11-05T08:33:29.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015233600000000001 2023-11-05T08:33:29.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:29.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.292213+0000", 2023-11-05T08:33:29.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4779e-05 2023-11-05T08:33:29.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:29.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.292239+0000", 2023-11-05T08:33:29.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6758999999999999e-05 2023-11-05T08:33:29.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:29.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.292487+0000", 2023-11-05T08:33:29.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024808600000000001 2023-11-05T08:33:29.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:29.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.292519+0000", 2023-11-05T08:33:29.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2125999999999997e-05 2023-11-05T08:33:29.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:29.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.295403+0000", 2023-11-05T08:33:29.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028833539999999999 2023-11-05T08:33:29.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:29.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.295553+0000", 2023-11-05T08:33:29.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000150626 2023-11-05T08:33:29.434 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.434 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:29.434 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.435 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.16s1 56/45 ECSubWrite(tid=64, reqid=client.4691.0:370, at_version=56'21, trim_to=0'0, roll_forward_to=51'19))", 2023-11-05T08:33:29.435 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:09.297268+0000", 2023-11-05T08:33:29.435 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 19.923562674999999, 2023-11-05T08:33:29.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053142609999999998, 2023-11-05T08:33:29.435 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:29.436 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:29.436 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:29.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:29.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297268+0000", 2023-11-05T08:33:29.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:29.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297268+0000", 2023-11-05T08:33:29.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:29.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297270+0000", 2023-11-05T08:33:29.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7570000000000001e-06 2023-11-05T08:33:29.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:29.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297426+0000", 2023-11-05T08:33:29.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015544199999999999 2023-11-05T08:33:29.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:29.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297457+0000", 2023-11-05T08:33:29.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1615999999999997e-05 2023-11-05T08:33:29.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:29.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297480+0000", 2023-11-05T08:33:29.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3326e-05 2023-11-05T08:33:29.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:29.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.299414+0000", 2023-11-05T08:33:29.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001933518 2023-11-05T08:33:29.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:29.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.299445+0000", 2023-11-05T08:33:29.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0701999999999998e-05 2023-11-05T08:33:29.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:29.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.302505+0000", 2023-11-05T08:33:29.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030607590000000001 2023-11-05T08:33:29.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:29.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.302583+0000", 2023-11-05T08:33:29.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7140999999999999e-05 2023-11-05T08:33:29.445 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.445 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:29.445 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.446 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.16s1 56/45 ECSubWrite(tid=65, reqid=client.4691.0:371, at_version=56'22, trim_to=0'0, roll_forward_to=56'20))", 2023-11-05T08:33:29.446 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:09.304840+0000", 2023-11-05T08:33:29.446 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 19.915990999000002, 2023-11-05T08:33:29.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032268710000000001, 2023-11-05T08:33:29.447 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:29.447 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:29.447 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:29.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:29.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.304840+0000", 2023-11-05T08:33:29.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:29.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.304840+0000", 2023-11-05T08:33:29.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:29.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:29.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.304842+0000", 2023-11-05T08:33:29.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5939999999999999e-06 2023-11-05T08:33:29.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:29.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.304981+0000", 2023-11-05T08:33:29.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013962600000000001 2023-11-05T08:33:29.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:29.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.305009+0000", 2023-11-05T08:33:29.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7426000000000001e-05 2023-11-05T08:33:29.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:29.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.305024+0000", 2023-11-05T08:33:29.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5449000000000001e-05 2023-11-05T08:33:29.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:29.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.305212+0000", 2023-11-05T08:33:29.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000188252 2023-11-05T08:33:29.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:29.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.305246+0000", 2023-11-05T08:33:29.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3321999999999997e-05 2023-11-05T08:33:29.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:29.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.307927+0000", 2023-11-05T08:33:29.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026809889999999999 2023-11-05T08:33:29.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:29.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:29.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:29.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.308067+0000", 2023-11-05T08:33:29.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014021300000000001 2023-11-05T08:33:29.457 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.457 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:29.457 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.457 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:29.457 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:29.458 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:33:29.459 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-11-05T08:33:29.460 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:33:29.457+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:33:29.479 INFO:teuthology.orchestra.run.smithi002.stderr:noscrub is set 2023-11-05T08:33:29.487 INFO:tasks.rados.rados.0.smithi090.stdout:239: done (1 left) 2023-11-05T08:33:29.488 INFO:tasks.rados.rados.0.smithi090.stdout:240: rollback oid 23 current snap is 21 2023-11-05T08:33:29.488 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 23 to 4 2023-11-05T08:33:29.488 INFO:tasks.rados.rados.0.smithi090.stdout:241: rmattr oid 36 current snap is 21 2023-11-05T08:33:29.488 INFO:tasks.rados.rados.0.smithi090.stdout:242: delete oid 42 current snap is 21 2023-11-05T08:33:29.491 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2023-11-05T08:33:29.500 INFO:tasks.rados.rados.0.smithi090.stdout:240: finishing rollback tid 0 to smithi09039252-23 2023-11-05T08:33:29.504 INFO:tasks.rados.rados.0.smithi090.stdout:241: done (3 left) 2023-11-05T08:33:29.504 INFO:tasks.rados.rados.0.smithi090.stdout:242: done (2 left) 2023-11-05T08:33:29.504 INFO:tasks.rados.rados.0.smithi090.stdout:243: delete oid 5 current snap is 21 2023-11-05T08:33:29.505 INFO:tasks.rados.rados.0.smithi090.stdout:240: finishing rollback tid 1 to smithi09039252-23 2023-11-05T08:33:29.505 INFO:tasks.rados.rados.0.smithi090.stdout:240: finishing rollback tid 2 to smithi09039252-23 2023-11-05T08:33:29.505 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 23 v 21 (ObjNum 60 snap 1 seq_num 60) dirty exists 2023-11-05T08:33:29.512 INFO:tasks.rados.rados.0.smithi090.stdout:240: done (2 left) 2023-11-05T08:33:29.512 INFO:tasks.rados.rados.0.smithi090.stdout:243: done (1 left) 2023-11-05T08:33:29.513 INFO:tasks.rados.rados.0.smithi090.stdout:244: snap_create 2023-11-05T08:33:29.559 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-05T08:33:29.585 INFO:teuthology.orchestra.run.smithi002.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-05T08:33:29.587 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-05T08:33:29.588 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-11-05T08:33:29.661 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:33:29.659+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:33:29.699 INFO:teuthology.orchestra.run.smithi002.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-05T08:33:29.702 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-05T08:33:29.703 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-11-05T08:33:29.763 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:33:29.757+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:33:29.813 INFO:teuthology.orchestra.run.smithi002.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-05T08:33:29.816 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-05T08:33:29.817 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-11-05T08:33:29.908 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:33:29.905+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:33:29.964 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:33:29.961+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:33:29.985 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:33:29.985 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [], 2023-11-05T08:33:29.985 INFO:teuthology.orchestra.run.smithi002.stdout: "num_ops": 0 2023-11-05T08:33:29.986 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:33:30.000 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-11-05T08:33:30.065 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:33:30.064+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:33:30.166 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:33:30.161+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:33:30.173 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:33:30.174 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [], 2023-11-05T08:33:30.174 INFO:teuthology.orchestra.run.smithi002.stdout: "complaint_time": 30, 2023-11-05T08:33:30.174 INFO:teuthology.orchestra.run.smithi002.stdout: "num_blocked_ops": 0 2023-11-05T08:33:30.174 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:33:30.190 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-11-05T08:33:30.267 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:33:30.265+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:33:30.314 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:33:30.314 INFO:teuthology.orchestra.run.smithi002.stdout: "size": 20, 2023-11-05T08:33:30.314 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 600, 2023-11-05T08:33:30.314 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [ 2023-11-05T08:33:30.315 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.315 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.5s2 35/30 ECSubWrite(tid=20, reqid=client.4691.0:221, at_version=35'11, trim_to=0'0, roll_forward_to=33'9))", 2023-11-05T08:33:30.315 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:48.086738+0000", 2023-11-05T08:33:30.315 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 42.223420562999998, 2023-11-05T08:33:30.315 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0065328219999999998, 2023-11-05T08:33:30.316 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:30.316 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:30.316 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:30.316 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.316 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:30.317 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.086738+0000", 2023-11-05T08:33:30.317 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.317 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.317 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.317 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:30.318 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.086738+0000", 2023-11-05T08:33:30.318 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.318 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.318 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.319 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:30.319 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.086739+0000", 2023-11-05T08:33:30.319 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.22e-07 2023-11-05T08:33:30.319 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.319 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.320 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:30.320 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.087297+0000", 2023-11-05T08:33:30.320 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00055798899999999995 2023-11-05T08:33:30.320 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.320 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.321 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:30.321 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.087338+0000", 2023-11-05T08:33:30.321 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.1171000000000003e-05 2023-11-05T08:33:30.321 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.321 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.322 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:30.322 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.087364+0000", 2023-11-05T08:33:30.322 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.5826999999999999e-05 2023-11-05T08:33:30.322 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.322 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.323 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:30.323 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.087572+0000", 2023-11-05T08:33:30.323 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000208574 2023-11-05T08:33:30.323 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.324 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.324 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:30.324 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.087598+0000", 2023-11-05T08:33:30.324 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.5901000000000002e-05 2023-11-05T08:33:30.324 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.325 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.325 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:30.325 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.093224+0000", 2023-11-05T08:33:30.325 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0056260210000000001 2023-11-05T08:33:30.325 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.325 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.326 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:30.326 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:48.093271+0000", 2023-11-05T08:33:30.326 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.7117000000000003e-05 2023-11-05T08:33:30.326 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.326 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:30.327 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.327 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.327 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.327 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:287 3.11s0 3:88a306be:::smithi09039252-16:head [append 0~770048] snapc 8=[8,7,6,5,3] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:33:30.328 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:53.182407+0000", 2023-11-05T08:33:30.328 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 37.127752156, 2023-11-05T08:33:30.328 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.011495974, 2023-11-05T08:33:30.328 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:30.328 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:30.329 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:30.329 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:30.329 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:30.329 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 287 2023-11-05T08:33:30.329 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.329 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:30.330 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.330 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:30.330 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.182407+0000", 2023-11-05T08:33:30.330 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.330 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.331 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.331 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:30.331 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.182407+0000", 2023-11-05T08:33:30.331 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.331 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.332 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.332 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:30.332 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.182409+0000", 2023-11-05T08:33:30.332 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.3590000000000002e-06 2023-11-05T08:33:30.332 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.333 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.333 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:30.333 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.183780+0000", 2023-11-05T08:33:30.333 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00137108 2023-11-05T08:33:30.333 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.333 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.334 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:30.334 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.183800+0000", 2023-11-05T08:33:30.334 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.9819000000000001e-05 2023-11-05T08:33:30.334 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.334 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.335 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:30.335 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.183810+0000", 2023-11-05T08:33:30.335 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.6940000000000007e-06 2023-11-05T08:33:30.335 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.335 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.336 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:30.336 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.183873+0000", 2023-11-05T08:33:30.336 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.3536999999999998e-05 2023-11-05T08:33:30.336 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.337 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.337 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for peered", 2023-11-05T08:33:30.337 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.183877+0000", 2023-11-05T08:33:30.337 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.1779999999999996e-06 2023-11-05T08:33:30.337 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.338 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.338 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:30.338 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.171366+0000", 2023-11-05T08:33:30.338 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.98748831999999997 2023-11-05T08:33:30.338 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.339 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.339 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:30.339 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.171750+0000", 2023-11-05T08:33:30.339 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00038434800000000001 2023-11-05T08:33:30.339 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.340 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.340 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:30.340 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.174288+0000", 2023-11-05T08:33:30.340 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.002537957 2023-11-05T08:33:30.340 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.341 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.341 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:30.341 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.181565+0000", 2023-11-05T08:33:30.341 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0072769530000000001 2023-11-05T08:33:30.341 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.342 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.342 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:33:30.342 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.193786+0000", 2023-11-05T08:33:30.342 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.012221037000000001 2023-11-05T08:33:30.342 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.343 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.343 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:30.343 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.193903+0000", 2023-11-05T08:33:30.343 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000116692 2023-11-05T08:33:30.343 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.344 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:30.344 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.344 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.344 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.344 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:288 3.11s0 3:88a306be:::smithi09039252-16:head [append 0~376832] snapc 8=[8,7,6,5,3] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:33:30.345 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:53.186355+0000", 2023-11-05T08:33:30.345 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 37.123804376999999, 2023-11-05T08:33:30.345 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0078397800000001, 2023-11-05T08:33:30.345 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:30.345 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:30.346 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:30.346 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:30.346 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:30.346 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 288 2023-11-05T08:33:30.347 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.347 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:30.347 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.347 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:30.347 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186355+0000", 2023-11-05T08:33:30.349 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.349 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.349 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.350 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:30.350 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186355+0000", 2023-11-05T08:33:30.350 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.350 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.350 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.351 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:30.351 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186355+0000", 2023-11-05T08:33:30.351 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.9400000000000004e-07 2023-11-05T08:33:30.351 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.351 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.352 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:30.352 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186649+0000", 2023-11-05T08:33:30.352 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00029353199999999999 2023-11-05T08:33:30.352 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.352 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.353 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:30.353 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186662+0000", 2023-11-05T08:33:30.353 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.2728000000000001e-05 2023-11-05T08:33:30.353 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.354 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.354 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:30.354 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186667+0000", 2023-11-05T08:33:30.354 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.8270000000000003e-06 2023-11-05T08:33:30.355 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.355 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.355 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:30.355 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186742+0000", 2023-11-05T08:33:30.355 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.4328999999999999e-05 2023-11-05T08:33:30.356 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.356 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.356 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for peered", 2023-11-05T08:33:30.356 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186744+0000", 2023-11-05T08:33:30.357 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.266e-06 2023-11-05T08:33:30.357 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.357 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.357 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:30.358 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.176861+0000", 2023-11-05T08:33:30.358 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.990116577 2023-11-05T08:33:30.358 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.358 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.358 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:30.359 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.177014+0000", 2023-11-05T08:33:30.359 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000153661 2023-11-05T08:33:30.359 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.359 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.359 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:30.360 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.178253+0000", 2023-11-05T08:33:30.360 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0012384399999999999 2023-11-05T08:33:30.360 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.360 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.361 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:30.361 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.181931+0000", 2023-11-05T08:33:30.361 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0036777730000000001 2023-11-05T08:33:30.361 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.362 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.362 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:33:30.362 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.194093+0000", 2023-11-05T08:33:30.362 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.012162885999999999 2023-11-05T08:33:30.362 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.363 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.363 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:30.363 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.194194+0000", 2023-11-05T08:33:30.363 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00010086699999999999 2023-11-05T08:33:30.364 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.364 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:30.364 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.364 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.364 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.365 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:289 3.11s0 3:88a306be:::smithi09039252-16:head [setxattr _header (56) in=7b] snapc 8=[8,7,6,5,3] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:33:30.365 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:53.186682+0000", 2023-11-05T08:33:30.365 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 37.123477029999997, 2023-11-05T08:33:30.365 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.008829143, 2023-11-05T08:33:30.365 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:30.365 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:30.366 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:30.366 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:30.366 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:30.366 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 289 2023-11-05T08:33:30.366 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.367 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:30.367 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.367 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:30.367 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186682+0000", 2023-11-05T08:33:30.367 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.368 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.368 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.368 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:30.368 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186682+0000", 2023-11-05T08:33:30.369 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.369 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.369 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.369 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:30.369 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186683+0000", 2023-11-05T08:33:30.370 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.8999999999999996e-07 2023-11-05T08:33:30.370 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.370 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.370 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:30.370 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186687+0000", 2023-11-05T08:33:30.371 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.933e-06 2023-11-05T08:33:30.371 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.371 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.371 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:30.372 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186697+0000", 2023-11-05T08:33:30.372 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.6120000000000005e-06 2023-11-05T08:33:30.372 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.372 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.372 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:30.373 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186704+0000", 2023-11-05T08:33:30.373 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.8929999999999999e-06 2023-11-05T08:33:30.373 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.373 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.373 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:30.374 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186817+0000", 2023-11-05T08:33:30.374 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000113229 2023-11-05T08:33:30.374 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.374 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.374 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for peered", 2023-11-05T08:33:30.375 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186819+0000", 2023-11-05T08:33:30.375 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.57e-06 2023-11-05T08:33:30.375 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.375 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.375 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:30.376 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.179928+0000", 2023-11-05T08:33:30.376 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.993109099 2023-11-05T08:33:30.376 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.376 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.377 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:30.377 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.180084+0000", 2023-11-05T08:33:30.377 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00015647 2023-11-05T08:33:30.377 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.377 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.378 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:30.378 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.181012+0000", 2023-11-05T08:33:30.378 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00092730299999999998 2023-11-05T08:33:30.378 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.378 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.379 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:30.379 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.183961+0000", 2023-11-05T08:33:30.379 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0029494510000000001 2023-11-05T08:33:30.379 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.379 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.380 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:33:30.380 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.195373+0000", 2023-11-05T08:33:30.380 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.011411607000000001 2023-11-05T08:33:30.380 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.380 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.380 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:30.381 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.195511+0000", 2023-11-05T08:33:30.381 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000138286 2023-11-05T08:33:30.381 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.381 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:30.381 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.382 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.382 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.382 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:290 3.11s0 3:88a306be:::smithi09039252-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:33:30.382 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:53.186775+0000", 2023-11-05T08:33:30.382 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 37.123383545000003, 2023-11-05T08:33:30.383 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0117209439999999, 2023-11-05T08:33:30.383 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:30.383 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "started", 2023-11-05T08:33:30.383 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:30.383 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:30.384 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:30.384 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 290 2023-11-05T08:33:30.384 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.384 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:30.385 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.385 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:30.385 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186775+0000", 2023-11-05T08:33:30.385 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.385 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.385 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.386 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:30.386 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186775+0000", 2023-11-05T08:33:30.386 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.386 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.387 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.387 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:30.387 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186776+0000", 2023-11-05T08:33:30.387 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.7999999999999993e-07 2023-11-05T08:33:30.387 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.388 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.388 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:30.388 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186780+0000", 2023-11-05T08:33:30.388 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.9249999999999997e-06 2023-11-05T08:33:30.388 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.389 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.389 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:30.389 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186792+0000", 2023-11-05T08:33:30.389 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.1841999999999999e-05 2023-11-05T08:33:30.389 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.390 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.390 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:30.390 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186800+0000", 2023-11-05T08:33:30.390 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.1559999999999995e-06 2023-11-05T08:33:30.390 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.391 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.391 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:30.391 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186855+0000", 2023-11-05T08:33:30.391 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.4892999999999998e-05 2023-11-05T08:33:30.392 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.392 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.392 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for peered", 2023-11-05T08:33:30.392 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186856+0000", 2023-11-05T08:33:30.392 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.1999999999999999e-07 2023-11-05T08:33:30.393 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.393 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.393 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:30.393 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.181764+0000", 2023-11-05T08:33:30.393 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.99490820300000005 2023-11-05T08:33:30.394 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.394 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.394 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for rw locks", 2023-11-05T08:33:30.394 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.181917+0000", 2023-11-05T08:33:30.394 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00015282300000000001 2023-11-05T08:33:30.395 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.395 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.395 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:30.395 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.196083+0000", 2023-11-05T08:33:30.395 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.014166247999999999 2023-11-05T08:33:30.396 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.396 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.396 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:30.396 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.196295+0000", 2023-11-05T08:33:30.396 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00021158599999999999 2023-11-05T08:33:30.396 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.397 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.397 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:30.397 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.198496+0000", 2023-11-05T08:33:30.397 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0022016679999999999 2023-11-05T08:33:30.397 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.398 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:30.398 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.398 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.398 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.398 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:333 3.1s0 3:863368a9:::smithi09039252-6:head [read 0~98304,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2023-11-05T08:33:30.399 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:04.226324+0000", 2023-11-05T08:33:30.399 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 26.083834462999999, 2023-11-05T08:33:30.399 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0064759040000000002, 2023-11-05T08:33:30.399 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:30.400 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "started", 2023-11-05T08:33:30.400 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:30.400 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:30.400 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:30.400 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 333 2023-11-05T08:33:30.401 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.401 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:30.401 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.401 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:30.401 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226324+0000", 2023-11-05T08:33:30.402 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.402 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.402 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.402 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:30.403 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226324+0000", 2023-11-05T08:33:30.403 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.403 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.403 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.403 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:30.403 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226326+0000", 2023-11-05T08:33:30.404 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.243e-06 2023-11-05T08:33:30.404 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.404 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.404 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:30.405 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226335+0000", 2023-11-05T08:33:30.405 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.2560000000000008e-06 2023-11-05T08:33:30.405 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.405 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.405 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:30.406 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226356+0000", 2023-11-05T08:33:30.406 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.1406000000000001e-05 2023-11-05T08:33:30.406 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.406 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.406 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:30.407 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226372+0000", 2023-11-05T08:33:30.407 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.5923e-05 2023-11-05T08:33:30.407 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.407 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.407 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:30.408 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.228444+0000", 2023-11-05T08:33:30.408 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0020712930000000001 2023-11-05T08:33:30.408 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.408 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.408 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:30.409 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.228711+0000", 2023-11-05T08:33:30.409 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00026721699999999999 2023-11-05T08:33:30.409 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.409 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.409 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:30.410 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.232800+0000", 2023-11-05T08:33:30.410 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0040895660000000002 2023-11-05T08:33:30.410 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.410 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:30.410 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.411 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.411 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.411 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:334 3.1s0 3:863368a9:::smithi09039252-6:head [read 0~98304,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2023-11-05T08:33:30.411 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:04.226390+0000", 2023-11-05T08:33:30.411 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 26.083769262000001, 2023-11-05T08:33:30.412 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.017587591999999999, 2023-11-05T08:33:30.412 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:30.412 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "started", 2023-11-05T08:33:30.412 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:30.412 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:30.413 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:30.413 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 334 2023-11-05T08:33:30.413 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.413 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:30.413 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.414 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:30.414 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226390+0000", 2023-11-05T08:33:30.414 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.414 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.414 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.415 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:30.415 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226390+0000", 2023-11-05T08:33:30.415 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.415 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.415 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.416 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:30.416 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226390+0000", 2023-11-05T08:33:30.416 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.1999999999999998e-07 2023-11-05T08:33:30.416 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.416 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.417 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:30.417 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226396+0000", 2023-11-05T08:33:30.417 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.3229999999999997e-06 2023-11-05T08:33:30.417 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.417 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.418 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:30.418 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226409+0000", 2023-11-05T08:33:30.418 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.3096e-05 2023-11-05T08:33:30.418 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.418 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.419 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:30.419 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226417+0000", 2023-11-05T08:33:30.419 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.1300000000000001e-06 2023-11-05T08:33:30.419 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.419 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.420 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:30.420 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.233238+0000", 2023-11-05T08:33:30.420 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0068212999999999998 2023-11-05T08:33:30.420 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.420 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.421 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:30.421 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.233474+0000", 2023-11-05T08:33:30.421 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000235286 2023-11-05T08:33:30.421 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.421 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.422 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:30.422 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.243977+0000", 2023-11-05T08:33:30.422 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.010503637 2023-11-05T08:33:30.422 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.422 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:30.423 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.423 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.423 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.423 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:367 3.bs0 3:d0f842ed:::smithi09039252-18:head [read 0~294912,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2023-11-05T08:33:30.424 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:08.269934+0000", 2023-11-05T08:33:30.424 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 22.040224637000001, 2023-11-05T08:33:30.424 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.007346366, 2023-11-05T08:33:30.424 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:30.424 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "started", 2023-11-05T08:33:30.425 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:30.425 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:30.425 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:30.425 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 367 2023-11-05T08:33:30.425 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.426 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:30.426 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.426 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:30.426 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.269934+0000", 2023-11-05T08:33:30.426 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.427 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.427 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.427 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:30.427 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.269934+0000", 2023-11-05T08:33:30.427 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.428 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.428 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.428 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:30.428 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.269935+0000", 2023-11-05T08:33:30.428 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.03e-07 2023-11-05T08:33:30.429 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.429 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.429 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:30.429 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.269937+0000", 2023-11-05T08:33:30.429 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.283e-06 2023-11-05T08:33:30.430 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.430 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.430 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:30.430 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.269942+0000", 2023-11-05T08:33:30.430 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.9219999999999996e-06 2023-11-05T08:33:30.431 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.431 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.431 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:30.431 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.269945+0000", 2023-11-05T08:33:30.431 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.3519999999999998e-06 2023-11-05T08:33:30.432 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.432 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.432 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:30.432 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.270155+0000", 2023-11-05T08:33:30.433 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00021031899999999999 2023-11-05T08:33:30.433 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.433 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.433 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:33:30.433 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.270181+0000", 2023-11-05T08:33:30.434 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.5415999999999999e-05 2023-11-05T08:33:30.434 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.434 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.434 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:30.434 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.275639+0000", 2023-11-05T08:33:30.435 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00545788 2023-11-05T08:33:30.435 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.435 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.435 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:30.435 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.275752+0000", 2023-11-05T08:33:30.436 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000113107 2023-11-05T08:33:30.436 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.436 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.436 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:30.436 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:08.277281+0000", 2023-11-05T08:33:30.437 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001528784 2023-11-05T08:33:30.437 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.437 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:30.437 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.437 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.438 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.438 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:416 3.1s0 3:8539e931:::smithi09039252-41:head [append 0~671744] snapc 1c=[1c,1b,1a,17,7] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2023-11-05T08:33:30.438 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:15.354023+0000", 2023-11-05T08:33:30.438 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 14.956136022999999, 2023-11-05T08:33:30.438 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.016832657000000001, 2023-11-05T08:33:30.439 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:30.439 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:30.439 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:30.439 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:30.439 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:30.440 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 416 2023-11-05T08:33:30.440 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.440 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:30.440 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.440 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:30.441 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.354023+0000", 2023-11-05T08:33:30.441 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.441 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.441 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.442 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:30.442 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.354023+0000", 2023-11-05T08:33:30.442 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.442 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.442 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.443 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:30.443 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.354026+0000", 2023-11-05T08:33:30.443 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.7669999999999999e-06 2023-11-05T08:33:30.443 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.443 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.444 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:30.444 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.355125+0000", 2023-11-05T08:33:30.444 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001099399 2023-11-05T08:33:30.444 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.444 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.445 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:30.445 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.355163+0000", 2023-11-05T08:33:30.445 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.8232000000000003e-05 2023-11-05T08:33:30.445 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.445 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.445 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:30.446 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.355182+0000", 2023-11-05T08:33:30.446 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.9153000000000001e-05 2023-11-05T08:33:30.446 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.446 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.446 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:30.447 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.355293+0000", 2023-11-05T08:33:30.447 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00011033500000000001 2023-11-05T08:33:30.447 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.447 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.447 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "op must wait for map", 2023-11-05T08:33:30.448 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.355304+0000", 2023-11-05T08:33:30.448 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.1253000000000001e-05 2023-11-05T08:33:30.448 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.448 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.448 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:30.449 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.359303+0000", 2023-11-05T08:33:30.449 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.003999173 2023-11-05T08:33:30.449 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.449 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.449 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:30.450 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.359491+0000", 2023-11-05T08:33:30.450 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00018750099999999999 2023-11-05T08:33:30.450 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.450 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.450 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:30.451 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.360889+0000", 2023-11-05T08:33:30.451 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001397971 2023-11-05T08:33:30.451 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.451 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.452 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:30.452 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.363175+0000", 2023-11-05T08:33:30.452 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.002286738 2023-11-05T08:33:30.452 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.452 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.453 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:33:30.453 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.370683+0000", 2023-11-05T08:33:30.453 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0075076700000000001 2023-11-05T08:33:30.453 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.453 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.453 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:30.454 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.370856+0000", 2023-11-05T08:33:30.454 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00017246500000000001 2023-11-05T08:33:30.454 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.454 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:30.455 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.455 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.455 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.455 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:417 3.1s0 3:8539e931:::smithi09039252-41:head [append 0~770048] snapc 1c=[1c,1b,1a,17,7] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2023-11-05T08:33:30.455 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:15.366368+0000", 2023-11-05T08:33:30.456 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 14.943790630000001, 2023-11-05T08:33:30.456 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0076147879999999999, 2023-11-05T08:33:30.456 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:30.456 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:30.456 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:30.457 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:30.457 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:30.457 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 417 2023-11-05T08:33:30.457 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.457 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:30.458 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.458 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:30.458 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.366368+0000", 2023-11-05T08:33:30.458 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.458 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.459 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.459 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:30.459 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.366368+0000", 2023-11-05T08:33:30.459 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.459 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.460 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.460 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:30.460 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.366372+0000", 2023-11-05T08:33:30.460 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.647e-06 2023-11-05T08:33:30.461 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.461 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.461 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:30.461 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.367322+0000", 2023-11-05T08:33:30.461 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00094998200000000004 2023-11-05T08:33:30.461 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.462 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.462 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:30.462 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.367352+0000", 2023-11-05T08:33:30.462 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.0301999999999999e-05 2023-11-05T08:33:30.462 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.463 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.463 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:30.463 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.367366+0000", 2023-11-05T08:33:30.463 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.3793e-05 2023-11-05T08:33:30.463 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.464 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.464 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:30.464 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.367437+0000", 2023-11-05T08:33:30.464 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.1477999999999996e-05 2023-11-05T08:33:30.464 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.465 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.465 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:30.465 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.367556+0000", 2023-11-05T08:33:30.465 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00011814700000000001 2023-11-05T08:33:30.465 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.466 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.466 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:30.466 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.368588+0000", 2023-11-05T08:33:30.466 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001032264 2023-11-05T08:33:30.466 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.467 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.467 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:30.467 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.370883+0000", 2023-11-05T08:33:30.467 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0022946580000000002 2023-11-05T08:33:30.467 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.468 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.468 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:33:30.468 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.373790+0000", 2023-11-05T08:33:30.468 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0029076480000000001 2023-11-05T08:33:30.468 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.469 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.469 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:30.469 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.373983+0000", 2023-11-05T08:33:30.469 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000192869 2023-11-05T08:33:30.469 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.470 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:30.470 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.470 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.470 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.471 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:418 3.1s0 3:8539e931:::smithi09039252-41:head [setxattr _header (57) in=7b] snapc 1c=[1c,1b,1a,17,7] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2023-11-05T08:33:30.471 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:15.367398+0000", 2023-11-05T08:33:30.471 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 14.942760832999999, 2023-11-05T08:33:30.471 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0079965609999999993, 2023-11-05T08:33:30.471 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:30.472 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:30.472 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:30.472 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:30.472 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:30.472 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 418 2023-11-05T08:33:30.473 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.473 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:30.473 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.473 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:30.474 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.367398+0000", 2023-11-05T08:33:30.474 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.474 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.474 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.475 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:30.475 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.367398+0000", 2023-11-05T08:33:30.475 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.475 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.475 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.476 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:30.476 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.367399+0000", 2023-11-05T08:33:30.476 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0699999999999999e-06 2023-11-05T08:33:30.476 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.476 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.477 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:30.477 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.367404+0000", 2023-11-05T08:33:30.477 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.1000000000000003e-06 2023-11-05T08:33:30.477 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.477 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.478 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:30.478 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.367416+0000", 2023-11-05T08:33:30.478 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.154e-05 2023-11-05T08:33:30.478 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.479 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.479 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:30.479 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.367424+0000", 2023-11-05T08:33:30.479 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.2889999999999998e-06 2023-11-05T08:33:30.479 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.480 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.480 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:30.480 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.369523+0000", 2023-11-05T08:33:30.480 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0020993610000000001 2023-11-05T08:33:30.480 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.481 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.481 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:30.481 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.369627+0000", 2023-11-05T08:33:30.481 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00010375800000000001 2023-11-05T08:33:30.481 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.482 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.482 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:30.482 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.370111+0000", 2023-11-05T08:33:30.482 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000484303 2023-11-05T08:33:30.482 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.483 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.483 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:30.483 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.371243+0000", 2023-11-05T08:33:30.483 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001131993 2023-11-05T08:33:30.483 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.484 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.484 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:33:30.484 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.375135+0000", 2023-11-05T08:33:30.484 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0038920199999999999 2023-11-05T08:33:30.484 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.485 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.485 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:30.485 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.375395+0000", 2023-11-05T08:33:30.485 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000259127 2023-11-05T08:33:30.485 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.486 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:30.486 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.486 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.486 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.486 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:419 3.1s0 3:8539e931:::smithi09039252-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2023-11-05T08:33:30.487 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:15.367440+0000", 2023-11-05T08:33:30.487 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 14.942719093999999, 2023-11-05T08:33:30.487 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.011845488, 2023-11-05T08:33:30.487 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:30.487 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "started", 2023-11-05T08:33:30.488 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:30.488 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:30.488 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:30.488 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 419 2023-11-05T08:33:30.488 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.489 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:30.489 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.489 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:30.489 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.367440+0000", 2023-11-05T08:33:30.489 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.490 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.490 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.490 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:30.490 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.367440+0000", 2023-11-05T08:33:30.491 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.491 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.491 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.491 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:30.491 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.367441+0000", 2023-11-05T08:33:30.492 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.8299999999999996e-07 2023-11-05T08:33:30.492 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.492 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.492 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:30.493 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.367444+0000", 2023-11-05T08:33:30.493 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.2499999999999998e-06 2023-11-05T08:33:30.493 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.493 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.493 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:30.494 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.367459+0000", 2023-11-05T08:33:30.494 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4763000000000001e-05 2023-11-05T08:33:30.494 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.494 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.495 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:30.495 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.367467+0000", 2023-11-05T08:33:30.495 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.0700000000000007e-06 2023-11-05T08:33:30.495 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.495 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.496 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:30.496 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.370959+0000", 2023-11-05T08:33:30.496 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.003491854 2023-11-05T08:33:30.496 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.496 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.497 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for rw locks", 2023-11-05T08:33:30.497 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.371115+0000", 2023-11-05T08:33:30.497 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000156102 2023-11-05T08:33:30.497 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.497 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.498 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:30.498 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.376485+0000", 2023-11-05T08:33:30.498 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0053701720000000003 2023-11-05T08:33:30.498 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.498 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.499 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:30.499 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.376689+0000", 2023-11-05T08:33:30.499 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000203771 2023-11-05T08:33:30.499 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.499 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.500 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:30.500 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.379285+0000", 2023-11-05T08:33:30.500 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.002596723 2023-11-05T08:33:30.500 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.500 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:30.501 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.501 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.501 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.501 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:485 3.1s0 3:8539e931:::smithi09039252-41:head [append 0~0] snapc 1c=[1a,7] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2023-11-05T08:33:30.501 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:18.565322+0000", 2023-11-05T08:33:30.501 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 11.744836580999999, 2023-11-05T08:33:30.502 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0068891819999999998, 2023-11-05T08:33:30.502 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:30.502 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:30.502 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:30.502 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:30.503 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:30.503 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 485 2023-11-05T08:33:30.503 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.503 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:30.503 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.504 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:30.504 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.565322+0000", 2023-11-05T08:33:30.504 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.504 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.504 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.505 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:30.505 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.565322+0000", 2023-11-05T08:33:30.505 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.505 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.505 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.505 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:30.506 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.565325+0000", 2023-11-05T08:33:30.506 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.0199999999999999e-06 2023-11-05T08:33:30.506 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.506 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.506 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:30.507 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.565339+0000", 2023-11-05T08:33:30.507 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.3679000000000001e-05 2023-11-05T08:33:30.507 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.507 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.507 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:30.508 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.565385+0000", 2023-11-05T08:33:30.508 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.6387999999999997e-05 2023-11-05T08:33:30.508 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.508 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.508 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:30.509 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.565398+0000", 2023-11-05T08:33:30.509 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.2150999999999999e-05 2023-11-05T08:33:30.509 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.509 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.509 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:30.510 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.565484+0000", 2023-11-05T08:33:30.510 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.6821000000000001e-05 2023-11-05T08:33:30.510 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.510 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.510 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:30.511 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.565641+0000", 2023-11-05T08:33:30.511 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00015630900000000001 2023-11-05T08:33:30.511 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.511 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.511 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:30.512 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.566318+0000", 2023-11-05T08:33:30.512 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00067696599999999998 2023-11-05T08:33:30.512 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.512 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.512 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:30.513 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.567700+0000", 2023-11-05T08:33:30.513 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001382807 2023-11-05T08:33:30.513 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.513 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.513 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:33:30.514 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.572145+0000", 2023-11-05T08:33:30.514 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0044445830000000002 2023-11-05T08:33:30.514 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.514 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.514 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:30.515 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.572211+0000", 2023-11-05T08:33:30.515 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.6458000000000006e-05 2023-11-05T08:33:30.515 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.515 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:30.516 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.516 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.516 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.516 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:486 3.1s0 3:8539e931:::smithi09039252-41:head [rollback 1a] snapc 1c=[1a,7] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2023-11-05T08:33:30.516 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:18.565430+0000", 2023-11-05T08:33:30.517 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 11.744729349, 2023-11-05T08:33:30.517 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0070540380000000003, 2023-11-05T08:33:30.517 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:30.517 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:30.517 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:30.518 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:30.518 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:30.518 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 486 2023-11-05T08:33:30.518 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.518 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:30.519 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.519 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:30.519 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.565430+0000", 2023-11-05T08:33:30.519 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.519 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.520 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.520 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:30.520 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.565430+0000", 2023-11-05T08:33:30.520 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.520 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.521 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.521 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:30.521 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.565430+0000", 2023-11-05T08:33:30.521 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.5799999999999998e-07 2023-11-05T08:33:30.521 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.522 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.522 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:30.522 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.565435+0000", 2023-11-05T08:33:30.522 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.2819999999999998e-06 2023-11-05T08:33:30.522 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.523 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.523 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:30.523 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.565446+0000", 2023-11-05T08:33:30.523 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.1418e-05 2023-11-05T08:33:30.523 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.524 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.524 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:30.524 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.565455+0000", 2023-11-05T08:33:30.524 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.5609999999999999e-06 2023-11-05T08:33:30.524 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.525 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.525 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:30.525 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.566642+0000", 2023-11-05T08:33:30.525 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0011878979999999999 2023-11-05T08:33:30.526 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.526 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.526 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:30.526 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.566740+0000", 2023-11-05T08:33:30.526 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.7676000000000005e-05 2023-11-05T08:33:30.527 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.527 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.527 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:30.527 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.567315+0000", 2023-11-05T08:33:30.528 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00057442500000000004 2023-11-05T08:33:30.528 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.528 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.528 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:30.528 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.568421+0000", 2023-11-05T08:33:30.529 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001106517 2023-11-05T08:33:30.529 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.529 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.529 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:33:30.530 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.572427+0000", 2023-11-05T08:33:30.530 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0040059060000000001 2023-11-05T08:33:30.530 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.530 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.530 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:30.531 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.572484+0000", 2023-11-05T08:33:30.531 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.6597e-05 2023-11-05T08:33:30.531 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.531 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:30.531 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.532 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.532 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.532 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:487 3.1s0 3:8539e931:::smithi09039252-41:head [append 0~0] snapc 1c=[1a,7] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2023-11-05T08:33:30.532 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:18.565485+0000", 2023-11-05T08:33:30.532 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 11.744673726, 2023-11-05T08:33:30.533 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0079309570000000006, 2023-11-05T08:33:30.533 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:30.533 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:30.533 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:30.533 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:30.534 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:30.534 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 487 2023-11-05T08:33:30.534 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.534 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:30.534 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.535 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:30.535 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.565485+0000", 2023-11-05T08:33:30.535 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.535 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.536 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.536 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:30.536 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.565485+0000", 2023-11-05T08:33:30.536 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.536 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.537 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.537 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:30.537 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.565486+0000", 2023-11-05T08:33:30.537 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.9999999999999997e-07 2023-11-05T08:33:30.537 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.538 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.538 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:30.538 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.565490+0000", 2023-11-05T08:33:30.538 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.1450000000000001e-06 2023-11-05T08:33:30.538 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.539 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.539 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:30.539 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.565502+0000", 2023-11-05T08:33:30.539 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.2007999999999999e-05 2023-11-05T08:33:30.540 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.540 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.540 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:30.540 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.565516+0000", 2023-11-05T08:33:30.540 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.414e-05 2023-11-05T08:33:30.541 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.541 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.541 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:30.541 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.567739+0000", 2023-11-05T08:33:30.541 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0022223719999999998 2023-11-05T08:33:30.542 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.542 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.542 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:30.542 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.567821+0000", 2023-11-05T08:33:30.542 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.2038000000000003e-05 2023-11-05T08:33:30.543 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.543 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.543 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:30.543 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.568170+0000", 2023-11-05T08:33:30.543 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00034985000000000002 2023-11-05T08:33:30.544 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.544 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.544 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:30.544 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.568896+0000", 2023-11-05T08:33:30.544 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00072510200000000004 2023-11-05T08:33:30.545 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.545 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.545 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:33:30.545 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.573332+0000", 2023-11-05T08:33:30.545 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0044361100000000001 2023-11-05T08:33:30.546 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.546 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.546 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:30.546 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.573416+0000", 2023-11-05T08:33:30.546 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.4492000000000004e-05 2023-11-05T08:33:30.547 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.547 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:30.547 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.547 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.547 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.548 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:496 3.bs0 3:dd412ebb:::smithi09039252-37:head [append 0~802816] snapc 1c=[7] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2023-11-05T08:33:30.548 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:19.389413+0000", 2023-11-05T08:33:30.548 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 10.920745424, 2023-11-05T08:33:30.548 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.027801348, 2023-11-05T08:33:30.548 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:30.548 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:30.549 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:30.549 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:30.549 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:30.549 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 496 2023-11-05T08:33:30.549 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.550 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:30.550 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.550 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:30.550 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.389413+0000", 2023-11-05T08:33:30.550 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.551 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.551 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.551 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:30.551 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.389413+0000", 2023-11-05T08:33:30.551 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.552 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.552 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.552 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:30.552 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.389417+0000", 2023-11-05T08:33:30.552 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.4769999999999999e-06 2023-11-05T08:33:30.553 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.553 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.553 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:30.553 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.391074+0000", 2023-11-05T08:33:30.553 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001657419 2023-11-05T08:33:30.554 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.554 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.554 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:30.554 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.391113+0000", 2023-11-05T08:33:30.554 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.8279999999999999e-05 2023-11-05T08:33:30.555 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.555 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.555 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:30.555 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.391131+0000", 2023-11-05T08:33:30.555 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.8428999999999999e-05 2023-11-05T08:33:30.556 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.556 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.556 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:30.556 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.391273+0000", 2023-11-05T08:33:30.556 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000141707 2023-11-05T08:33:30.556 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.557 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.557 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "op must wait for map", 2023-11-05T08:33:30.557 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.391282+0000", 2023-11-05T08:33:30.557 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.0410000000000003e-06 2023-11-05T08:33:30.557 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.558 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.558 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:30.558 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.397377+0000", 2023-11-05T08:33:30.558 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.006094721 2023-11-05T08:33:30.558 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.559 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.559 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:30.559 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.397568+0000", 2023-11-05T08:33:30.559 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00019143799999999999 2023-11-05T08:33:30.559 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.560 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.560 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:30.560 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.399855+0000", 2023-11-05T08:33:30.560 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.002287491 2023-11-05T08:33:30.560 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.561 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.561 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:30.561 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.405746+0000", 2023-11-05T08:33:30.561 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0058907439999999998 2023-11-05T08:33:30.561 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.562 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.562 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:33:30.562 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.416872+0000", 2023-11-05T08:33:30.562 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.011125823999999999 2023-11-05T08:33:30.562 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.563 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.563 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:30.563 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.417215+0000", 2023-11-05T08:33:30.563 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000342777 2023-11-05T08:33:30.563 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.564 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:30.564 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.564 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.564 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.564 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:497 3.bs0 3:dd412ebb:::smithi09039252-37:head [append 0~638976] snapc 1c=[7] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2023-11-05T08:33:30.565 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:19.394162+0000", 2023-11-05T08:33:30.565 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 10.915997172000001, 2023-11-05T08:33:30.565 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.029484893000000002, 2023-11-05T08:33:30.565 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:30.565 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:30.566 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:30.566 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:30.566 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:30.566 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 497 2023-11-05T08:33:30.567 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.567 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:30.567 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.567 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:30.567 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.394162+0000", 2023-11-05T08:33:30.567 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.568 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.568 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.568 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:30.568 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.394162+0000", 2023-11-05T08:33:30.568 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.569 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.569 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.569 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:30.569 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.394164+0000", 2023-11-05T08:33:30.570 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.2000000000000001e-06 2023-11-05T08:33:30.570 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.570 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.570 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:30.570 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.394956+0000", 2023-11-05T08:33:30.571 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00079165300000000002 2023-11-05T08:33:30.571 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.571 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.571 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:30.571 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.394980+0000", 2023-11-05T08:33:30.572 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.4851999999999999e-05 2023-11-05T08:33:30.572 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.572 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.572 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:30.572 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.394996+0000", 2023-11-05T08:33:30.572 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.5498999999999998e-05 2023-11-05T08:33:30.573 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.573 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.573 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:30.573 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395277+0000", 2023-11-05T08:33:30.573 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00028066300000000001 2023-11-05T08:33:30.574 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.574 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.574 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:33:30.574 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395284+0000", 2023-11-05T08:33:30.575 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.6580000000000002e-06 2023-11-05T08:33:30.575 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.575 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.575 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:30.575 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.402581+0000", 2023-11-05T08:33:30.576 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0072969560000000003 2023-11-05T08:33:30.576 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.576 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.576 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:30.576 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.402751+0000", 2023-11-05T08:33:30.576 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000169922 2023-11-05T08:33:30.577 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.577 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.577 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:30.577 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.404210+0000", 2023-11-05T08:33:30.578 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0014591649999999999 2023-11-05T08:33:30.578 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.578 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.578 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:30.578 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.408453+0000", 2023-11-05T08:33:30.579 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0042427790000000003 2023-11-05T08:33:30.579 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.579 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.579 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:33:30.579 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.423250+0000", 2023-11-05T08:33:30.580 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.014797258000000001 2023-11-05T08:33:30.580 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.580 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.580 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:30.581 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.423647+0000", 2023-11-05T08:33:30.581 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00039628800000000001 2023-11-05T08:33:30.581 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.581 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:30.581 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.582 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.582 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.582 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:498 3.bs0 3:dd412ebb:::smithi09039252-37:head [append 0~131072] snapc 1c=[7] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2023-11-05T08:33:30.582 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:19.395177+0000", 2023-11-05T08:33:30.582 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 10.914982204999999, 2023-11-05T08:33:30.583 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.029134344999999999, 2023-11-05T08:33:30.583 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:30.583 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:30.583 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:30.583 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:30.584 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:30.584 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 498 2023-11-05T08:33:30.584 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.584 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:30.585 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.585 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:30.585 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395177+0000", 2023-11-05T08:33:30.585 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.585 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.586 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.586 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:30.586 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395177+0000", 2023-11-05T08:33:30.586 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.587 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.587 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.587 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:30.587 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395178+0000", 2023-11-05T08:33:30.587 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.499e-06 2023-11-05T08:33:30.588 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.588 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.588 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:30.588 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395362+0000", 2023-11-05T08:33:30.589 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00018334499999999999 2023-11-05T08:33:30.589 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.589 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.589 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:30.589 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395382+0000", 2023-11-05T08:33:30.589 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.0520999999999999e-05 2023-11-05T08:33:30.590 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.590 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.590 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:30.590 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395395+0000", 2023-11-05T08:33:30.590 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.2641e-05 2023-11-05T08:33:30.591 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.591 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.591 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:30.591 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395577+0000", 2023-11-05T08:33:30.592 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00018265999999999999 2023-11-05T08:33:30.592 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.592 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.592 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:33:30.592 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395590+0000", 2023-11-05T08:33:30.593 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.2476e-05 2023-11-05T08:33:30.593 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.593 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.593 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:30.593 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.405933+0000", 2023-11-05T08:33:30.594 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.010343629999999999 2023-11-05T08:33:30.594 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.594 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.594 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:30.594 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.406106+0000", 2023-11-05T08:33:30.595 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000172172 2023-11-05T08:33:30.595 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.595 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.595 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:30.596 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.406978+0000", 2023-11-05T08:33:30.596 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00087286199999999997 2023-11-05T08:33:30.596 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.596 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.596 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:30.597 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.408818+0000", 2023-11-05T08:33:30.597 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0018392 2023-11-05T08:33:30.597 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.597 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.597 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:33:30.598 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.423925+0000", 2023-11-05T08:33:30.598 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.015107517000000001 2023-11-05T08:33:30.598 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.598 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.598 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:30.599 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.424311+0000", 2023-11-05T08:33:30.599 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00038582200000000002 2023-11-05T08:33:30.599 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.599 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:30.600 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.600 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.600 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.600 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:499 3.bs0 3:dd412ebb:::smithi09039252-37:head [setxattr _header (57) in=7b] snapc 1c=[7] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2023-11-05T08:33:30.601 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:19.395440+0000", 2023-11-05T08:33:30.601 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 10.914718409000001, 2023-11-05T08:33:30.601 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.029659106000000001, 2023-11-05T08:33:30.601 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:30.601 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:30.602 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:30.602 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:30.602 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:30.602 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 499 2023-11-05T08:33:30.602 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.603 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:30.603 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.603 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:30.603 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395440+0000", 2023-11-05T08:33:30.603 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.604 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.604 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.604 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:30.604 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395440+0000", 2023-11-05T08:33:30.604 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.605 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.605 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.605 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:30.605 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395442+0000", 2023-11-05T08:33:30.605 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.02e-06 2023-11-05T08:33:30.606 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.606 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.606 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:30.606 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395447+0000", 2023-11-05T08:33:30.606 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.9880000000000001e-06 2023-11-05T08:33:30.607 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.607 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.607 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:30.607 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395462+0000", 2023-11-05T08:33:30.608 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4055e-05 2023-11-05T08:33:30.608 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.608 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.608 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:30.608 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395473+0000", 2023-11-05T08:33:30.609 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.1578e-05 2023-11-05T08:33:30.609 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.609 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.609 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:30.609 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395707+0000", 2023-11-05T08:33:30.610 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000233415 2023-11-05T08:33:30.610 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.610 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.610 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:33:30.610 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395721+0000", 2023-11-05T08:33:30.611 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4136e-05 2023-11-05T08:33:30.611 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.611 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.611 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:30.611 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.407602+0000", 2023-11-05T08:33:30.612 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.011880885000000001 2023-11-05T08:33:30.612 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.612 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.612 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:30.612 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.407685+0000", 2023-11-05T08:33:30.613 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.3615999999999994e-05 2023-11-05T08:33:30.613 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.613 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.613 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:30.613 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.408132+0000", 2023-11-05T08:33:30.614 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00044693100000000002 2023-11-05T08:33:30.614 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.614 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.614 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:30.614 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.409160+0000", 2023-11-05T08:33:30.615 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0010283740000000001 2023-11-05T08:33:30.615 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.615 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.615 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:33:30.615 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.424623+0000", 2023-11-05T08:33:30.616 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.015462075 2023-11-05T08:33:30.616 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.616 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.616 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:30.616 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.425100+0000", 2023-11-05T08:33:30.616 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00047703300000000003 2023-11-05T08:33:30.617 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.617 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:30.617 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.617 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.617 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.618 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:500 3.bs0 3:dd412ebb:::smithi09039252-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e66)", 2023-11-05T08:33:30.618 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:19.395494+0000", 2023-11-05T08:33:30.618 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 10.914665248, 2023-11-05T08:33:30.618 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.034730260999999998, 2023-11-05T08:33:30.618 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:30.619 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "started", 2023-11-05T08:33:30.619 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:30.619 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:30.619 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:30.620 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 500 2023-11-05T08:33:30.620 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.620 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:30.620 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.620 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:30.621 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395494+0000", 2023-11-05T08:33:30.621 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.621 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.621 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.621 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:30.622 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395494+0000", 2023-11-05T08:33:30.622 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:30.622 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.622 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.622 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:30.623 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395494+0000", 2023-11-05T08:33:30.623 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.5099999999999998e-07 2023-11-05T08:33:30.623 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.623 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.624 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:30.624 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395499+0000", 2023-11-05T08:33:30.624 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.2880000000000003e-06 2023-11-05T08:33:30.624 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.624 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.624 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:30.625 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395512+0000", 2023-11-05T08:33:30.625 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.3227999999999999e-05 2023-11-05T08:33:30.625 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.625 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.625 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:30.626 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395521+0000", 2023-11-05T08:33:30.626 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.4469999999999995e-06 2023-11-05T08:33:30.626 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.626 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.627 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:30.627 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395903+0000", 2023-11-05T08:33:30.627 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00038185600000000002 2023-11-05T08:33:30.627 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.627 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.628 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:33:30.628 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395916+0000", 2023-11-05T08:33:30.628 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.2233e-05 2023-11-05T08:33:30.628 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.628 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.629 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:30.629 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.408539+0000", 2023-11-05T08:33:30.629 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.012623013000000001 2023-11-05T08:33:30.629 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.629 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.630 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for rw locks", 2023-11-05T08:33:30.630 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.408699+0000", 2023-11-05T08:33:30.630 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00016055099999999999 2023-11-05T08:33:30.630 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.630 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.631 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:30.631 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.425989+0000", 2023-11-05T08:33:30.631 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.017289562000000001 2023-11-05T08:33:30.631 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.631 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.632 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:30.632 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.426378+0000", 2023-11-05T08:33:30.632 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00038890600000000001 2023-11-05T08:33:30.632 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:30.632 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:30.633 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:30.633 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.430224+0000", 2023-11-05T08:33:30.633 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0038463260000000002 2023-11-05T08:33:30.633 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.633 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:30.634 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.634 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:30.634 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:30.634 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:33:30.637 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2023-11-05T08:33:30.638 INFO:tasks.rados.rados.0.smithi090.stdout:244: done (1 left) 2023-11-05T08:33:30.638 INFO:tasks.rados.rados.0.smithi090.stdout:245: copy_from oid 7 from oid 5 current snap is 22 2023-11-05T08:33:30.638 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 30 current snap is 22 2023-11-05T08:33:30.639 INFO:tasks.rados.rados.0.smithi090.stdout:246: seq_num 106 ranges {1671168=622592,2293760=442368,2736128=524288,3260416=196608} 2023-11-05T08:33:30.639 INFO:tasks.rados.rados.0.smithi090.stdout:246: writing smithi09039252-30 from 1671168 to 2293760 tid 1 2023-11-05T08:33:30.639 INFO:tasks.rados.rados.0.smithi090.stdout:246: writing smithi09039252-30 from 2293760 to 2736128 tid 2 2023-11-05T08:33:30.639 INFO:tasks.rados.rados.0.smithi090.stdout:246: writing smithi09039252-30 from 2736128 to 3260416 tid 3 2023-11-05T08:33:30.640 INFO:tasks.rados.rados.0.smithi090.stdout:246: writing smithi09039252-30 from 3260416 to 3457024 tid 4 2023-11-05T08:33:30.640 INFO:tasks.rados.rados.0.smithi090.stdout:247: snap_create 2023-11-05T08:33:30.640 INFO:tasks.rados.rados.0.smithi090.stdout:245: finishing copy_from to smithi09039252-7 2023-11-05T08:33:30.640 INFO:tasks.rados.rados.0.smithi090.stdout:245: got expected ENOENT (src dne) 2023-11-05T08:33:30.640 INFO:tasks.rados.rados.0.smithi090.stdout:245: finishing copy_from racing read to smithi09039252-7 2023-11-05T08:33:30.641 INFO:tasks.rados.rados.0.smithi090.stdout:246: finishing write tid 1 to smithi09039252-30 2023-11-05T08:33:30.641 INFO:tasks.rados.rados.0.smithi090.stdout:246: finishing write tid 2 to smithi09039252-30 2023-11-05T08:33:30.641 INFO:tasks.rados.rados.0.smithi090.stdout:246: finishing write tid 3 to smithi09039252-30 2023-11-05T08:33:30.641 INFO:tasks.rados.rados.0.smithi090.stdout:246: finishing write tid 4 to smithi09039252-30 2023-11-05T08:33:30.642 INFO:tasks.rados.rados.0.smithi090.stdout:246: finishing write tid 5 to smithi09039252-30 2023-11-05T08:33:30.642 INFO:tasks.rados.rados.0.smithi090.stdout:246: finishing write tid 6 to smithi09039252-30 2023-11-05T08:33:30.642 INFO:tasks.rados.rados.0.smithi090.stdout:246: oid 30 updating version 0 to 45 2023-11-05T08:33:30.642 INFO:tasks.rados.rados.0.smithi090.stdout:246: oid 30 updating version 45 to 46 2023-11-05T08:33:30.642 INFO:tasks.rados.rados.0.smithi090.stdout:246: oid 30 updating version 46 to 47 2023-11-05T08:33:30.643 INFO:tasks.rados.rados.0.smithi090.stdout:246: oid 30 updating version 47 to 48 2023-11-05T08:33:30.643 INFO:tasks.rados.rados.0.smithi090.stdout:246: oid 30 version 48 is already newer than 44 2023-11-05T08:33:30.643 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 30 v 48 (ObjNum 106 snap 22 seq_num 106) dirty exists 2023-11-05T08:33:30.643 INFO:tasks.rados.rados.0.smithi090.stdout:246: left oid 30 (ObjNum 106 snap 22 seq_num 106) 2023-11-05T08:33:30.644 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:30.635+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:30.736 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:33:30.733+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:33:30.779 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:33:30.780 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [], 2023-11-05T08:33:30.780 INFO:teuthology.orchestra.run.smithi002.stdout: "num_ops": 0 2023-11-05T08:33:30.780 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:33:30.794 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2023-11-05T08:33:30.837 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:33:30.836+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:33:30.922 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:33:30.923 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [], 2023-11-05T08:33:30.923 INFO:teuthology.orchestra.run.smithi002.stdout: "complaint_time": 30, 2023-11-05T08:33:30.923 INFO:teuthology.orchestra.run.smithi002.stdout: "num_blocked_ops": 0 2023-11-05T08:33:30.923 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:33:30.937 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2023-11-05T08:33:30.938 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:33:30.933+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:33:30.970 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-05T08:33:31.039 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:33:31.037+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:33:31.066 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:33:31.066 INFO:teuthology.orchestra.run.smithi002.stdout: "size": 20, 2023-11-05T08:33:31.066 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 600, 2023-11-05T08:33:31.066 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [ 2023-11-05T08:33:31.067 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.067 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.ds5 36/30 ECSubWrite(tid=23, reqid=osd.7.0:23, at_version=36'14, trim_to=0'0, roll_forward_to=34'8))", 2023-11-05T08:33:31.067 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:49.078802+0000", 2023-11-05T08:33:31.067 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 41.985788722000002, 2023-11-05T08:33:31.067 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0035119359999999998, 2023-11-05T08:33:31.068 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:31.068 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:31.068 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:31.068 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.068 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:31.069 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.078802+0000", 2023-11-05T08:33:31.069 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.069 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.069 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.069 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:31.070 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.078802+0000", 2023-11-05T08:33:31.070 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.070 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.070 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.071 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:31.071 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.078802+0000", 2023-11-05T08:33:31.071 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.5899999999999996e-07 2023-11-05T08:33:31.071 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.071 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.072 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:31.072 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.078834+0000", 2023-11-05T08:33:31.072 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.1696999999999998e-05 2023-11-05T08:33:31.072 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.072 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.073 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:31.073 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.078845+0000", 2023-11-05T08:33:31.073 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0992000000000001e-05 2023-11-05T08:33:31.073 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.073 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.074 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:31.074 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.078851+0000", 2023-11-05T08:33:31.074 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.1360000000000002e-06 2023-11-05T08:33:31.074 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.074 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.075 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:31.075 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.079095+0000", 2023-11-05T08:33:31.075 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00024419600000000002 2023-11-05T08:33:31.075 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.075 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.076 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:31.076 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.079105+0000", 2023-11-05T08:33:31.076 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0341e-05 2023-11-05T08:33:31.076 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.076 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.077 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:31.077 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.082270+0000", 2023-11-05T08:33:31.077 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0031646719999999999 2023-11-05T08:33:31.077 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.077 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.077 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:31.078 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.082313+0000", 2023-11-05T08:33:31.078 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.3343000000000002e-05 2023-11-05T08:33:31.078 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.078 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:31.078 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.079 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.079 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.079 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:278 3.14s0 3:292b6c2e:::smithi09039252-40:head [copy-from ver 10] snapc 8=[8,7,6,5,3] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:33:31.079 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:53.143346+0000", 2023-11-05T08:33:31.079 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 37.921244450000003, 2023-11-05T08:33:31.080 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0524983919999999, 2023-11-05T08:33:31.080 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:31.080 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:31.080 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:31.080 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:31.081 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:31.081 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 278 2023-11-05T08:33:31.081 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.081 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:31.081 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.082 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:31.082 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143346+0000", 2023-11-05T08:33:31.082 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.082 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.082 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.082 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:31.083 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143346+0000", 2023-11-05T08:33:31.083 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.083 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.083 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.083 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:31.084 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143347+0000", 2023-11-05T08:33:31.084 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4470000000000001e-06 2023-11-05T08:33:31.084 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.084 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.084 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:31.085 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143353+0000", 2023-11-05T08:33:31.085 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.7790000000000001e-06 2023-11-05T08:33:31.085 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.085 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.085 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:31.086 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143367+0000", 2023-11-05T08:33:31.086 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4321000000000001e-05 2023-11-05T08:33:31.086 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.086 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.086 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:31.087 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143374+0000", 2023-11-05T08:33:31.087 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.8029999999999999e-06 2023-11-05T08:33:31.087 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.087 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.087 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:31.088 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143446+0000", 2023-11-05T08:33:31.088 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.2080000000000001e-05 2023-11-05T08:33:31.088 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.088 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.088 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for peered", 2023-11-05T08:33:31.088 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143450+0000", 2023-11-05T08:33:31.089 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.5449999999999999e-06 2023-11-05T08:33:31.089 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.089 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.089 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:31.089 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.169316+0000", 2023-11-05T08:33:31.089 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.025866494 2023-11-05T08:33:31.090 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.090 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.090 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:31.090 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.169711+0000", 2023-11-05T08:33:31.090 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00039465700000000003 2023-11-05T08:33:31.091 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.091 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.091 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:31.091 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.185825+0000", 2023-11-05T08:33:31.091 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.016113624 2023-11-05T08:33:31.092 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.092 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.092 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:31.092 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.190579+0000", 2023-11-05T08:33:31.092 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0047547370000000002 2023-11-05T08:33:31.093 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.093 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.093 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:33:31.093 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.195484+0000", 2023-11-05T08:33:31.093 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0049050200000000004 2023-11-05T08:33:31.093 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.094 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.094 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:31.094 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.195844+0000", 2023-11-05T08:33:31.094 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000359885 2023-11-05T08:33:31.094 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.095 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:31.095 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.095 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.095 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.095 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:279 3.14s0 3:292b6c2e:::smithi09039252-40:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:33:31.096 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:53.143390+0000", 2023-11-05T08:33:31.096 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 37.921200374000001, 2023-11-05T08:33:31.096 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0469768799999999, 2023-11-05T08:33:31.096 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:31.096 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "started", 2023-11-05T08:33:31.097 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:31.097 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:31.097 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:31.097 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 279 2023-11-05T08:33:31.097 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.097 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:31.098 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.098 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:31.098 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143390+0000", 2023-11-05T08:33:31.098 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.098 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.099 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.099 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:31.099 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143390+0000", 2023-11-05T08:33:31.099 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.099 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.100 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.100 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:31.100 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143390+0000", 2023-11-05T08:33:31.100 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.5999999999999999e-07 2023-11-05T08:33:31.100 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.101 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.101 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:31.101 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143392+0000", 2023-11-05T08:33:31.101 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.0209999999999998e-06 2023-11-05T08:33:31.101 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.102 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.102 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:31.102 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143399+0000", 2023-11-05T08:33:31.102 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.6869999999999997e-06 2023-11-05T08:33:31.102 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.103 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.103 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:31.103 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143405+0000", 2023-11-05T08:33:31.103 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.0619999999999997e-06 2023-11-05T08:33:31.103 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.104 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.104 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:31.104 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143483+0000", 2023-11-05T08:33:31.104 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.8268000000000003e-05 2023-11-05T08:33:31.104 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.105 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.105 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for peered", 2023-11-05T08:33:31.105 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143486+0000", 2023-11-05T08:33:31.105 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.322e-06 2023-11-05T08:33:31.105 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.106 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.106 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:31.106 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.172078+0000", 2023-11-05T08:33:31.106 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0285927589999999 2023-11-05T08:33:31.106 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.106 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.107 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for blocked object", 2023-11-05T08:33:31.107 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.172240+0000", 2023-11-05T08:33:31.107 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00016178899999999999 2023-11-05T08:33:31.107 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.107 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.108 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:31.108 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.189404+0000", 2023-11-05T08:33:31.108 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.017164220000000001 2023-11-05T08:33:31.108 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.108 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.109 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:31.109 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.189710+0000", 2023-11-05T08:33:31.109 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000305617 2023-11-05T08:33:31.109 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.109 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.110 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:31.110 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.190367+0000", 2023-11-05T08:33:31.110 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00065677499999999996 2023-11-05T08:33:31.110 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.110 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:31.111 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.111 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.111 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.111 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es4 41/30 ECSubWrite(tid=40, reqid=osd.2.0:40, at_version=41'27, trim_to=0'0, roll_forward_to=40'25))", 2023-11-05T08:33:31.111 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:54.120687+0000", 2023-11-05T08:33:31.112 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 36.943903589999998, 2023-11-05T08:33:31.112 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0097705859999999995, 2023-11-05T08:33:31.112 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:31.112 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:31.112 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:31.113 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.113 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:31.113 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120687+0000", 2023-11-05T08:33:31.113 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.113 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.114 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.114 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:31.114 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120687+0000", 2023-11-05T08:33:31.114 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.114 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.115 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.115 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:31.115 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120688+0000", 2023-11-05T08:33:31.115 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.64e-06 2023-11-05T08:33:31.116 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.116 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.116 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:31.116 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120741+0000", 2023-11-05T08:33:31.116 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.2274000000000001e-05 2023-11-05T08:33:31.117 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.117 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.117 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:31.117 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120770+0000", 2023-11-05T08:33:31.117 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.9768999999999999e-05 2023-11-05T08:33:31.117 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.118 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.118 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:31.118 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120795+0000", 2023-11-05T08:33:31.118 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.4805000000000001e-05 2023-11-05T08:33:31.119 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.119 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.119 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:31.119 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.124189+0000", 2023-11-05T08:33:31.119 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0033935279999999998 2023-11-05T08:33:31.119 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.120 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.120 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:31.120 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.124214+0000", 2023-11-05T08:33:31.120 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.5823e-05 2023-11-05T08:33:31.120 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.121 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.121 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:31.121 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.130330+0000", 2023-11-05T08:33:31.121 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0061156190000000001 2023-11-05T08:33:31.121 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.122 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.122 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:31.122 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.130457+0000", 2023-11-05T08:33:31.122 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00012712800000000001 2023-11-05T08:33:31.123 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.123 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:31.123 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.123 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.123 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.124 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es4 42/30 ECSubWrite(tid=43, reqid=osd.2.0:43, at_version=42'29, trim_to=0'0, roll_forward_to=41'27))", 2023-11-05T08:33:31.124 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:55.140218+0000", 2023-11-05T08:33:31.124 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 35.924371749000002, 2023-11-05T08:33:31.124 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0038969510000000001, 2023-11-05T08:33:31.124 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:31.125 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:31.125 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:31.125 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.125 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:31.125 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.140218+0000", 2023-11-05T08:33:31.126 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.126 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.126 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.126 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:31.126 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.140218+0000", 2023-11-05T08:33:31.126 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.127 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.127 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.127 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:31.127 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.140219+0000", 2023-11-05T08:33:31.127 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.7000000000000005e-07 2023-11-05T08:33:31.128 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.128 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.128 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:31.128 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.140283+0000", 2023-11-05T08:33:31.129 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.3738000000000005e-05 2023-11-05T08:33:31.129 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.129 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.129 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:31.129 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.140317+0000", 2023-11-05T08:33:31.130 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.4498000000000001e-05 2023-11-05T08:33:31.130 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.130 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.130 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:31.131 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.140342+0000", 2023-11-05T08:33:31.131 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.4742000000000001e-05 2023-11-05T08:33:31.131 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.131 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.131 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:31.131 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.140849+0000", 2023-11-05T08:33:31.132 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00050743999999999995 2023-11-05T08:33:31.132 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.132 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.132 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:31.132 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.140876+0000", 2023-11-05T08:33:31.133 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.6798999999999999e-05 2023-11-05T08:33:31.133 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.133 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.133 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:31.133 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.144042+0000", 2023-11-05T08:33:31.134 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0031655870000000001 2023-11-05T08:33:31.134 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.134 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.134 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:31.134 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.144115+0000", 2023-11-05T08:33:31.135 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.3577000000000004e-05 2023-11-05T08:33:31.135 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.135 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:31.135 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.135 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.136 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.136 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpReadReply(3.14s0 47/46 ECSubReadReply(tid=22, attrs_read=0))", 2023-11-05T08:33:31.136 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:00.232364+0000", 2023-11-05T08:33:31.136 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 30.832225776000001, 2023-11-05T08:33:31.136 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0035804690000000002, 2023-11-05T08:33:31.137 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:31.137 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:31.137 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:31.137 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.137 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:31.137 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.232364+0000", 2023-11-05T08:33:31.138 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.138 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.138 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.138 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:31.138 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.232364+0000", 2023-11-05T08:33:31.139 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.139 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.139 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.139 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:31.139 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.232366+0000", 2023-11-05T08:33:31.140 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.393e-06 2023-11-05T08:33:31.140 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.140 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.140 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:31.140 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.232910+0000", 2023-11-05T08:33:31.141 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000543938 2023-11-05T08:33:31.141 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.141 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.141 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:31.141 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.232931+0000", 2023-11-05T08:33:31.141 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.1599000000000001e-05 2023-11-05T08:33:31.142 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.142 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.142 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:31.142 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.232961+0000", 2023-11-05T08:33:31.142 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.9949000000000001e-05 2023-11-05T08:33:31.143 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.143 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.143 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:31.143 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.233087+0000", 2023-11-05T08:33:31.143 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00012517499999999999 2023-11-05T08:33:31.144 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.144 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.144 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:31.144 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.235945+0000", 2023-11-05T08:33:31.144 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.002858415 2023-11-05T08:33:31.145 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.145 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:31.145 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.145 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.145 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.146 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDPGPushReply(3.14s0 47/46 [PushReplyOp(3:292b6c2e:::smithi09039252-40:7)])", 2023-11-05T08:33:31.146 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:00.249065+0000", 2023-11-05T08:33:31.146 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 30.815525509, 2023-11-05T08:33:31.146 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0037023899999999998, 2023-11-05T08:33:31.146 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:31.147 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:31.147 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:31.147 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.147 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:31.147 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.249065+0000", 2023-11-05T08:33:31.147 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.148 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.148 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.148 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:31.148 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.249065+0000", 2023-11-05T08:33:31.148 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.149 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.149 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.149 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:31.149 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.249066+0000", 2023-11-05T08:33:31.149 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.73e-06 2023-11-05T08:33:31.150 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.150 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.150 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:31.150 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.249080+0000", 2023-11-05T08:33:31.150 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.3196e-05 2023-11-05T08:33:31.150 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.151 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.151 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:31.151 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.249117+0000", 2023-11-05T08:33:31.151 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.7419000000000002e-05 2023-11-05T08:33:31.151 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.152 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.152 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:31.152 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.249134+0000", 2023-11-05T08:33:31.152 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.7079e-05 2023-11-05T08:33:31.152 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.152 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.153 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:31.153 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.249248+0000", 2023-11-05T08:33:31.153 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000113929 2023-11-05T08:33:31.153 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.153 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.154 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:31.154 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.252767+0000", 2023-11-05T08:33:31.154 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.003519037 2023-11-05T08:33:31.154 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.154 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:31.155 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.155 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.155 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.155 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDPGPushReply(3.14s0 47/46 [PushReplyOp(3:292b6c2e:::smithi09039252-40:head)])", 2023-11-05T08:33:31.155 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:00.292862+0000", 2023-11-05T08:33:31.156 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 30.771728682999999, 2023-11-05T08:33:31.156 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0063631419999999996, 2023-11-05T08:33:31.156 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:31.156 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:31.156 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:31.156 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.157 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:31.157 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.292862+0000", 2023-11-05T08:33:31.157 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.157 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.158 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.158 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:31.158 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.292862+0000", 2023-11-05T08:33:31.158 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.158 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.159 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.159 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:31.159 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.292862+0000", 2023-11-05T08:33:31.159 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.0200000000000001e-07 2023-11-05T08:33:31.159 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.160 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.160 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:31.160 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.292867+0000", 2023-11-05T08:33:31.160 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.7829999999999998e-06 2023-11-05T08:33:31.160 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.161 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.161 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:31.161 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.292877+0000", 2023-11-05T08:33:31.161 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0312000000000001e-05 2023-11-05T08:33:31.161 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.162 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.162 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:31.162 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.292889+0000", 2023-11-05T08:33:31.162 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.2019e-05 2023-11-05T08:33:31.165 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.165 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.165 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:31.166 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.293399+0000", 2023-11-05T08:33:31.166 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00050963099999999997 2023-11-05T08:33:31.166 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.166 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.166 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:31.167 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.299225+0000", 2023-11-05T08:33:31.167 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0058256949999999997 2023-11-05T08:33:31.167 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.167 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:31.168 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.168 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.168 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.168 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.16s3 51/45 ECSubWrite(tid=58, reqid=osd.11.0:58, at_version=51'19, trim_to=0'0, roll_forward_to=49'17))", 2023-11-05T08:33:31.169 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:04.227640+0000", 2023-11-05T08:33:31.169 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 26.836949893, 2023-11-05T08:33:31.169 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.003497839, 2023-11-05T08:33:31.169 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:31.169 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:31.170 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:31.170 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.170 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:31.170 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.227640+0000", 2023-11-05T08:33:31.171 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.171 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.171 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.171 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:31.171 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.227640+0000", 2023-11-05T08:33:31.172 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.172 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.172 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.172 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:31.172 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.227643+0000", 2023-11-05T08:33:31.173 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.2129999999999998e-06 2023-11-05T08:33:31.173 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.173 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.174 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:31.174 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.227704+0000", 2023-11-05T08:33:31.174 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.1748000000000006e-05 2023-11-05T08:33:31.174 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.175 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.175 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:31.175 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.227744+0000", 2023-11-05T08:33:31.175 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.9598999999999998e-05 2023-11-05T08:33:31.175 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.176 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.176 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:31.176 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.227769+0000", 2023-11-05T08:33:31.176 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.5182999999999999e-05 2023-11-05T08:33:31.176 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.177 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.177 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:31.177 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.227954+0000", 2023-11-05T08:33:31.177 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000185167 2023-11-05T08:33:31.177 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.178 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.178 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:31.178 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.227977+0000", 2023-11-05T08:33:31.178 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.2586e-05 2023-11-05T08:33:31.178 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.179 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.179 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:31.179 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.231028+0000", 2023-11-05T08:33:31.179 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0030515400000000002 2023-11-05T08:33:31.179 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.180 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.180 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:31.180 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.231138+0000", 2023-11-05T08:33:31.180 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00010980300000000001 2023-11-05T08:33:31.180 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.180 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:31.181 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.181 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.181 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.181 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.16s3 56/45 ECSubWrite(tid=63, reqid=client.4691.0:369, at_version=56'20, trim_to=0'0, roll_forward_to=51'19))", 2023-11-05T08:33:31.181 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:09.293412+0000", 2023-11-05T08:33:31.182 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 21.771177787999999, 2023-11-05T08:33:31.182 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0049197820000000001, 2023-11-05T08:33:31.182 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:31.182 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:31.182 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:31.183 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.183 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:31.183 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.293412+0000", 2023-11-05T08:33:31.183 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.183 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.183 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.184 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:31.184 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.293412+0000", 2023-11-05T08:33:31.184 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.184 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.184 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.185 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:31.185 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.293413+0000", 2023-11-05T08:33:31.185 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.0299999999999998e-07 2023-11-05T08:33:31.185 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.185 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.186 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:31.186 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.293961+0000", 2023-11-05T08:33:31.186 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00054824799999999999 2023-11-05T08:33:31.186 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.186 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.187 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:31.187 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.293997+0000", 2023-11-05T08:33:31.187 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.5991999999999997e-05 2023-11-05T08:33:31.187 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.187 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.188 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:31.188 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.294022+0000", 2023-11-05T08:33:31.188 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.4865999999999999e-05 2023-11-05T08:33:31.188 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.188 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.188 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:31.189 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.294234+0000", 2023-11-05T08:33:31.189 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00021178399999999999 2023-11-05T08:33:31.189 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.189 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.189 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:31.190 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.294264+0000", 2023-11-05T08:33:31.190 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.0309000000000001e-05 2023-11-05T08:33:31.190 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.190 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.190 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:31.191 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.298238+0000", 2023-11-05T08:33:31.191 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0039737959999999999 2023-11-05T08:33:31.191 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.191 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.192 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:31.192 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.298332+0000", 2023-11-05T08:33:31.192 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.3984e-05 2023-11-05T08:33:31.192 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.192 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:31.192 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.193 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.193 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.193 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:374 3.14s0 3:292b6c2e:::smithi09039252-40:head [copy-from ver 2] snapc 15=[7] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2023-11-05T08:33:31.193 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:09.297419+0000", 2023-11-05T08:33:31.194 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 21.767171040000001, 2023-11-05T08:33:31.194 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.03004254, 2023-11-05T08:33:31.194 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:31.194 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:31.194 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:31.195 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:31.195 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:31.195 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 374 2023-11-05T08:33:31.195 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.195 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:31.196 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.196 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:31.196 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297419+0000", 2023-11-05T08:33:31.196 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.196 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.197 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.197 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:31.197 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297419+0000", 2023-11-05T08:33:31.197 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.197 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.197 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.198 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:31.198 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297422+0000", 2023-11-05T08:33:31.198 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.717e-06 2023-11-05T08:33:31.198 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.198 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.199 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:31.199 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297437+0000", 2023-11-05T08:33:31.199 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4793e-05 2023-11-05T08:33:31.199 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.199 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.200 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:31.200 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297468+0000", 2023-11-05T08:33:31.200 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.1519000000000001e-05 2023-11-05T08:33:31.200 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.200 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.200 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:31.201 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297490+0000", 2023-11-05T08:33:31.201 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.1651999999999999e-05 2023-11-05T08:33:31.201 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.201 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.201 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:31.202 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.300337+0000", 2023-11-05T08:33:31.202 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0028475219999999999 2023-11-05T08:33:31.202 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.202 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.202 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:31.203 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.300717+0000", 2023-11-05T08:33:31.203 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00037937600000000001 2023-11-05T08:33:31.203 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.203 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.203 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:31.204 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.319194+0000", 2023-11-05T08:33:31.204 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.018477303 2023-11-05T08:33:31.204 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.204 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.204 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:31.204 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.322493+0000", 2023-11-05T08:33:31.205 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0032987250000000002 2023-11-05T08:33:31.205 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.205 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.205 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:33:31.205 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.326714+0000", 2023-11-05T08:33:31.206 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0042208319999999999 2023-11-05T08:33:31.206 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.206 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.206 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:31.206 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.327462+0000", 2023-11-05T08:33:31.207 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00074810100000000004 2023-11-05T08:33:31.207 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.207 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:31.207 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.207 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.208 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.208 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:375 3.14s0 3:292b6c2e:::smithi09039252-40:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e56)", 2023-11-05T08:33:31.208 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:09.297530+0000", 2023-11-05T08:33:31.208 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 21.767060213000001, 2023-11-05T08:33:31.208 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.024433857999999999, 2023-11-05T08:33:31.208 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:31.209 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "started", 2023-11-05T08:33:31.209 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:31.209 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:31.209 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:31.209 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 375 2023-11-05T08:33:31.210 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.210 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:31.210 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.210 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:31.210 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297530+0000", 2023-11-05T08:33:31.211 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.211 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.211 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.211 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:31.212 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297530+0000", 2023-11-05T08:33:31.212 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.212 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.212 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.212 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:31.213 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297531+0000", 2023-11-05T08:33:31.213 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.48e-06 2023-11-05T08:33:31.213 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.213 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.213 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:31.213 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297540+0000", 2023-11-05T08:33:31.214 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.49e-06 2023-11-05T08:33:31.214 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.214 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.214 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:31.214 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297566+0000", 2023-11-05T08:33:31.215 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.6152e-05 2023-11-05T08:33:31.215 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.215 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.215 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:31.215 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297583+0000", 2023-11-05T08:33:31.216 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.6676000000000001e-05 2023-11-05T08:33:31.216 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.216 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.216 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:31.216 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.301232+0000", 2023-11-05T08:33:31.217 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.003649209 2023-11-05T08:33:31.217 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.217 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.217 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for blocked object", 2023-11-05T08:33:31.217 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.301367+0000", 2023-11-05T08:33:31.218 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00013454600000000001 2023-11-05T08:33:31.218 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.218 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.218 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:31.218 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.321294+0000", 2023-11-05T08:33:31.219 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.019927261000000002 2023-11-05T08:33:31.219 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.219 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.219 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:31.219 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.321567+0000", 2023-11-05T08:33:31.220 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00027297299999999998 2023-11-05T08:33:31.220 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.220 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.220 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:31.220 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.321964+0000", 2023-11-05T08:33:31.221 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00039707099999999998 2023-11-05T08:33:31.221 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.221 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:31.221 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.221 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.222 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.222 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.14s5 56/48 ECSubWrite(tid=32, reqid=client.4691.0:374, at_version=56'19, trim_to=0'0, roll_forward_to=49'18))", 2023-11-05T08:33:31.222 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:09.318983+0000", 2023-11-05T08:33:31.222 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 21.745607505999999, 2023-11-05T08:33:31.222 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00432556, 2023-11-05T08:33:31.223 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:31.223 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:31.223 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:31.223 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.223 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:31.224 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.318983+0000", 2023-11-05T08:33:31.224 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.224 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.224 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.224 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:31.225 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.318983+0000", 2023-11-05T08:33:31.225 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.225 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.225 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.225 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:31.226 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.318983+0000", 2023-11-05T08:33:31.226 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.226 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.226 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.227 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:31.227 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.318983+0000", 2023-11-05T08:33:31.227 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.227 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.227 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.228 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:31.228 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.319029+0000", 2023-11-05T08:33:31.228 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.6675999999999998e-05 2023-11-05T08:33:31.228 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.228 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.229 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:31.229 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.319054+0000", 2023-11-05T08:33:31.229 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.4669e-05 2023-11-05T08:33:31.229 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.229 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.230 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:31.230 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.319246+0000", 2023-11-05T08:33:31.230 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00019165499999999999 2023-11-05T08:33:31.230 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.230 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.231 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:31.231 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.319278+0000", 2023-11-05T08:33:31.231 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.2744999999999999e-05 2023-11-05T08:33:31.231 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.231 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.232 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:31.232 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.323256+0000", 2023-11-05T08:33:31.232 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0039773589999999998 2023-11-05T08:33:31.232 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.232 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.233 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:31.233 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.323308+0000", 2023-11-05T08:33:31.233 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.2456000000000001e-05 2023-11-05T08:33:31.233 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.233 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:31.233 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.234 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.234 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.234 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es4 57/30 ECSubWrite(tid=60, reqid=client.4691.0:380, at_version=57'45, trim_to=0'0, roll_forward_to=49'43))", 2023-11-05T08:33:31.234 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:10.285514+0000", 2023-11-05T08:33:31.234 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 20.779076637999999, 2023-11-05T08:33:31.235 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.003439372, 2023-11-05T08:33:31.235 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:31.235 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:31.235 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:31.236 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.236 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:31.236 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285514+0000", 2023-11-05T08:33:31.236 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.236 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.237 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.237 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:31.237 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285514+0000", 2023-11-05T08:33:31.237 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.237 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.238 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.238 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:31.238 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285514+0000", 2023-11-05T08:33:31.238 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.9000000000000002e-07 2023-11-05T08:33:31.238 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.239 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.239 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:31.239 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285558+0000", 2023-11-05T08:33:31.239 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.4229e-05 2023-11-05T08:33:31.239 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.240 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.240 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:31.240 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285584+0000", 2023-11-05T08:33:31.240 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.5706000000000001e-05 2023-11-05T08:33:31.240 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.241 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.241 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:31.241 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285602+0000", 2023-11-05T08:33:31.241 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.7863000000000002e-05 2023-11-05T08:33:31.241 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.241 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.242 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:31.242 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285821+0000", 2023-11-05T08:33:31.242 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000219416 2023-11-05T08:33:31.242 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.242 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.243 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:31.243 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285862+0000", 2023-11-05T08:33:31.243 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.0923000000000001e-05 2023-11-05T08:33:31.243 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.243 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.244 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:31.244 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.288883+0000", 2023-11-05T08:33:31.244 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0030210430000000002 2023-11-05T08:33:31.244 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.244 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.245 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:31.245 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.288953+0000", 2023-11-05T08:33:31.245 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.9801999999999993e-05 2023-11-05T08:33:31.245 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.245 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:31.245 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.246 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.246 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.246 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.10s4 62/45 ECSubWrite(tid=219, reqid=client.4691.0:424, at_version=62'11, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:31.246 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:15.391016+0000", 2023-11-05T08:33:31.246 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 15.673574053999999, 2023-11-05T08:33:31.247 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0049263429999999997, 2023-11-05T08:33:31.247 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:31.247 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:31.247 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:31.248 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.248 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:31.248 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.391016+0000", 2023-11-05T08:33:31.248 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.248 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.248 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.249 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:31.249 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.391016+0000", 2023-11-05T08:33:31.249 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.249 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.249 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.249 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:31.250 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.391017+0000", 2023-11-05T08:33:31.250 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.7e-07 2023-11-05T08:33:31.250 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.250 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.251 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:31.251 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.391866+0000", 2023-11-05T08:33:31.251 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00084949500000000002 2023-11-05T08:33:31.251 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.251 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.251 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:31.252 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.391906+0000", 2023-11-05T08:33:31.252 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.0302000000000001e-05 2023-11-05T08:33:31.252 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.252 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.253 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:31.253 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.391932+0000", 2023-11-05T08:33:31.253 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.5439000000000002e-05 2023-11-05T08:33:31.253 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.253 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.254 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:31.254 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.392053+0000", 2023-11-05T08:33:31.254 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00012140400000000001 2023-11-05T08:33:31.254 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.254 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.254 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:31.255 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.392067+0000", 2023-11-05T08:33:31.255 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4051e-05 2023-11-05T08:33:31.255 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.255 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.255 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:31.256 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.395819+0000", 2023-11-05T08:33:31.256 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0037515489999999999 2023-11-05T08:33:31.256 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.256 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.256 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:31.257 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.395943+0000", 2023-11-05T08:33:31.257 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000123733 2023-11-05T08:33:31.257 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.257 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:31.257 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.258 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.258 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.258 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.10s4 62/45 ECSubWrite(tid=220, reqid=client.4691.0:425, at_version=62'12, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:31.258 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:15.397304+0000", 2023-11-05T08:33:31.258 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 15.667285855999999, 2023-11-05T08:33:31.259 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0042497419999999999, 2023-11-05T08:33:31.259 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:31.259 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:31.259 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:31.259 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.260 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:31.260 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.397304+0000", 2023-11-05T08:33:31.260 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.260 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.260 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.261 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:31.261 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.397304+0000", 2023-11-05T08:33:31.261 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.261 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.261 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.262 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:31.262 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.397306+0000", 2023-11-05T08:33:31.262 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.5290000000000001e-06 2023-11-05T08:33:31.262 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.263 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.263 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:31.263 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.397686+0000", 2023-11-05T08:33:31.263 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00038012700000000001 2023-11-05T08:33:31.263 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.264 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.264 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:31.264 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.397726+0000", 2023-11-05T08:33:31.264 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.9842000000000002e-05 2023-11-05T08:33:31.264 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.265 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.265 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:31.265 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.397748+0000", 2023-11-05T08:33:31.265 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.2079e-05 2023-11-05T08:33:31.265 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.266 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.266 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:31.266 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.397923+0000", 2023-11-05T08:33:31.266 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000174637 2023-11-05T08:33:31.266 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.267 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.267 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:31.267 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.397959+0000", 2023-11-05T08:33:31.267 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.6885999999999999e-05 2023-11-05T08:33:31.267 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.267 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.268 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:31.268 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.401476+0000", 2023-11-05T08:33:31.268 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0035168809999999999 2023-11-05T08:33:31.268 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.268 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.269 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:31.269 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.401554+0000", 2023-11-05T08:33:31.269 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.7761000000000003e-05 2023-11-05T08:33:31.269 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.269 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:31.270 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.270 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.270 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.270 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.10s4 62/45 ECSubWrite(tid=221, reqid=client.4691.0:426, at_version=62'13, trim_to=0'0, roll_forward_to=62'11))", 2023-11-05T08:33:31.270 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:15.398990+0000", 2023-11-05T08:33:31.271 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 15.665599939, 2023-11-05T08:33:31.271 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0037293809999999999, 2023-11-05T08:33:31.271 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:31.271 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:31.271 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:31.272 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.272 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:31.272 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.398990+0000", 2023-11-05T08:33:31.272 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.272 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.273 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.273 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:31.273 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.398990+0000", 2023-11-05T08:33:31.273 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.273 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.274 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.274 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:31.274 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.398999+0000", 2023-11-05T08:33:31.274 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.7660000000000004e-06 2023-11-05T08:33:31.274 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.275 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.275 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:31.275 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.399128+0000", 2023-11-05T08:33:31.275 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00012880699999999999 2023-11-05T08:33:31.275 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.275 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.276 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:31.276 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.399160+0000", 2023-11-05T08:33:31.276 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.1761999999999998e-05 2023-11-05T08:33:31.276 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.276 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.277 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:31.277 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.399191+0000", 2023-11-05T08:33:31.277 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.1053000000000002e-05 2023-11-05T08:33:31.277 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.277 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.278 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:31.278 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.400107+0000", 2023-11-05T08:33:31.278 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00091625699999999997 2023-11-05T08:33:31.278 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.278 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.279 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:31.279 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.400140+0000", 2023-11-05T08:33:31.279 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.3385e-05 2023-11-05T08:33:31.279 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.279 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.280 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:31.280 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.402645+0000", 2023-11-05T08:33:31.280 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0025045129999999999 2023-11-05T08:33:31.280 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.280 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.280 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:31.281 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.402720+0000", 2023-11-05T08:33:31.281 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.4838000000000004e-05 2023-11-05T08:33:31.281 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.281 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:31.281 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.282 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.282 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.282 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.10s4 62/45 ECSubWrite(tid=222, reqid=client.4691.0:427, at_version=62'14, trim_to=0'0, roll_forward_to=62'11))", 2023-11-05T08:33:31.282 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:15.399828+0000", 2023-11-05T08:33:31.282 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 15.664762714, 2023-11-05T08:33:31.283 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0036514389999999998, 2023-11-05T08:33:31.283 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:31.283 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:31.283 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:31.283 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.284 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:31.284 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.399828+0000", 2023-11-05T08:33:31.284 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.284 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.284 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.284 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:31.285 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.399828+0000", 2023-11-05T08:33:31.285 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.285 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.285 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.285 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:31.286 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.399829+0000", 2023-11-05T08:33:31.286 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4300000000000001e-06 2023-11-05T08:33:31.286 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.286 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.286 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:31.287 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.399871+0000", 2023-11-05T08:33:31.287 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.1848999999999999e-05 2023-11-05T08:33:31.287 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.287 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.287 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:31.288 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.399902+0000", 2023-11-05T08:33:31.288 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.0985999999999998e-05 2023-11-05T08:33:31.288 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.288 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.288 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:31.289 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.399928+0000", 2023-11-05T08:33:31.289 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.6426000000000001e-05 2023-11-05T08:33:31.289 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.289 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.289 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:31.289 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.401704+0000", 2023-11-05T08:33:31.290 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001775694 2023-11-05T08:33:31.290 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.290 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.290 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:31.290 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.401736+0000", 2023-11-05T08:33:31.291 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.2511000000000001e-05 2023-11-05T08:33:31.291 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.291 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.291 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:31.291 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.403426+0000", 2023-11-05T08:33:31.292 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0016898390000000001 2023-11-05T08:33:31.292 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.292 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.292 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:31.292 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.403479+0000", 2023-11-05T08:33:31.293 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.2704000000000002e-05 2023-11-05T08:33:31.293 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.293 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:31.293 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.293 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.294 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.294 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:573 3.19s0 3:99e86b02:::smithi09039252-36:head [] snapc 27=[7] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2023-11-05T08:33:31.294 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:29.488425+0000", 2023-11-05T08:33:31.294 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 1.576165262, 2023-11-05T08:33:31.294 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.003519749, 2023-11-05T08:33:31.294 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:31.295 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:31.295 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:31.295 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:31.295 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:31.295 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 573 2023-11-05T08:33:31.296 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.296 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:31.296 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.296 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:31.296 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:29.488425+0000", 2023-11-05T08:33:31.297 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.297 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.297 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.297 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:31.297 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:29.488425+0000", 2023-11-05T08:33:31.298 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.298 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.298 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.298 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:31.298 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:29.488427+0000", 2023-11-05T08:33:31.299 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.4770000000000002e-06 2023-11-05T08:33:31.299 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.299 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.299 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:31.299 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:29.488438+0000", 2023-11-05T08:33:31.300 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0668e-05 2023-11-05T08:33:31.300 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.300 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.300 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:31.300 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:29.488464+0000", 2023-11-05T08:33:31.300 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.6380000000000002e-05 2023-11-05T08:33:31.301 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.301 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.301 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:31.301 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:29.488477+0000", 2023-11-05T08:33:31.301 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.2425999999999999e-05 2023-11-05T08:33:31.302 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.302 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.302 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:31.302 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:29.488579+0000", 2023-11-05T08:33:31.302 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000102174 2023-11-05T08:33:31.303 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.303 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.303 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "op must wait for map", 2023-11-05T08:33:31.303 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:29.488592+0000", 2023-11-05T08:33:31.303 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.3062e-05 2023-11-05T08:33:31.304 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.304 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.304 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:31.304 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:29.491866+0000", 2023-11-05T08:33:31.304 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0032740550000000001 2023-11-05T08:33:31.305 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.305 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.305 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:31.305 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:29.491945+0000", 2023-11-05T08:33:31.305 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.8506999999999999e-05 2023-11-05T08:33:31.306 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.306 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:31.306 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.306 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.306 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.307 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.fs4 77/37 ECSubWrite(tid=130, reqid=client.4691.0:579, at_version=77'44, trim_to=0'0, roll_forward_to=70'42))", 2023-11-05T08:33:31.307 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:30.540343+0000", 2023-11-05T08:33:31.307 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 0.524247401, 2023-11-05T08:33:31.307 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0057579060000000001, 2023-11-05T08:33:31.307 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:31.308 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:31.308 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:31.308 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.308 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:31.308 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:30.540343+0000", 2023-11-05T08:33:31.309 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.309 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.309 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.309 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:31.309 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:30.540343+0000", 2023-11-05T08:33:31.310 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:31.310 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.310 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.310 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:31.311 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:30.540344+0000", 2023-11-05T08:33:31.311 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.5999999999999991e-07 2023-11-05T08:33:31.311 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.311 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.311 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:31.312 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:30.540957+0000", 2023-11-05T08:33:31.312 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00061291099999999995 2023-11-05T08:33:31.312 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.312 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.312 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:31.312 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:30.540985+0000", 2023-11-05T08:33:31.313 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.7999000000000001e-05 2023-11-05T08:33:31.313 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.313 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.313 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:31.313 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:30.541002+0000", 2023-11-05T08:33:31.314 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.7105999999999999e-05 2023-11-05T08:33:31.314 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.314 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.314 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:31.314 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:30.541176+0000", 2023-11-05T08:33:31.315 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00017453100000000001 2023-11-05T08:33:31.315 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.315 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.315 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:31.315 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:30.541198+0000", 2023-11-05T08:33:31.316 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.1773e-05 2023-11-05T08:33:31.316 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.316 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.316 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:31.316 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:30.546033+0000", 2023-11-05T08:33:31.316 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.004835092 2023-11-05T08:33:31.317 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:31.317 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:31.317 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:31.317 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:30.546101+0000", 2023-11-05T08:33:31.317 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.7533999999999994e-05 2023-11-05T08:33:31.318 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.318 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:31.318 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.318 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:31.318 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:31.319 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:33:31.320 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2023-11-05T08:33:31.321 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-11-05T08:33:31.322 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1: {} 2023-11-05T08:33:31.322 INFO:teuthology.orchestra.run.smithi002.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-11-05T08:33:31.322 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2: {} 2023-11-05T08:33:31.323 INFO:teuthology.orchestra.run.smithi002.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-11-05T08:33:31.323 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-11-05T08:33:31.324 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4: {} 2023-11-05T08:33:31.324 INFO:teuthology.orchestra.run.smithi002.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-11-05T08:33:31.324 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5: {} 2023-11-05T08:33:31.325 INFO:teuthology.orchestra.run.smithi002.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-11-05T08:33:31.325 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6: {} 2023-11-05T08:33:31.326 INFO:teuthology.orchestra.run.smithi002.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-11-05T08:33:31.326 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7: {} 2023-11-05T08:33:31.327 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:33:31.319+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:33:31.327 INFO:teuthology.orchestra.run.smithi002.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-11-05T08:33:31.327 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8: {} 2023-11-05T08:33:31.328 INFO:teuthology.orchestra.run.smithi002.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-11-05T08:33:31.328 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9: {} 2023-11-05T08:33:31.329 INFO:teuthology.orchestra.run.smithi002.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-11-05T08:33:31.329 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10: {} 2023-11-05T08:33:31.329 INFO:teuthology.orchestra.run.smithi002.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-11-05T08:33:31.330 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11: {} 2023-11-05T08:33:31.330 INFO:teuthology.orchestra.run.smithi002.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-11-05T08:33:31.331 INFO:teuthology.orchestra.run.smithi002.stdout:osd.12: {} 2023-11-05T08:33:31.331 INFO:teuthology.orchestra.run.smithi002.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-11-05T08:33:31.353 INFO:teuthology.orchestra.run.smithi002.stdout:osd.13: {} 2023-11-05T08:33:31.353 INFO:teuthology.orchestra.run.smithi002.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-11-05T08:33:31.375 INFO:teuthology.orchestra.run.smithi002.stdout:osd.14: {} 2023-11-05T08:33:31.376 INFO:teuthology.orchestra.run.smithi002.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-11-05T08:33:31.398 INFO:teuthology.orchestra.run.smithi002.stdout:osd.15: {} 2023-11-05T08:33:31.399 INFO:teuthology.orchestra.run.smithi002.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-11-05T08:33:31.403 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-05T08:33:31.403 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-05T08:33:31.421 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:33:31.416+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:33:31.458 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:31.458 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:31.474 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2023-11-05T08:33:31.511 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-11-05T08:33:31.513 INFO:tasks.rados.rados.0.smithi090.stdout:245: done (3 left) 2023-11-05T08:33:31.513 INFO:tasks.rados.rados.0.smithi090.stdout:246: done (2 left) 2023-11-05T08:33:31.513 INFO:tasks.rados.rados.0.smithi090.stdout:247: done (1 left) 2023-11-05T08:33:31.514 INFO:tasks.rados.rados.0.smithi090.stdout:248: rollback oid 10 current snap is 23 2023-11-05T08:33:31.514 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 10 to 22 2023-11-05T08:33:31.514 INFO:tasks.rados.rados.0.smithi090.stdout:249: copy_from oid 14 from oid 4 current snap is 23 2023-11-05T08:33:31.514 INFO:tasks.rados.rados.0.smithi090.stdout:250: read oid 40 snap 22 2023-11-05T08:33:31.514 INFO:tasks.rados.rados.0.smithi090.stdout:250: expect (ObjNum 17 snap 0 seq_num 17) 2023-11-05T08:33:31.515 INFO:tasks.rados.rados.0.smithi090.stdout:251: read oid 35 snap -1 2023-11-05T08:33:31.515 INFO:tasks.rados.rados.0.smithi090.stdout:251: expect (ObjNum 34 snap 0 seq_num 34) 2023-11-05T08:33:31.515 INFO:tasks.rados.rados.0.smithi090.stdout:252: snap_remove snap 21 2023-11-05T08:33:31.521 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:33:31.516+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:33:31.602 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:31.602 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:31.618 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2023-11-05T08:33:31.625 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:33:31.624+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:33:31.724 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:33:31.717+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:33:31.758 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:31.759 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:31.772 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-11-05T08:33:31.824 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:31.820+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:31.907 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:31.907 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:31.921 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-11-05T08:33:31.963 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:33:31.960+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:33:32.026 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:33:32.021+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:33:32.091 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:32.091 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:32.105 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-11-05T08:33:32.126 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-05T08:33:32.229 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:33:32.225+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:33:32.230 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:32.230 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:32.244 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-11-05T08:33:32.329 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~36s 2023-11-05T08:33:32.329 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:33:32.324+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:33:32.371 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:32.372 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:32.386 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-11-05T08:33:32.431 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:33:32.429+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:33:32.514 INFO:teuthology.orchestra.run.smithi002.stderr:nodeep-scrub is set 2023-11-05T08:33:32.523 INFO:tasks.rados.rados.0.smithi090.stdout:250: expect (ObjNum 17 snap 0 seq_num 17) 2023-11-05T08:33:32.531 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:32.528+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:32.545 INFO:tasks.rados.rados.0.smithi090.stdout:249: finishing copy_from racing read to smithi09039252-14 2023-11-05T08:33:32.545 INFO:tasks.rados.rados.0.smithi090.stdout:248: finishing rollback tid 0 to smithi09039252-10 2023-11-05T08:33:32.545 INFO:tasks.rados.rados.0.smithi090.stdout:248: finishing rollback tid 1 to smithi09039252-10 2023-11-05T08:33:32.545 INFO:tasks.rados.rados.0.smithi090.stdout:248: finishing rollback tid 2 to smithi09039252-10 2023-11-05T08:33:32.545 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 10 v 25 (ObjNum 9 snap 0 seq_num 9) dirty exists 2023-11-05T08:33:32.546 INFO:tasks.rados.rados.0.smithi090.stdout:249: finishing copy_from to smithi09039252-14 2023-11-05T08:33:32.546 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 14 v 4 (ObjNum 3 snap 0 seq_num 3) dirty exists 2023-11-05T08:33:32.546 INFO:tasks.rados.rados.0.smithi090.stdout:251: expect (ObjNum 34 snap 0 seq_num 34) 2023-11-05T08:33:32.561 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:32.561 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:32.576 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-11-05T08:33:32.577 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [3] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 7, 11, 15] 2023-11-05T08:33:32.578 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-11-05T08:33:32.578 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-11-05T08:33:32.578 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-11-05T08:33:32.583 INFO:tasks.rados.rados.0.smithi090.stdout:248: done (5 left) 2023-11-05T08:33:32.584 INFO:tasks.rados.rados.0.smithi090.stdout:249: done (4 left) 2023-11-05T08:33:32.584 INFO:tasks.rados.rados.0.smithi090.stdout:250: done (3 left) 2023-11-05T08:33:32.584 INFO:tasks.rados.rados.0.smithi090.stdout:251: done (2 left) 2023-11-05T08:33:32.584 INFO:tasks.rados.rados.0.smithi090.stdout:252: done (1 left) 2023-11-05T08:33:32.585 INFO:tasks.rados.rados.0.smithi090.stdout:253: setattr oid 22 current snap is 23 2023-11-05T08:33:32.588 INFO:tasks.rados.rados.0.smithi090.stdout:254: delete oid 19 current snap is 23 2023-11-05T08:33:32.598 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 22 v 19 (ObjNum 21 snap 0 seq_num 21) dirty exists 2023-11-05T08:33:32.599 INFO:tasks.rados.rados.0.smithi090.stdout:253: done (2 left) 2023-11-05T08:33:32.599 INFO:tasks.rados.rados.0.smithi090.stdout:254: done (1 left) 2023-11-05T08:33:32.599 INFO:tasks.rados.rados.0.smithi090.stdout:255: rmattr oid 23 current snap is 23 2023-11-05T08:33:32.599 INFO:tasks.rados.rados.0.smithi090.stdout:255: done (1 left) 2023-11-05T08:33:32.599 INFO:tasks.rados.rados.0.smithi090.stdout:256: rmattr oid 41 current snap is 23 2023-11-05T08:33:32.600 INFO:tasks.rados.rados.0.smithi090.stdout:256: done (1 left) 2023-11-05T08:33:32.600 INFO:tasks.rados.rados.0.smithi090.stdout:257: snap_create 2023-11-05T08:33:32.632 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:33:32.631+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:33:32.727 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-11-05T08:33:32.730 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-05T08:33:32.731 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-05T08:33:32.731 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-11-05T08:33:32.733 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:33:32.728+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:33:32.742 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:32.742 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:32.757 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2023-11-05T08:33:32.833 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-05T08:33:32.896 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-11-05T08:33:32.927 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:32.927 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:32.940 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:33:32.937+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:33:32.942 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2023-11-05T08:33:33.036 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:33:33.033+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:33:33.063 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-11-05T08:33:33.070 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:33.070 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:33.085 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2023-11-05T08:33:33.137 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:33:33.132+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:33:33.211 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:33.211 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:33.225 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2023-11-05T08:33:33.237 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-11-05T08:33:33.238 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:33:33.233+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:33:33.240 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-05T08:33:33.241 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-11-05T08:33:33.241 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 reset_purged_snaps_last 2023-11-05T08:33:33.339 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:33:33.333+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:33:33.395 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:33.395 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:33.405 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-05T08:33:33.406 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 reset_purged_snaps_last 2023-11-05T08:33:33.409 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2023-11-05T08:33:33.454 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:33:33.452+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:33:33.529 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:33.530 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:33.530 INFO:tasks.rados.rados.0.smithi090.stdout:257: done (1 left) 2023-11-05T08:33:33.531 INFO:tasks.rados.rados.0.smithi090.stdout:258: read oid 4 snap -1 2023-11-05T08:33:33.531 INFO:tasks.rados.rados.0.smithi090.stdout:258: expect (ObjNum 3 snap 0 seq_num 3) 2023-11-05T08:33:33.531 INFO:tasks.rados.rados.0.smithi090.stdout:259: rollback oid 31 current snap is 24 2023-11-05T08:33:33.531 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 31 to 23 2023-11-05T08:33:33.531 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 9 current snap is 24 2023-11-05T08:33:33.532 INFO:tasks.rados.rados.0.smithi090.stdout:260: seq_num 108 ranges {114688=655360,770048=606208,1376256=475136,1851392=81920} 2023-11-05T08:33:33.533 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-11-05T08:33:33.541 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:33:33.537+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:33:33.546 INFO:tasks.rados.rados.0.smithi090.stdout:260: writing smithi09039252-9 from 114688 to 770048 tid 1 2023-11-05T08:33:33.547 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2023-11-05T08:33:33.555 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1: {} 2023-11-05T08:33:33.555 INFO:teuthology.orchestra.run.smithi002.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-11-05T08:33:33.558 INFO:tasks.rados.rados.0.smithi090.stdout:260: writing smithi09039252-9 from 770048 to 1376256 tid 2 2023-11-05T08:33:33.566 INFO:tasks.rados.rados.0.smithi090.stdout:260: writing smithi09039252-9 from 1376256 to 1851392 tid 3 2023-11-05T08:33:33.568 INFO:tasks.rados.rados.0.smithi090.stdout:260: writing smithi09039252-9 from 1851392 to 1933312 tid 4 2023-11-05T08:33:33.569 INFO:tasks.rados.rados.0.smithi090.stdout:261: snap_create 2023-11-05T08:33:33.569 INFO:tasks.rados.rados.0.smithi090.stdout:258: expect (ObjNum 3 snap 0 seq_num 3) 2023-11-05T08:33:33.575 INFO:tasks.rados.rados.0.smithi090.stdout:259: finishing rollback tid 0 to smithi09039252-31 2023-11-05T08:33:33.575 INFO:tasks.rados.rados.0.smithi090.stdout:259: finishing rollback tid 1 to smithi09039252-31 2023-11-05T08:33:33.575 INFO:tasks.rados.rados.0.smithi090.stdout:259: finishing rollback tid 2 to smithi09039252-31 2023-11-05T08:33:33.575 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 31 v 38 (ObjNum 87 snap 16 seq_num 87) dirty exists 2023-11-05T08:33:33.576 INFO:tasks.rados.rados.0.smithi090.stdout:260: finishing write tid 1 to smithi09039252-9 2023-11-05T08:33:33.576 INFO:tasks.rados.rados.0.smithi090.stdout:260: finishing write tid 2 to smithi09039252-9 2023-11-05T08:33:33.576 INFO:tasks.rados.rados.0.smithi090.stdout:260: finishing write tid 3 to smithi09039252-9 2023-11-05T08:33:33.577 INFO:tasks.rados.rados.0.smithi090.stdout:260: finishing write tid 4 to smithi09039252-9 2023-11-05T08:33:33.577 INFO:tasks.rados.rados.0.smithi090.stdout:260: finishing write tid 5 to smithi09039252-9 2023-11-05T08:33:33.579 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2: {} 2023-11-05T08:33:33.580 INFO:teuthology.orchestra.run.smithi002.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-11-05T08:33:33.580 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-11-05T08:33:33.581 INFO:tasks.rados.rados.0.smithi090.stdout:260: finishing write tid 6 to smithi09039252-9 2023-11-05T08:33:33.581 INFO:tasks.rados.rados.0.smithi090.stdout:260: oid 9 updating version 0 to 41 2023-11-05T08:33:33.581 INFO:tasks.rados.rados.0.smithi090.stdout:260: oid 9 updating version 41 to 42 2023-11-05T08:33:33.581 INFO:tasks.rados.rados.0.smithi090.stdout:260: oid 9 updating version 42 to 43 2023-11-05T08:33:33.582 INFO:tasks.rados.rados.0.smithi090.stdout:260: oid 9 version 43 is already newer than 40 2023-11-05T08:33:33.582 INFO:tasks.rados.rados.0.smithi090.stdout:260: oid 9 version 43 is already newer than 39 2023-11-05T08:33:33.582 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 9 v 43 (ObjNum 108 snap 24 seq_num 108) dirty exists 2023-11-05T08:33:33.582 INFO:tasks.rados.rados.0.smithi090.stdout:260: left oid 9 (ObjNum 108 snap 24 seq_num 108) 2023-11-05T08:33:33.596 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4: {} 2023-11-05T08:33:33.596 INFO:teuthology.orchestra.run.smithi002.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-11-05T08:33:33.615 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5: {} 2023-11-05T08:33:33.616 INFO:teuthology.orchestra.run.smithi002.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-11-05T08:33:33.617 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 reset_purged_snaps_last 2023-11-05T08:33:33.636 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6: {} 2023-11-05T08:33:33.636 INFO:teuthology.orchestra.run.smithi002.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-11-05T08:33:33.641 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:33.639+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:33.656 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7: {} 2023-11-05T08:33:33.656 INFO:teuthology.orchestra.run.smithi002.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-11-05T08:33:33.672 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8: {} 2023-11-05T08:33:33.673 INFO:teuthology.orchestra.run.smithi002.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-11-05T08:33:33.691 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9: {} 2023-11-05T08:33:33.692 INFO:teuthology.orchestra.run.smithi002.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-11-05T08:33:33.711 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10: {} 2023-11-05T08:33:33.712 INFO:teuthology.orchestra.run.smithi002.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-11-05T08:33:33.718 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:33.718 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:33.732 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11: {} 2023-11-05T08:33:33.732 INFO:teuthology.orchestra.run.smithi002.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-11-05T08:33:33.733 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-11-05T08:33:33.742 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:33:33.740+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:33:33.749 INFO:teuthology.orchestra.run.smithi002.stdout:osd.12: {} 2023-11-05T08:33:33.750 INFO:teuthology.orchestra.run.smithi002.stderr:osd.12: osd_enable_op_tracker = 'true' 2023-11-05T08:33:33.770 INFO:teuthology.orchestra.run.smithi002.stdout:osd.13: {} 2023-11-05T08:33:33.770 INFO:teuthology.orchestra.run.smithi002.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-11-05T08:33:33.781 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 reset_purged_snaps_last 2023-11-05T08:33:33.790 INFO:teuthology.orchestra.run.smithi002.stdout:osd.14: {} 2023-11-05T08:33:33.791 INFO:teuthology.orchestra.run.smithi002.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-11-05T08:33:33.812 INFO:teuthology.orchestra.run.smithi002.stdout:osd.15: {} 2023-11-05T08:33:33.812 INFO:teuthology.orchestra.run.smithi002.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-11-05T08:33:33.817 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-05T08:33:33.818 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-05T08:33:33.843 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:33:33.841+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:33:33.863 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:33:33.864 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2023-11-05T08:33:33.864 INFO:teuthology.orchestra.run.smithi069.stdout: "num_ops": 0 2023-11-05T08:33:33.864 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:33:33.878 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-11-05T08:33:33.944 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:33:33.943+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:33:33.956 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 reset_purged_snaps_last 2023-11-05T08:33:34.036 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:33:34.037 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2023-11-05T08:33:34.037 INFO:teuthology.orchestra.run.smithi069.stdout: "complaint_time": 30, 2023-11-05T08:33:34.037 INFO:teuthology.orchestra.run.smithi069.stdout: "num_blocked_ops": 0 2023-11-05T08:33:34.037 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:33:34.045 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:33:34.043+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:33:34.051 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-11-05T08:33:34.123 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 reset_purged_snaps_last 2023-11-05T08:33:34.144 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-05T08:33:34.177 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:33:34.177 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 20, 2023-11-05T08:33:34.177 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 600, 2023-11-05T08:33:34.177 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [ 2023-11-05T08:33:34.178 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.178 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_repop(client.4281.0:140 1.0 e23/22 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 23'98, mlcod=23'90)", 2023-11-05T08:33:34.178 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:29.364836+0000", 2023-11-05T08:33:34.178 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 64.808766352999996, 2023-11-05T08:33:34.178 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0063794710000000003, 2023-11-05T08:33:34.179 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:34.179 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:34.179 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:34.179 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.179 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:34.180 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.364836+0000", 2023-11-05T08:33:34.180 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.180 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.180 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.180 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:34.180 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.364836+0000", 2023-11-05T08:33:34.181 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.181 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.181 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.181 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:34.181 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.364838+0000", 2023-11-05T08:33:34.182 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.663e-06 2023-11-05T08:33:34.182 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.182 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.182 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:34.182 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.364935+0000", 2023-11-05T08:33:34.182 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.7565999999999994e-05 2023-11-05T08:33:34.183 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.183 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.183 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:34.183 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.364964+0000", 2023-11-05T08:33:34.183 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8626999999999999e-05 2023-11-05T08:33:34.184 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.184 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.184 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:34.184 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.364977+0000", 2023-11-05T08:33:34.184 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3412e-05 2023-11-05T08:33:34.184 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.185 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.185 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:34.185 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.365131+0000", 2023-11-05T08:33:34.185 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00015339300000000001 2023-11-05T08:33:34.185 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.186 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.186 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:33:34.186 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.365170+0000", 2023-11-05T08:33:34.186 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.9632000000000003e-05 2023-11-05T08:33:34.186 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.186 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.187 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:33:34.187 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.371117+0000", 2023-11-05T08:33:34.187 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0059463249999999997 2023-11-05T08:33:34.187 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.187 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.188 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:34.188 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:29.371215+0000", 2023-11-05T08:33:34.188 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.8852999999999997e-05 2023-11-05T08:33:34.188 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.188 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:34.188 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.189 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.189 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.189 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.fs1 35/30 ECSubWrite(tid=25, reqid=client.4691.0:228, at_version=35'13, trim_to=0'0, roll_forward_to=33'11))", 2023-11-05T08:33:34.189 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:48.124260+0000", 2023-11-05T08:33:34.189 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 46.049342834000001, 2023-11-05T08:33:34.190 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0069525890000000003, 2023-11-05T08:33:34.190 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:34.190 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:34.190 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:34.190 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.191 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:34.191 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.124260+0000", 2023-11-05T08:33:34.191 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.191 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.191 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.192 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:34.192 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.124260+0000", 2023-11-05T08:33:34.192 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.192 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.192 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.193 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:34.193 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.124260+0000", 2023-11-05T08:33:34.193 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.6300000000000001e-07 2023-11-05T08:33:34.193 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.193 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.194 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:34.194 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.125159+0000", 2023-11-05T08:33:34.194 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000898886 2023-11-05T08:33:34.194 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.194 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.194 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:34.195 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.125202+0000", 2023-11-05T08:33:34.195 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.3040999999999997e-05 2023-11-05T08:33:34.195 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.195 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.195 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:34.195 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.125227+0000", 2023-11-05T08:33:34.196 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4774e-05 2023-11-05T08:33:34.196 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.196 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.196 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:34.196 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.125405+0000", 2023-11-05T08:33:34.197 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00017825600000000001 2023-11-05T08:33:34.197 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.197 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.197 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:34.197 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.125435+0000", 2023-11-05T08:33:34.197 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0148e-05 2023-11-05T08:33:34.198 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.198 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.198 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:34.198 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.131127+0000", 2023-11-05T08:33:34.198 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0056919070000000004 2023-11-05T08:33:34.199 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.199 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.199 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:34.199 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.131212+0000", 2023-11-05T08:33:34.199 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.5214000000000001e-05 2023-11-05T08:33:34.200 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.200 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:34.200 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.200 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.200 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.200 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.4s1 36/30 ECSubWrite(tid=10, reqid=osd.8.0:10, at_version=36'8, trim_to=0'0, roll_forward_to=34'6))", 2023-11-05T08:33:34.201 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.079150+0000", 2023-11-05T08:33:34.201 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 45.094452234999999, 2023-11-05T08:33:34.201 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00708488, 2023-11-05T08:33:34.201 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:34.201 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:34.202 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:34.202 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.202 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:34.202 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079150+0000", 2023-11-05T08:33:34.202 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.202 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.203 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.203 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:34.203 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079150+0000", 2023-11-05T08:33:34.203 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.203 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.204 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.204 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:34.204 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079151+0000", 2023-11-05T08:33:34.204 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.8299999999999996e-07 2023-11-05T08:33:34.204 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.205 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.205 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:34.205 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079199+0000", 2023-11-05T08:33:34.205 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.7726999999999999e-05 2023-11-05T08:33:34.205 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.205 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.206 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:34.206 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079227+0000", 2023-11-05T08:33:34.206 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8772000000000002e-05 2023-11-05T08:33:34.206 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.206 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.207 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:34.207 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079250+0000", 2023-11-05T08:33:34.207 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2422e-05 2023-11-05T08:33:34.207 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.207 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.208 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:34.208 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079718+0000", 2023-11-05T08:33:34.208 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00046772399999999998 2023-11-05T08:33:34.208 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.208 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.208 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:34.209 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079741+0000", 2023-11-05T08:33:34.209 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3659000000000002e-05 2023-11-05T08:33:34.209 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.209 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.209 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:34.209 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.086158+0000", 2023-11-05T08:33:34.210 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0064169769999999999 2023-11-05T08:33:34.210 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.210 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.210 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:34.210 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.086235+0000", 2023-11-05T08:33:34.211 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.6816000000000005e-05 2023-11-05T08:33:34.211 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.211 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:34.211 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.211 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.212 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.212 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.8s2 36/30 ECSubWrite(tid=23, reqid=osd.5.0:23, at_version=36'15, trim_to=0'0, roll_forward_to=34'13))", 2023-11-05T08:33:34.212 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.079405+0000", 2023-11-05T08:33:34.212 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 45.094197108000003, 2023-11-05T08:33:34.212 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0067666469999999998, 2023-11-05T08:33:34.212 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:34.213 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:34.213 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:34.213 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.213 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:34.213 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079405+0000", 2023-11-05T08:33:34.214 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.214 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.214 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.214 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:34.214 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079405+0000", 2023-11-05T08:33:34.214 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.215 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.215 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.215 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:34.215 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079406+0000", 2023-11-05T08:33:34.215 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.9999999999999998e-07 2023-11-05T08:33:34.216 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.216 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.216 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:34.216 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079446+0000", 2023-11-05T08:33:34.216 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.0034999999999999e-05 2023-11-05T08:33:34.216 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.217 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.217 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:34.217 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079490+0000", 2023-11-05T08:33:34.217 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.4413999999999999e-05 2023-11-05T08:33:34.217 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.218 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.218 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:34.218 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079509+0000", 2023-11-05T08:33:34.218 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8828999999999999e-05 2023-11-05T08:33:34.218 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.219 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.219 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:34.219 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079677+0000", 2023-11-05T08:33:34.219 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00016785199999999999 2023-11-05T08:33:34.219 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.219 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.220 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:34.220 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079702+0000", 2023-11-05T08:33:34.220 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5072e-05 2023-11-05T08:33:34.220 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.220 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.221 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:34.221 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.086071+0000", 2023-11-05T08:33:34.221 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0063687170000000003 2023-11-05T08:33:34.221 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.221 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.222 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:34.222 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.086172+0000", 2023-11-05T08:33:34.222 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000101228 2023-11-05T08:33:34.225 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.225 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:34.225 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.226 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.226 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.226 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:274 3.es0 3:74767c3c:::smithi09039252-8:head [append 0~737280] snapc 8=[8,7,6,5,3] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:33:34.226 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:53.134268+0000", 2023-11-05T08:33:34.226 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 41.039334078000003, 2023-11-05T08:33:34.227 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.014769740999999999, 2023-11-05T08:33:34.227 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:34.227 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:34.227 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:33:34.227 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:33:34.227 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:34.228 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 274 2023-11-05T08:33:34.228 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.228 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:34.228 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.228 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:34.229 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.134268+0000", 2023-11-05T08:33:34.229 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.229 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.229 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.229 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:34.230 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.134268+0000", 2023-11-05T08:33:34.230 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.230 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.230 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.230 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:34.231 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.134270+0000", 2023-11-05T08:33:34.231 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8560000000000001e-06 2023-11-05T08:33:34.231 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.231 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.231 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:34.232 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.135843+0000", 2023-11-05T08:33:34.232 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0015724210000000001 2023-11-05T08:33:34.232 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.232 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.232 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:34.232 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.135876+0000", 2023-11-05T08:33:34.233 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.3895e-05 2023-11-05T08:33:34.233 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.233 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.233 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:34.233 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.135929+0000", 2023-11-05T08:33:34.234 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.2930000000000003e-05 2023-11-05T08:33:34.234 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.234 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.234 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:34.234 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.136066+0000", 2023-11-05T08:33:34.235 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000136673 2023-11-05T08:33:34.235 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.235 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.235 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:33:34.235 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.136335+0000", 2023-11-05T08:33:34.235 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00026863599999999999 2023-11-05T08:33:34.236 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.236 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.236 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:34.236 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.138586+0000", 2023-11-05T08:33:34.236 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0022515629999999998 2023-11-05T08:33:34.237 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.237 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.237 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:34.237 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.148630+0000", 2023-11-05T08:33:34.237 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.010043778 2023-11-05T08:33:34.237 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.238 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.238 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:33:34.238 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.148798+0000", 2023-11-05T08:33:34.238 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00016756799999999999 2023-11-05T08:33:34.238 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.239 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.239 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:34.239 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.149038+0000", 2023-11-05T08:33:34.239 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00024042099999999999 2023-11-05T08:33:34.239 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.240 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:34.240 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.240 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.240 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.240 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:275 3.es0 3:74767c3c:::smithi09039252-8:head [append 0~376832] snapc 8=[8,7,6,5,3] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:33:34.240 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:53.142766+0000", 2023-11-05T08:33:34.241 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 41.030836661999999, 2023-11-05T08:33:34.241 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0071270860000000004, 2023-11-05T08:33:34.241 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:34.241 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:34.241 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:33:34.242 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:33:34.242 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:34.242 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 275 2023-11-05T08:33:34.242 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.242 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:34.243 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.243 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:34.243 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.142766+0000", 2023-11-05T08:33:34.243 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.243 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.243 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.244 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:34.244 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.142766+0000", 2023-11-05T08:33:34.244 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.244 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.244 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.245 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:34.245 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.142768+0000", 2023-11-05T08:33:34.245 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3989999999999998e-06 2023-11-05T08:33:34.245 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.245 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.245 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:34.246 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143223+0000", 2023-11-05T08:33:34.246 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00045515200000000001 2023-11-05T08:33:34.246 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.246 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.246 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:34.247 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143247+0000", 2023-11-05T08:33:34.247 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3898999999999999e-05 2023-11-05T08:33:34.247 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.247 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.247 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:34.248 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143266+0000", 2023-11-05T08:33:34.248 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8372999999999998e-05 2023-11-05T08:33:34.248 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.248 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.248 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:34.249 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143492+0000", 2023-11-05T08:33:34.249 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000226319 2023-11-05T08:33:34.249 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.249 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.249 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:33:34.249 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143751+0000", 2023-11-05T08:33:34.250 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00025883200000000002 2023-11-05T08:33:34.250 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.250 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.250 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:34.250 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.145441+0000", 2023-11-05T08:33:34.251 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0016905620000000001 2023-11-05T08:33:34.251 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.251 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.251 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:34.251 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.149302+0000", 2023-11-05T08:33:34.252 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0038602860000000001 2023-11-05T08:33:34.252 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.252 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.252 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:33:34.252 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.149825+0000", 2023-11-05T08:33:34.253 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00052301300000000002 2023-11-05T08:33:34.253 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.253 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.253 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:34.253 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.149893+0000", 2023-11-05T08:33:34.254 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.8250999999999994e-05 2023-11-05T08:33:34.254 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.254 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:34.254 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.254 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.255 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.255 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:276 3.es0 3:74767c3c:::smithi09039252-8:head [setxattr _header (55) in=7b] snapc 8=[8,7,6,5,3] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:33:34.255 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:53.143364+0000", 2023-11-05T08:33:34.255 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 41.030238707000002, 2023-11-05T08:33:34.255 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0074923510000000004, 2023-11-05T08:33:34.256 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:34.256 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:34.256 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:33:34.256 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:33:34.256 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:34.256 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 276 2023-11-05T08:33:34.257 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.257 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:34.257 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.257 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:34.257 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143364+0000", 2023-11-05T08:33:34.258 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.258 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.258 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.258 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:34.258 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143364+0000", 2023-11-05T08:33:34.259 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.259 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.259 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.259 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:34.259 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143366+0000", 2023-11-05T08:33:34.260 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0420000000000001e-06 2023-11-05T08:33:34.260 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.260 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.260 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:34.260 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143375+0000", 2023-11-05T08:33:34.260 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.5429999999999999e-06 2023-11-05T08:33:34.261 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.261 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.261 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:34.261 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143407+0000", 2023-11-05T08:33:34.261 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.1563999999999998e-05 2023-11-05T08:33:34.262 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.262 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.262 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:34.262 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143421+0000", 2023-11-05T08:33:34.262 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4479e-05 2023-11-05T08:33:34.263 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.263 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.263 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:34.263 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.146951+0000", 2023-11-05T08:33:34.263 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.003530156 2023-11-05T08:33:34.264 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.264 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.264 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:33:34.264 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.147140+0000", 2023-11-05T08:33:34.264 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00018828000000000001 2023-11-05T08:33:34.265 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.265 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.265 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:34.265 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.148038+0000", 2023-11-05T08:33:34.265 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00089799300000000003 2023-11-05T08:33:34.266 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.266 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.266 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:34.266 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.149508+0000", 2023-11-05T08:33:34.266 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001470652 2023-11-05T08:33:34.266 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.267 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.267 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:33:34.267 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.150659+0000", 2023-11-05T08:33:34.267 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001150376 2023-11-05T08:33:34.267 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.268 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.268 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:34.268 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.150856+0000", 2023-11-05T08:33:34.268 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000197266 2023-11-05T08:33:34.268 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.269 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:34.269 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.269 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.269 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.269 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:277 3.es0 3:74767c3c:::smithi09039252-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:33:34.269 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:53.143452+0000", 2023-11-05T08:33:34.270 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 41.030150095000003, 2023-11-05T08:33:34.270 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0099967070000000005, 2023-11-05T08:33:34.270 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:34.270 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2023-11-05T08:33:34.270 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:33:34.271 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:33:34.271 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:34.271 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 277 2023-11-05T08:33:34.271 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.271 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:34.272 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.272 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:34.272 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143452+0000", 2023-11-05T08:33:34.272 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.272 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.272 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.273 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:34.273 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143452+0000", 2023-11-05T08:33:34.273 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.273 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.273 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.274 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:34.274 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143453+0000", 2023-11-05T08:33:34.274 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0640000000000001e-06 2023-11-05T08:33:34.274 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.274 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.275 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:34.275 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143459+0000", 2023-11-05T08:33:34.275 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.5330000000000002e-06 2023-11-05T08:33:34.275 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.275 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.276 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:34.276 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143491+0000", 2023-11-05T08:33:34.276 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2156000000000002e-05 2023-11-05T08:33:34.276 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.276 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.277 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:34.277 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143504+0000", 2023-11-05T08:33:34.277 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3156000000000001e-05 2023-11-05T08:33:34.277 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.277 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.277 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:34.278 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.150256+0000", 2023-11-05T08:33:34.278 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0067515980000000001 2023-11-05T08:33:34.278 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.278 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.278 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for rw locks", 2023-11-05T08:33:34.279 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.150342+0000", 2023-11-05T08:33:34.279 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.5951000000000004e-05 2023-11-05T08:33:34.279 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.279 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.279 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:34.280 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.151851+0000", 2023-11-05T08:33:34.280 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0015094590000000001 2023-11-05T08:33:34.280 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.280 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.280 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:33:34.281 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.151933+0000", 2023-11-05T08:33:34.281 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.1898999999999996e-05 2023-11-05T08:33:34.281 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.281 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.281 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:34.282 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.153449+0000", 2023-11-05T08:33:34.282 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0015158909999999999 2023-11-05T08:33:34.282 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.282 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:34.282 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.282 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.283 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.283 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.11s5 41/40 ECSubWrite(tid=65, reqid=client.4691.0:287, at_version=41'13, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:34.283 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.173811+0000", 2023-11-05T08:33:34.283 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 39.999791500999997, 2023-11-05T08:33:34.283 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0074165059999999998, 2023-11-05T08:33:34.284 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:34.284 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:34.284 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:34.284 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.284 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:34.285 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.173811+0000", 2023-11-05T08:33:34.285 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.285 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.285 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.285 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:34.286 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.173811+0000", 2023-11-05T08:33:34.286 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.286 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.286 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.286 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:34.287 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.173812+0000", 2023-11-05T08:33:34.287 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.485e-06 2023-11-05T08:33:34.287 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.287 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.287 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:34.288 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.174484+0000", 2023-11-05T08:33:34.288 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00067199399999999998 2023-11-05T08:33:34.288 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.288 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.288 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:34.288 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.174513+0000", 2023-11-05T08:33:34.289 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8271000000000001e-05 2023-11-05T08:33:34.289 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.289 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.289 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:34.289 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.174531+0000", 2023-11-05T08:33:34.290 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8433000000000001e-05 2023-11-05T08:33:34.290 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.290 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.290 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:34.290 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.174759+0000", 2023-11-05T08:33:34.291 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00022838599999999999 2023-11-05T08:33:34.291 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.291 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.291 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:34.291 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.174781+0000", 2023-11-05T08:33:34.292 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1273000000000001e-05 2023-11-05T08:33:34.292 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.292 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.292 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:34.292 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.181145+0000", 2023-11-05T08:33:34.292 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0063640140000000003 2023-11-05T08:33:34.293 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.293 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.293 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:34.293 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.181227+0000", 2023-11-05T08:33:34.293 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.2650000000000003e-05 2023-11-05T08:33:34.294 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.294 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:34.294 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.294 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.294 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.295 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.11s5 41/40 ECSubWrite(tid=66, reqid=client.4691.0:288, at_version=41'14, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:34.295 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.177831+0000", 2023-11-05T08:33:34.295 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 39.995771687000001, 2023-11-05T08:33:34.295 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0064294060000000004, 2023-11-05T08:33:34.295 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:34.296 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:34.296 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:34.296 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.296 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:34.296 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.177831+0000", 2023-11-05T08:33:34.296 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.297 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.297 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.297 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:34.297 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.177831+0000", 2023-11-05T08:33:34.297 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.298 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.298 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.298 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:34.298 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.177832+0000", 2023-11-05T08:33:34.298 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4419999999999999e-06 2023-11-05T08:33:34.299 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.299 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.299 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:34.299 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.178026+0000", 2023-11-05T08:33:34.299 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00019376800000000001 2023-11-05T08:33:34.299 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.300 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.300 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:34.300 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.178053+0000", 2023-11-05T08:33:34.300 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6650000000000001e-05 2023-11-05T08:33:34.300 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.301 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.301 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:34.301 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.178070+0000", 2023-11-05T08:33:34.301 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.7385e-05 2023-11-05T08:33:34.301 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.302 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.302 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:34.302 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.178199+0000", 2023-11-05T08:33:34.302 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00012860700000000001 2023-11-05T08:33:34.302 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.302 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.303 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:34.303 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.178256+0000", 2023-11-05T08:33:34.303 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.7639000000000002e-05 2023-11-05T08:33:34.303 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.303 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.304 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:34.304 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184156+0000", 2023-11-05T08:33:34.304 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0058995260000000004 2023-11-05T08:33:34.304 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.304 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.305 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:34.305 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184260+0000", 2023-11-05T08:33:34.305 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00010438900000000001 2023-11-05T08:33:34.305 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.305 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:34.305 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.306 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.306 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.306 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.14s1 41/40 ECSubWrite(tid=17, reqid=client.4691.0:278, at_version=41'14, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:34.306 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.185630+0000", 2023-11-05T08:33:34.306 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 39.987972499999998, 2023-11-05T08:33:34.307 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0070243399999999996, 2023-11-05T08:33:34.307 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:34.307 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:34.307 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:34.307 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.308 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:34.308 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.185630+0000", 2023-11-05T08:33:34.308 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.308 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.308 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.308 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:34.309 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.185630+0000", 2023-11-05T08:33:34.309 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.309 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.309 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.309 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:34.310 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.185631+0000", 2023-11-05T08:33:34.310 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3480000000000001e-06 2023-11-05T08:33:34.310 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.310 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.311 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:34.311 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.186192+0000", 2023-11-05T08:33:34.311 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00056080799999999999 2023-11-05T08:33:34.311 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.311 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.312 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:34.312 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.186216+0000", 2023-11-05T08:33:34.312 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4048000000000001e-05 2023-11-05T08:33:34.312 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.312 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.313 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:34.313 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.186247+0000", 2023-11-05T08:33:34.313 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0505999999999999e-05 2023-11-05T08:33:34.313 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.313 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.314 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:34.314 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.186387+0000", 2023-11-05T08:33:34.314 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00014087999999999999 2023-11-05T08:33:34.314 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.314 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.314 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:34.315 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.186407+0000", 2023-11-05T08:33:34.315 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9160999999999998e-05 2023-11-05T08:33:34.315 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.315 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.315 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:34.316 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.192585+0000", 2023-11-05T08:33:34.316 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0061782080000000001 2023-11-05T08:33:34.316 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.316 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.316 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:34.316 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.192654+0000", 2023-11-05T08:33:34.317 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.9381000000000005e-05 2023-11-05T08:33:34.317 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.317 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:34.317 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.317 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.318 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.318 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.fs1 45/37 ECSubWrite(tid=40, reqid=client.4691.0:313, at_version=45'23, trim_to=0'0, roll_forward_to=38'21))", 2023-11-05T08:33:34.318 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:58.201456+0000", 2023-11-05T08:33:34.318 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 35.972145986999998, 2023-11-05T08:33:34.318 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0078928130000000003, 2023-11-05T08:33:34.319 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:34.319 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:34.319 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:34.319 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.319 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:34.319 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.201456+0000", 2023-11-05T08:33:34.320 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.320 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.320 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.320 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:34.320 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.201456+0000", 2023-11-05T08:33:34.321 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.321 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.321 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.321 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:34.321 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.201464+0000", 2023-11-05T08:33:34.322 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.4830000000000002e-06 2023-11-05T08:33:34.322 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.322 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.322 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:34.322 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.202475+0000", 2023-11-05T08:33:34.323 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001010911 2023-11-05T08:33:34.323 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.323 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.323 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:34.323 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.202500+0000", 2023-11-05T08:33:34.324 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5327999999999999e-05 2023-11-05T08:33:34.324 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.324 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.324 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:34.324 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.202520+0000", 2023-11-05T08:33:34.324 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9922999999999999e-05 2023-11-05T08:33:34.325 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.325 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.325 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:34.325 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.202653+0000", 2023-11-05T08:33:34.325 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00013329800000000001 2023-11-05T08:33:34.326 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.326 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.326 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:34.326 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.202672+0000", 2023-11-05T08:33:34.326 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8864000000000001e-05 2023-11-05T08:33:34.327 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.327 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.327 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:34.327 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.209293+0000", 2023-11-05T08:33:34.327 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0066211719999999998 2023-11-05T08:33:34.328 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.328 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.328 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:34.328 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.209349+0000", 2023-11-05T08:33:34.328 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.5834000000000001e-05 2023-11-05T08:33:34.329 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.329 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:34.329 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.329 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.329 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.329 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.fs1 45/37 ECSubWrite(tid=41, reqid=client.4691.0:314, at_version=45'24, trim_to=0'0, roll_forward_to=38'21))", 2023-11-05T08:33:34.330 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:58.203830+0000", 2023-11-05T08:33:34.330 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 35.969772702999997, 2023-11-05T08:33:34.330 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.007519492, 2023-11-05T08:33:34.330 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:34.330 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:34.331 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:34.331 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.331 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:34.331 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.203830+0000", 2023-11-05T08:33:34.331 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.332 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.332 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.332 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:34.332 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.203830+0000", 2023-11-05T08:33:34.332 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.333 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.333 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.333 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:34.333 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.203853+0000", 2023-11-05T08:33:34.333 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3842e-05 2023-11-05T08:33:34.334 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.334 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.334 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:34.334 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.204181+0000", 2023-11-05T08:33:34.334 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000327401 2023-11-05T08:33:34.335 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.335 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.335 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:34.335 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.204212+0000", 2023-11-05T08:33:34.335 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.1596e-05 2023-11-05T08:33:34.335 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.336 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.336 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:34.336 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.204235+0000", 2023-11-05T08:33:34.336 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2319000000000001e-05 2023-11-05T08:33:34.336 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.336 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.337 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:34.337 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.206582+0000", 2023-11-05T08:33:34.337 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0023471669999999998 2023-11-05T08:33:34.337 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.337 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.338 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:34.338 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.206606+0000", 2023-11-05T08:33:34.338 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3864999999999999e-05 2023-11-05T08:33:34.338 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.338 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.339 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:34.339 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.211313+0000", 2023-11-05T08:33:34.339 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0047070539999999996 2023-11-05T08:33:34.339 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.339 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.339 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:34.340 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.211349+0000", 2023-11-05T08:33:34.340 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.6248000000000003e-05 2023-11-05T08:33:34.340 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.340 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:34.340 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.341 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.341 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.341 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.fs1 45/37 ECSubWrite(tid=42, reqid=client.4691.0:315, at_version=45'25, trim_to=0'0, roll_forward_to=38'21))", 2023-11-05T08:33:34.341 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:58.205469+0000", 2023-11-05T08:33:34.341 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 35.968133141999999, 2023-11-05T08:33:34.341 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0069671580000000002, 2023-11-05T08:33:34.342 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:34.342 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:34.342 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:34.342 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.342 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:34.343 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.205469+0000", 2023-11-05T08:33:34.343 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.343 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.343 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.343 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:34.344 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.205469+0000", 2023-11-05T08:33:34.344 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.344 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.344 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.344 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:34.344 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.205471+0000", 2023-11-05T08:33:34.345 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6300000000000001e-06 2023-11-05T08:33:34.345 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.345 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.345 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:34.345 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.205521+0000", 2023-11-05T08:33:34.346 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.0593999999999997e-05 2023-11-05T08:33:34.346 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.346 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.346 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:34.346 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.205551+0000", 2023-11-05T08:33:34.347 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.9555000000000001e-05 2023-11-05T08:33:34.347 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.347 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.347 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:34.347 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.205571+0000", 2023-11-05T08:33:34.347 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0446e-05 2023-11-05T08:33:34.348 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.348 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.348 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:34.348 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.208849+0000", 2023-11-05T08:33:34.348 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0032774129999999999 2023-11-05T08:33:34.349 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.349 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.349 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:34.349 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.208871+0000", 2023-11-05T08:33:34.349 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2582000000000002e-05 2023-11-05T08:33:34.350 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.350 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.350 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:34.350 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.212407+0000", 2023-11-05T08:33:34.350 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0035357750000000001 2023-11-05T08:33:34.351 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.351 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.351 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:34.351 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:58.212436+0000", 2023-11-05T08:33:34.351 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.9162999999999998e-05 2023-11-05T08:33:34.351 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.352 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:34.352 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.352 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.352 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.352 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.18s2 52/45 ECSubWrite(tid=205, reqid=client.4691.0:351, at_version=52'24, trim_to=0'0, roll_forward_to=51'23))", 2023-11-05T08:33:34.353 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:05.282039+0000", 2023-11-05T08:33:34.353 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 28.891563369, 2023-11-05T08:33:34.353 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0066848970000000004, 2023-11-05T08:33:34.353 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:34.353 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:34.354 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:34.354 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.354 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:34.354 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.282039+0000", 2023-11-05T08:33:34.354 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.354 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.355 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.355 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:34.355 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.282039+0000", 2023-11-05T08:33:34.355 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.355 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.356 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.356 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:34.356 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.282040+0000", 2023-11-05T08:33:34.356 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.1999999999999999e-07 2023-11-05T08:33:34.356 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.356 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.357 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:34.357 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.282608+0000", 2023-11-05T08:33:34.357 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00056786599999999999 2023-11-05T08:33:34.357 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.357 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.358 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:34.358 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.282651+0000", 2023-11-05T08:33:34.358 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.3089e-05 2023-11-05T08:33:34.358 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.358 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.359 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:34.359 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.282674+0000", 2023-11-05T08:33:34.359 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3329e-05 2023-11-05T08:33:34.359 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.359 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.359 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:34.360 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.282911+0000", 2023-11-05T08:33:34.360 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000237443 2023-11-05T08:33:34.360 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.360 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.360 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:34.360 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.282943+0000", 2023-11-05T08:33:34.361 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.1928999999999999e-05 2023-11-05T08:33:34.361 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.361 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.361 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:34.361 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.288631+0000", 2023-11-05T08:33:34.362 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0056872720000000002 2023-11-05T08:33:34.362 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.362 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.362 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:34.362 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.288724+0000", 2023-11-05T08:33:34.362 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.3249000000000001e-05 2023-11-05T08:33:34.363 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.363 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:34.363 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.363 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.363 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.364 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:380 3.es0 3:7b35d524:::smithi09039252-34:head [delete] snapc 17=[17,7] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e57)", 2023-11-05T08:33:34.364 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:10.278638+0000", 2023-11-05T08:33:34.364 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 23.894964116000001, 2023-11-05T08:33:34.364 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.010515907, 2023-11-05T08:33:34.364 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:34.365 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:34.365 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:33:34.365 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:33:34.365 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:34.365 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 380 2023-11-05T08:33:34.365 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.366 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:34.366 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.366 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:34.366 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:10.278638+0000", 2023-11-05T08:33:34.366 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.367 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.367 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.367 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:34.367 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:10.278638+0000", 2023-11-05T08:33:34.367 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.368 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.368 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.368 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:34.368 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:10.278641+0000", 2023-11-05T08:33:34.368 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.9100000000000001e-06 2023-11-05T08:33:34.369 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.369 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.369 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:34.369 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:10.278654+0000", 2023-11-05T08:33:34.369 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2532e-05 2023-11-05T08:33:34.369 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.370 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.370 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:34.370 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:10.278690+0000", 2023-11-05T08:33:34.370 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.6486000000000003e-05 2023-11-05T08:33:34.370 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.371 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.371 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:34.371 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:10.278708+0000", 2023-11-05T08:33:34.371 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8142e-05 2023-11-05T08:33:34.371 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.372 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.372 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:34.372 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:10.278888+0000", 2023-11-05T08:33:34.372 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00017937399999999999 2023-11-05T08:33:34.372 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.373 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.373 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op must wait for map", 2023-11-05T08:33:34.373 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:10.278913+0000", 2023-11-05T08:33:34.373 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5488999999999999e-05 2023-11-05T08:33:34.373 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.374 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.374 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:34.374 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:10.283146+0000", 2023-11-05T08:33:34.374 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.004232446 2023-11-05T08:33:34.374 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.375 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.375 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:33:34.375 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:10.283306+0000", 2023-11-05T08:33:34.375 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00015997400000000001 2023-11-05T08:33:34.375 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.376 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.376 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:34.376 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:10.284790+0000", 2023-11-05T08:33:34.376 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00148408 2023-11-05T08:33:34.376 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.376 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.377 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:34.377 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:10.287627+0000", 2023-11-05T08:33:34.377 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0028376489999999998 2023-11-05T08:33:34.377 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.377 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.378 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:33:34.378 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:10.289000+0000", 2023-11-05T08:33:34.378 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001372828 2023-11-05T08:33:34.378 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.378 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.379 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:34.379 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:10.289154+0000", 2023-11-05T08:33:34.379 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00015399700000000001 2023-11-05T08:33:34.379 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.379 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:34.380 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.380 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.380 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.380 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.2s1 65/45 ECSubWrite(tid=243, reqid=client.4691.0:470, at_version=65'18, trim_to=0'0, roll_forward_to=55'16))", 2023-11-05T08:33:34.380 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:18.554843+0000", 2023-11-05T08:33:34.381 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 15.618759662, 2023-11-05T08:33:34.381 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0095377659999999996, 2023-11-05T08:33:34.381 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:34.381 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:34.381 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:34.381 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.382 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:34.382 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.554843+0000", 2023-11-05T08:33:34.382 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.382 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.382 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.383 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:34.383 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.554843+0000", 2023-11-05T08:33:34.383 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.383 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.383 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.383 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:34.384 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.554844+0000", 2023-11-05T08:33:34.384 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.353e-06 2023-11-05T08:33:34.384 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.384 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.384 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:34.385 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.556081+0000", 2023-11-05T08:33:34.385 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0012370700000000001 2023-11-05T08:33:34.385 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.385 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.385 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:34.385 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.556114+0000", 2023-11-05T08:33:34.386 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2792e-05 2023-11-05T08:33:34.386 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.386 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.386 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:34.386 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.556169+0000", 2023-11-05T08:33:34.387 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.5331000000000003e-05 2023-11-05T08:33:34.387 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.387 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.387 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:34.387 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.556309+0000", 2023-11-05T08:33:34.387 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000139906 2023-11-05T08:33:34.388 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.388 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.388 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:34.388 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.556337+0000", 2023-11-05T08:33:34.388 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7376e-05 2023-11-05T08:33:34.389 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.389 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.389 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:34.389 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.564283+0000", 2023-11-05T08:33:34.389 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0079467340000000004 2023-11-05T08:33:34.390 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.390 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.390 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:34.390 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.564380+0000", 2023-11-05T08:33:34.390 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.7204e-05 2023-11-05T08:33:34.390 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.391 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:34.391 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.391 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.391 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.391 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:484 3.es0 3:71b68062:::smithi09039252-25:head [stat,delete] snapc 1c=[1a,7] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2023-11-05T08:33:34.392 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:18.555264+0000", 2023-11-05T08:33:34.392 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 15.618337954999999, 2023-11-05T08:33:34.392 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0088314239999999992, 2023-11-05T08:33:34.392 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:34.392 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:34.393 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:33:34.393 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:33:34.393 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:34.393 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 484 2023-11-05T08:33:34.393 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.394 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:34.394 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.394 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:34.394 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555264+0000", 2023-11-05T08:33:34.394 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.394 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.395 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.395 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:34.395 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555264+0000", 2023-11-05T08:33:34.395 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.395 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.396 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.396 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:34.396 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555267+0000", 2023-11-05T08:33:34.396 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2029999999999999e-06 2023-11-05T08:33:34.396 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.397 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.397 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:34.397 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555276+0000", 2023-11-05T08:33:34.397 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.8670000000000006e-06 2023-11-05T08:33:34.397 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.398 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.398 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:34.398 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555306+0000", 2023-11-05T08:33:34.398 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.9249000000000001e-05 2023-11-05T08:33:34.398 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.398 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.399 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:34.399 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555323+0000", 2023-11-05T08:33:34.399 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6898999999999999e-05 2023-11-05T08:33:34.399 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.399 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.400 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:34.400 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555436+0000", 2023-11-05T08:33:34.400 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00011365400000000001 2023-11-05T08:33:34.400 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.400 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.401 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:33:34.401 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555573+0000", 2023-11-05T08:33:34.401 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000136735 2023-11-05T08:33:34.401 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.401 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.401 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:34.402 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.556564+0000", 2023-11-05T08:33:34.402 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00099118899999999996 2023-11-05T08:33:34.402 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.402 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.402 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:34.403 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.563037+0000", 2023-11-05T08:33:34.403 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0064729319999999998 2023-11-05T08:33:34.403 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.403 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.403 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:33:34.404 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.563963+0000", 2023-11-05T08:33:34.404 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00092605599999999997 2023-11-05T08:33:34.404 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.404 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.404 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:34.405 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.564096+0000", 2023-11-05T08:33:34.405 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00013264 2023-11-05T08:33:34.405 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.405 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:34.405 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.406 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.406 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.406 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:505 3.es0 3:775e6266:::smithi09039252-12:head [stat,delete] snapc 21=[21,7] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2023-11-05T08:33:34.406 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:20.379990+0000", 2023-11-05T08:33:34.406 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 13.793612674, 2023-11-05T08:33:34.406 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.01062186, 2023-11-05T08:33:34.407 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:34.407 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:34.407 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:33:34.407 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:33:34.407 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:34.408 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 505 2023-11-05T08:33:34.408 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.408 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:34.408 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.408 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:34.408 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:20.379990+0000", 2023-11-05T08:33:34.409 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.409 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.409 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.409 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:34.409 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:20.379990+0000", 2023-11-05T08:33:34.410 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.410 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.410 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.410 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:34.410 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:20.379993+0000", 2023-11-05T08:33:34.411 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.817e-06 2023-11-05T08:33:34.411 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.411 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.411 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:34.411 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:20.380009+0000", 2023-11-05T08:33:34.412 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5512999999999999e-05 2023-11-05T08:33:34.412 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.412 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.412 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:34.412 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:20.380054+0000", 2023-11-05T08:33:34.412 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.4984999999999997e-05 2023-11-05T08:33:34.413 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.413 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.413 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:34.413 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:20.380078+0000", 2023-11-05T08:33:34.413 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3883000000000002e-05 2023-11-05T08:33:34.414 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.414 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.414 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:34.414 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:20.380312+0000", 2023-11-05T08:33:34.414 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00023429700000000001 2023-11-05T08:33:34.415 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.415 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.415 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op must wait for map", 2023-11-05T08:33:34.415 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:20.380330+0000", 2023-11-05T08:33:34.415 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.802e-05 2023-11-05T08:33:34.416 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.416 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.416 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:34.416 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:20.384598+0000", 2023-11-05T08:33:34.416 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0042682420000000002 2023-11-05T08:33:34.416 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.417 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.417 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:33:34.417 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:20.384765+0000", 2023-11-05T08:33:34.417 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000166591 2023-11-05T08:33:34.417 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.418 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.418 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:34.418 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:20.386117+0000", 2023-11-05T08:33:34.418 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001352036 2023-11-05T08:33:34.418 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.419 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.419 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:34.419 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:20.389679+0000", 2023-11-05T08:33:34.419 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.003562225 2023-11-05T08:33:34.419 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.419 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.420 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:33:34.420 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:20.390456+0000", 2023-11-05T08:33:34.420 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00077675499999999996 2023-11-05T08:33:34.420 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.420 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.421 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:34.421 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:20.390612+0000", 2023-11-05T08:33:34.421 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000155496 2023-11-05T08:33:34.421 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.421 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:34.421 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.422 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.422 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.422 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.fs1 77/37 ECSubWrite(tid=130, reqid=client.4691.0:579, at_version=77'44, trim_to=0'0, roll_forward_to=70'42))", 2023-11-05T08:33:34.422 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:30.539501+0000", 2023-11-05T08:33:34.422 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 3.6341012510000001, 2023-11-05T08:33:34.423 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0067640199999999999, 2023-11-05T08:33:34.423 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:34.423 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:34.423 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:34.423 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.424 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:34.424 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.539501+0000", 2023-11-05T08:33:34.424 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.424 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.424 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.425 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:34.425 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.539501+0000", 2023-11-05T08:33:34.425 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.425 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.425 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.426 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:34.426 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.539502+0000", 2023-11-05T08:33:34.426 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.8400000000000004e-07 2023-11-05T08:33:34.426 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.426 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.427 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:34.429 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.540088+0000", 2023-11-05T08:33:34.429 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00058642299999999998 2023-11-05T08:33:34.429 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.429 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.430 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:34.430 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.540157+0000", 2023-11-05T08:33:34.430 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.8879000000000002e-05 2023-11-05T08:33:34.430 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.430 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.431 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:34.431 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.540175+0000", 2023-11-05T08:33:34.431 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8105999999999999e-05 2023-11-05T08:33:34.431 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.431 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.432 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:34.432 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.540376+0000", 2023-11-05T08:33:34.432 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00020035499999999999 2023-11-05T08:33:34.432 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.432 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.433 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:34.433 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.540409+0000", 2023-11-05T08:33:34.433 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.3762e-05 2023-11-05T08:33:34.433 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.433 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.433 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:34.434 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.546184+0000", 2023-11-05T08:33:34.434 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0057750329999999997 2023-11-05T08:33:34.434 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.434 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.434 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:34.435 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.546265+0000", 2023-11-05T08:33:34.435 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.0778000000000005e-05 2023-11-05T08:33:34.435 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.435 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:34.435 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.436 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.436 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:34.436 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:33:34.438 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 reset_purged_snaps_last 2023-11-05T08:33:34.439 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-11-05T08:33:34.440 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:34.432+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:34.538 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:33:34.533+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:33:34.541 INFO:tasks.rados.rados.0.smithi090.stdout:258: done (4 left) 2023-11-05T08:33:34.541 INFO:tasks.rados.rados.0.smithi090.stdout:259: done (3 left) 2023-11-05T08:33:34.541 INFO:tasks.rados.rados.0.smithi090.stdout:260: done (2 left) 2023-11-05T08:33:34.542 INFO:tasks.rados.rados.0.smithi090.stdout:261: done (1 left) 2023-11-05T08:33:34.542 INFO:tasks.rados.rados.0.smithi090.stdout:262: read oid 23 snap -1 2023-11-05T08:33:34.542 INFO:tasks.rados.rados.0.smithi090.stdout:262: expect (ObjNum 60 snap 1 seq_num 60) 2023-11-05T08:33:34.542 INFO:tasks.rados.rados.0.smithi090.stdout:263: snap_create 2023-11-05T08:33:34.543 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-11-05T08:33:34.563 INFO:tasks.rados.rados.0.smithi090.stdout:262: expect (ObjNum 60 snap 1 seq_num 60) 2023-11-05T08:33:34.575 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:33:34.575 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2023-11-05T08:33:34.575 INFO:teuthology.orchestra.run.smithi069.stdout: "num_ops": 0 2023-11-05T08:33:34.576 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:33:34.590 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-11-05T08:33:34.616 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 reset_purged_snaps_last 2023-11-05T08:33:34.639 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:33:34.635+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:33:34.739 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:33:34.737+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:33:34.758 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:33:34.758 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2023-11-05T08:33:34.758 INFO:teuthology.orchestra.run.smithi069.stdout: "complaint_time": 30, 2023-11-05T08:33:34.759 INFO:teuthology.orchestra.run.smithi069.stdout: "num_blocked_ops": 0 2023-11-05T08:33:34.759 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:33:34.772 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-11-05T08:33:34.776 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 reset_purged_snaps_last 2023-11-05T08:33:34.840 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:33:34.837+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:33:34.937 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 reset_purged_snaps_last 2023-11-05T08:33:34.941 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:33:34.937+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:33:34.943 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:33:34.943 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 20, 2023-11-05T08:33:34.943 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 600, 2023-11-05T08:33:34.943 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [ 2023-11-05T08:33:34.943 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.944 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.5s6 35/30 ECSubWrite(tid=20, reqid=client.4691.0:221, at_version=35'11, trim_to=0'0, roll_forward_to=33'9))", 2023-11-05T08:33:34.944 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:48.086545+0000", 2023-11-05T08:33:34.944 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 46.854611173000002, 2023-11-05T08:33:34.944 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.006420767, 2023-11-05T08:33:34.944 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:34.945 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:34.945 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:34.945 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.945 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:34.946 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.086545+0000", 2023-11-05T08:33:34.946 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.946 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.946 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.947 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:34.947 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.086545+0000", 2023-11-05T08:33:34.947 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.947 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.947 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.947 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:34.948 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.086545+0000", 2023-11-05T08:33:34.948 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.2300000000000002e-07 2023-11-05T08:33:34.948 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.948 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.948 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:34.949 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.086937+0000", 2023-11-05T08:33:34.949 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000391522 2023-11-05T08:33:34.949 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.949 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.949 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:34.950 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.086963+0000", 2023-11-05T08:33:34.950 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6114000000000001e-05 2023-11-05T08:33:34.950 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.950 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.950 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:34.951 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.086981+0000", 2023-11-05T08:33:34.951 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8289000000000001e-05 2023-11-05T08:33:34.951 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.951 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.951 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:34.952 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.087151+0000", 2023-11-05T08:33:34.952 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000169603 2023-11-05T08:33:34.952 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.952 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.952 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:34.952 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.087176+0000", 2023-11-05T08:33:34.953 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5488000000000001e-05 2023-11-05T08:33:34.953 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.953 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.953 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:34.953 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.092911+0000", 2023-11-05T08:33:34.954 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0057347580000000004 2023-11-05T08:33:34.954 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.954 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.954 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:34.954 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:48.092966+0000", 2023-11-05T08:33:34.955 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.4570000000000001e-05 2023-11-05T08:33:34.955 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.955 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:34.955 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.955 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.956 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.956 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.8s6 36/30 ECSubWrite(tid=23, reqid=osd.5.0:23, at_version=36'15, trim_to=0'0, roll_forward_to=34'13))", 2023-11-05T08:33:34.956 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.079553+0000", 2023-11-05T08:33:34.956 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 45.861602775000001, 2023-11-05T08:33:34.956 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0063101340000000002, 2023-11-05T08:33:34.956 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:34.957 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:34.957 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:34.957 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.957 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:34.957 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079553+0000", 2023-11-05T08:33:34.958 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.958 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.958 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.958 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:34.958 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079553+0000", 2023-11-05T08:33:34.959 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.959 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.959 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.959 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:34.959 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079554+0000", 2023-11-05T08:33:34.960 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.1299999999999998e-07 2023-11-05T08:33:34.960 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.960 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.960 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:34.960 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079604+0000", 2023-11-05T08:33:34.961 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.0157000000000001e-05 2023-11-05T08:33:34.961 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.961 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.961 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:34.961 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079671+0000", 2023-11-05T08:33:34.962 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.6919000000000001e-05 2023-11-05T08:33:34.962 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.962 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.962 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:34.962 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079696+0000", 2023-11-05T08:33:34.963 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4192000000000001e-05 2023-11-05T08:33:34.963 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.963 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.963 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:34.963 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079889+0000", 2023-11-05T08:33:34.964 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00019337400000000001 2023-11-05T08:33:34.964 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.964 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.964 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:34.964 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079920+0000", 2023-11-05T08:33:34.964 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0682000000000002e-05 2023-11-05T08:33:34.965 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.965 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.965 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:34.965 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.085773+0000", 2023-11-05T08:33:34.965 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0058536889999999996 2023-11-05T08:33:34.966 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.966 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.966 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:34.966 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.085864+0000", 2023-11-05T08:33:34.966 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.0208000000000001e-05 2023-11-05T08:33:34.967 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.967 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:34.967 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.967 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.967 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.968 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.es5 41/30 ECSubWrite(tid=40, reqid=osd.2.0:40, at_version=41'27, trim_to=0'0, roll_forward_to=40'25))", 2023-11-05T08:33:34.968 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.123345+0000", 2023-11-05T08:33:34.968 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 40.817810772000001, 2023-11-05T08:33:34.968 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0086456399999999996, 2023-11-05T08:33:34.968 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:34.969 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:34.969 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:34.969 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.969 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:34.969 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.123345+0000", 2023-11-05T08:33:34.969 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.970 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.970 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.970 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:34.970 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.123345+0000", 2023-11-05T08:33:34.970 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.971 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.971 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.971 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:34.971 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.123346+0000", 2023-11-05T08:33:34.971 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.3000000000000001e-07 2023-11-05T08:33:34.972 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.972 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.972 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:34.972 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.123401+0000", 2023-11-05T08:33:34.972 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.4583999999999998e-05 2023-11-05T08:33:34.973 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.973 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.973 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:34.973 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.123432+0000", 2023-11-05T08:33:34.973 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.1488000000000001e-05 2023-11-05T08:33:34.974 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.974 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.974 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:34.974 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.123458+0000", 2023-11-05T08:33:34.974 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5925999999999999e-05 2023-11-05T08:33:34.975 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.975 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.975 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:34.975 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.123705+0000", 2023-11-05T08:33:34.975 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00024677599999999998 2023-11-05T08:33:34.976 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.976 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.976 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:34.976 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.123734+0000", 2023-11-05T08:33:34.977 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.9002000000000001e-05 2023-11-05T08:33:34.977 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.977 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.977 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:34.977 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131911+0000", 2023-11-05T08:33:34.977 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0081774080000000006 2023-11-05T08:33:34.978 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.978 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.978 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:34.978 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131991+0000", 2023-11-05T08:33:34.978 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.9925999999999993e-05 2023-11-05T08:33:34.979 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.979 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:34.979 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.979 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.979 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.980 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.as5 41/33 ECSubWrite(tid=35, reqid=osd.7.0:35, at_version=41'12, trim_to=0'0, roll_forward_to=40'10))", 2023-11-05T08:33:34.980 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.129002+0000", 2023-11-05T08:33:34.980 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 40.812154225999997, 2023-11-05T08:33:34.980 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.010000177000000001, 2023-11-05T08:33:34.980 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:34.981 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:34.981 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:34.981 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.981 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:34.981 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.129002+0000", 2023-11-05T08:33:34.982 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.982 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.982 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.982 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:34.982 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.129002+0000", 2023-11-05T08:33:34.983 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.983 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.983 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.983 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:34.983 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.129003+0000", 2023-11-05T08:33:34.984 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.4000000000000001e-07 2023-11-05T08:33:34.984 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.984 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.984 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:34.984 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.129074+0000", 2023-11-05T08:33:34.984 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.1649999999999993e-05 2023-11-05T08:33:34.985 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.985 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.985 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:34.985 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.129105+0000", 2023-11-05T08:33:34.985 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.1161999999999997e-05 2023-11-05T08:33:34.986 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.986 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.986 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:34.986 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.129132+0000", 2023-11-05T08:33:34.986 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6874999999999999e-05 2023-11-05T08:33:34.987 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.987 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.987 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:34.987 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.133281+0000", 2023-11-05T08:33:34.987 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0041491239999999997 2023-11-05T08:33:34.988 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.988 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.988 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:34.988 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.133306+0000", 2023-11-05T08:33:34.988 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4408000000000001e-05 2023-11-05T08:33:34.989 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.989 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.989 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:34.989 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.138905+0000", 2023-11-05T08:33:34.989 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0055986899999999999 2023-11-05T08:33:34.990 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.990 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.990 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:34.990 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.139002+0000", 2023-11-05T08:33:34.990 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.7627999999999994e-05 2023-11-05T08:33:34.991 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.991 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:34.991 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:34.991 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.991 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.992 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.as5 41/33 ECSubWrite(tid=36, reqid=osd.7.0:36, at_version=41'14, trim_to=0'0, roll_forward_to=40'10))", 2023-11-05T08:33:34.992 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.131389+0000", 2023-11-05T08:33:34.992 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 40.809767237000003, 2023-11-05T08:33:34.992 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0079481799999999991, 2023-11-05T08:33:34.992 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:34.992 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:34.993 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:34.993 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.993 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:34.993 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131389+0000", 2023-11-05T08:33:34.993 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.994 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.994 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.994 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:34.994 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131389+0000", 2023-11-05T08:33:34.994 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:34.995 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.995 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.995 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:34.995 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131391+0000", 2023-11-05T08:33:34.995 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.753e-06 2023-11-05T08:33:34.995 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.996 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.996 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:34.996 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131444+0000", 2023-11-05T08:33:34.996 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.3363999999999999e-05 2023-11-05T08:33:34.996 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.997 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.997 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:34.997 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131540+0000", 2023-11-05T08:33:34.997 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.6075000000000004e-05 2023-11-05T08:33:34.997 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.998 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.998 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:34.998 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131580+0000", 2023-11-05T08:33:34.998 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.9351000000000003e-05 2023-11-05T08:33:34.998 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.999 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:34.999 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:34.999 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.135074+0000", 2023-11-05T08:33:34.999 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.003494259 2023-11-05T08:33:34.999 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:34.999 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.000 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:35.000 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.135103+0000", 2023-11-05T08:33:35.000 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.9342000000000001e-05 2023-11-05T08:33:35.000 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.000 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.001 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:35.001 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.139247+0000", 2023-11-05T08:33:35.001 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0041437940000000001 2023-11-05T08:33:35.001 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.001 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.002 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:35.002 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.139337+0000", 2023-11-05T08:33:35.002 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.0241999999999994e-05 2023-11-05T08:33:35.002 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.002 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:35.003 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.003 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.003 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.003 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.es5 41/30 ECSubWrite(tid=41, reqid=osd.2.0:40, at_version=0'0, trim_to=0'0, roll_forward_to=41'27))", 2023-11-05T08:33:35.003 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.149656+0000", 2023-11-05T08:33:35.004 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 40.791500055, 2023-11-05T08:33:35.004 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0061863129999999997, 2023-11-05T08:33:35.004 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:35.004 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:35.004 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:35.005 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.005 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:35.005 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.149656+0000", 2023-11-05T08:33:35.005 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.005 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.005 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.006 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:35.006 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.149656+0000", 2023-11-05T08:33:35.006 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.006 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.006 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.007 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:35.007 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.149658+0000", 2023-11-05T08:33:35.007 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.86e-06 2023-11-05T08:33:35.007 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.007 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.008 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:35.008 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.149687+0000", 2023-11-05T08:33:35.008 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8824999999999999e-05 2023-11-05T08:33:35.008 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.008 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.009 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:35.009 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.149719+0000", 2023-11-05T08:33:35.009 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2404999999999999e-05 2023-11-05T08:33:35.009 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.009 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.010 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:35.010 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.149742+0000", 2023-11-05T08:33:35.010 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2779e-05 2023-11-05T08:33:35.010 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.010 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.011 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:35.011 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.149992+0000", 2023-11-05T08:33:35.011 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00024954300000000002 2023-11-05T08:33:35.011 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.011 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.012 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:35.012 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.150017+0000", 2023-11-05T08:33:35.012 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5698e-05 2023-11-05T08:33:35.012 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.012 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.012 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:35.013 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.155762+0000", 2023-11-05T08:33:35.013 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0057443559999999999 2023-11-05T08:33:35.013 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.013 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.013 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:35.014 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.155842+0000", 2023-11-05T08:33:35.014 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.0846999999999994e-05 2023-11-05T08:33:35.014 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.014 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:35.014 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.015 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.015 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.015 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.2s3 41/40 ECSubWrite(tid=188, reqid=osd.13.0:188, at_version=41'11, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:35.015 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.170574+0000", 2023-11-05T08:33:35.015 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 40.770582212000001, 2023-11-05T08:33:35.016 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.006673193, 2023-11-05T08:33:35.016 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:35.016 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:35.016 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:35.016 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.016 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:35.017 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.170574+0000", 2023-11-05T08:33:35.017 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.017 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.017 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.017 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:35.018 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.170574+0000", 2023-11-05T08:33:35.018 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.018 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.018 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.018 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:35.019 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.170575+0000", 2023-11-05T08:33:35.019 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3629999999999999e-06 2023-11-05T08:33:35.019 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.019 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.019 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:35.019 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.170619+0000", 2023-11-05T08:33:35.020 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.4057999999999997e-05 2023-11-05T08:33:35.020 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.020 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.020 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:35.020 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.170643+0000", 2023-11-05T08:33:35.021 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4070000000000002e-05 2023-11-05T08:33:35.021 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.021 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.021 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:35.021 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.170665+0000", 2023-11-05T08:33:35.022 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1216000000000002e-05 2023-11-05T08:33:35.022 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.022 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.022 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:35.022 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.170789+0000", 2023-11-05T08:33:35.023 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000124416 2023-11-05T08:33:35.023 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.023 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.023 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:35.023 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.170810+0000", 2023-11-05T08:33:35.023 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0735999999999999e-05 2023-11-05T08:33:35.024 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.024 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.024 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:35.024 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.177176+0000", 2023-11-05T08:33:35.024 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0063664289999999998 2023-11-05T08:33:35.025 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.025 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.025 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:35.025 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.177247+0000", 2023-11-05T08:33:35.025 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.0905e-05 2023-11-05T08:33:35.026 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.026 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:35.026 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.026 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.026 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.027 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.as5 42/33 ECSubWrite(tid=38, reqid=osd.7.0:38, at_version=42'16, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:33:35.027 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:55.127656+0000", 2023-11-05T08:33:35.027 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 39.813500263999998, 2023-11-05T08:33:35.027 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.016413512000000002, 2023-11-05T08:33:35.027 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:35.028 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:35.028 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:35.028 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.028 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:35.028 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127656+0000", 2023-11-05T08:33:35.029 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.029 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.029 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.029 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:35.029 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127656+0000", 2023-11-05T08:33:35.029 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.030 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.030 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.030 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:35.030 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127656+0000", 2023-11-05T08:33:35.030 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.2699999999999999e-07 2023-11-05T08:33:35.031 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.031 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.031 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:35.031 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127732+0000", 2023-11-05T08:33:35.031 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.5978999999999999e-05 2023-11-05T08:33:35.032 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.032 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.032 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:35.032 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127763+0000", 2023-11-05T08:33:35.032 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0672e-05 2023-11-05T08:33:35.033 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.033 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.033 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:35.033 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127785+0000", 2023-11-05T08:33:35.033 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2059e-05 2023-11-05T08:33:35.033 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.034 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.034 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:35.034 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127975+0000", 2023-11-05T08:33:35.034 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000189946 2023-11-05T08:33:35.034 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.035 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.035 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:35.035 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127999+0000", 2023-11-05T08:33:35.035 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3555999999999999e-05 2023-11-05T08:33:35.035 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.036 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.036 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:35.036 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.143967+0000", 2023-11-05T08:33:35.036 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.015968781000000001 2023-11-05T08:33:35.036 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.037 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.037 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:35.037 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.144069+0000", 2023-11-05T08:33:35.037 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000101992 2023-11-05T08:33:35.037 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.038 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:35.038 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.038 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.038 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.038 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.as5 42/33 ECSubWrite(tid=39, reqid=osd.7.0:39, at_version=42'18, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:33:35.039 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:55.127880+0000", 2023-11-05T08:33:35.039 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 39.813275779000001, 2023-11-05T08:33:35.041 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.019355170000000001, 2023-11-05T08:33:35.041 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:35.041 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:35.041 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:35.042 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.042 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:35.042 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127880+0000", 2023-11-05T08:33:35.042 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.043 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.043 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.043 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:35.043 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127880+0000", 2023-11-05T08:33:35.043 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.044 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.044 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.044 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:35.044 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127882+0000", 2023-11-05T08:33:35.044 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.187e-06 2023-11-05T08:33:35.044 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.045 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.045 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:35.045 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127934+0000", 2023-11-05T08:33:35.045 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.2553000000000003e-05 2023-11-05T08:33:35.045 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.046 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.046 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:35.046 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127961+0000", 2023-11-05T08:33:35.046 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6999e-05 2023-11-05T08:33:35.046 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.047 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.047 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:35.047 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127983+0000", 2023-11-05T08:33:35.047 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1778999999999999e-05 2023-11-05T08:33:35.047 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.048 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.048 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:35.048 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.129155+0000", 2023-11-05T08:33:35.048 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001172022 2023-11-05T08:33:35.048 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.049 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.049 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:35.049 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.129180+0000", 2023-11-05T08:33:35.049 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4841000000000002e-05 2023-11-05T08:33:35.049 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.050 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.050 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:35.050 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.147146+0000", 2023-11-05T08:33:35.050 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.017965990000000001 2023-11-05T08:33:35.050 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.051 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.051 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:35.051 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.147236+0000", 2023-11-05T08:33:35.051 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.9798999999999999e-05 2023-11-05T08:33:35.051 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.051 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:35.052 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.052 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.052 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.052 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.es5 42/30 ECSubWrite(tid=43, reqid=osd.2.0:43, at_version=42'29, trim_to=0'0, roll_forward_to=41'27))", 2023-11-05T08:33:35.052 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:55.134988+0000", 2023-11-05T08:33:35.053 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 39.806167963, 2023-11-05T08:33:35.053 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.012343942, 2023-11-05T08:33:35.053 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:35.053 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:35.053 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:35.054 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.054 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:35.054 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.134988+0000", 2023-11-05T08:33:35.054 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.054 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.054 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.055 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:35.055 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.134988+0000", 2023-11-05T08:33:35.055 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.055 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.055 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.056 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:35.056 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.134989+0000", 2023-11-05T08:33:35.056 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.6000000000000003e-07 2023-11-05T08:33:35.056 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.056 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.057 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:35.057 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.135039+0000", 2023-11-05T08:33:35.057 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.9879999999999997e-05 2023-11-05T08:33:35.057 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.057 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.057 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:35.058 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.135069+0000", 2023-11-05T08:33:35.058 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0445000000000001e-05 2023-11-05T08:33:35.058 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.058 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.058 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:35.059 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.135094+0000", 2023-11-05T08:33:35.059 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4646000000000001e-05 2023-11-05T08:33:35.059 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.059 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.059 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:35.060 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.135758+0000", 2023-11-05T08:33:35.060 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00066366799999999996 2023-11-05T08:33:35.060 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.060 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.060 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:35.061 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.135780+0000", 2023-11-05T08:33:35.061 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2509000000000001e-05 2023-11-05T08:33:35.061 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.061 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.061 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:35.062 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.147237+0000", 2023-11-05T08:33:35.062 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.011456401999999999 2023-11-05T08:33:35.062 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.062 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.062 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:35.063 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.147332+0000", 2023-11-05T08:33:35.063 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.5631999999999995e-05 2023-11-05T08:33:35.063 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.063 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:35.063 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.064 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.064 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.064 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.18s1 51/45 ECSubWrite(tid=202, reqid=client.4691.0:335, at_version=51'23, trim_to=0'0, roll_forward_to=49'21))", 2023-11-05T08:33:35.064 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:04.244158+0000", 2023-11-05T08:33:35.064 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 30.696997746000001, 2023-11-05T08:33:35.065 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0079120630000000004, 2023-11-05T08:33:35.065 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:35.065 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:35.065 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:35.065 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.066 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:35.066 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:04.244158+0000", 2023-11-05T08:33:35.066 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.066 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.066 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.067 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:35.067 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:04.244158+0000", 2023-11-05T08:33:35.067 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.067 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.067 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.068 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:35.068 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:04.244160+0000", 2023-11-05T08:33:35.068 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6670000000000001e-06 2023-11-05T08:33:35.068 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.068 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.069 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:35.069 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:04.244222+0000", 2023-11-05T08:33:35.069 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.1734000000000002e-05 2023-11-05T08:33:35.069 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.069 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.069 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:35.070 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:04.244256+0000", 2023-11-05T08:33:35.070 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.3822000000000002e-05 2023-11-05T08:33:35.070 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.070 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.070 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:35.071 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:04.244278+0000", 2023-11-05T08:33:35.071 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2419e-05 2023-11-05T08:33:35.071 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.071 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.071 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:35.072 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:04.246542+0000", 2023-11-05T08:33:35.072 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0022635989999999998 2023-11-05T08:33:35.072 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.072 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.072 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:35.073 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:04.246558+0000", 2023-11-05T08:33:35.073 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5936000000000001e-05 2023-11-05T08:33:35.073 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.073 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.073 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:35.074 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:04.251996+0000", 2023-11-05T08:33:35.074 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0054384029999999996 2023-11-05T08:33:35.074 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.074 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.074 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:35.075 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:04.252071+0000", 2023-11-05T08:33:35.075 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.4482999999999998e-05 2023-11-05T08:33:35.075 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.075 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:35.075 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.076 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.076 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.076 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.as5 52/45 ECSubWrite(tid=45, reqid=client.4691.0:349, at_version=52'21, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:35.076 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:05.263668+0000", 2023-11-05T08:33:35.076 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 29.677488293, 2023-11-05T08:33:35.077 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0089752809999999999, 2023-11-05T08:33:35.077 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:35.077 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:35.077 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:35.077 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.078 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:35.078 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.263668+0000", 2023-11-05T08:33:35.078 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.078 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.078 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.079 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:35.079 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.263668+0000", 2023-11-05T08:33:35.079 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.079 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.079 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.080 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:35.080 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.263669+0000", 2023-11-05T08:33:35.080 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.3600000000000003e-07 2023-11-05T08:33:35.080 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.080 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.080 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:35.081 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.264055+0000", 2023-11-05T08:33:35.081 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00038651799999999997 2023-11-05T08:33:35.081 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.081 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.081 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:35.082 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.264091+0000", 2023-11-05T08:33:35.082 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.6089e-05 2023-11-05T08:33:35.082 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.082 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.082 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:35.083 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.264119+0000", 2023-11-05T08:33:35.083 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7665e-05 2023-11-05T08:33:35.083 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.083 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.083 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:35.084 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.264344+0000", 2023-11-05T08:33:35.084 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00022567400000000001 2023-11-05T08:33:35.084 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.084 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.084 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:35.085 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.264372+0000", 2023-11-05T08:33:35.085 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7118999999999999e-05 2023-11-05T08:33:35.085 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.085 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.085 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:35.086 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.272564+0000", 2023-11-05T08:33:35.086 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0081919829999999999 2023-11-05T08:33:35.086 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.086 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.086 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:35.087 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.272643+0000", 2023-11-05T08:33:35.087 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.9597000000000004e-05 2023-11-05T08:33:35.087 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.087 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:35.087 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.088 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.088 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.088 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.as5 61/45 ECSubWrite(tid=57, reqid=client.4691.0:405, at_version=61'23, trim_to=0'0, roll_forward_to=52'21))", 2023-11-05T08:33:35.088 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:14.365789+0000", 2023-11-05T08:33:35.088 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 20.575367549999999, 2023-11-05T08:33:35.089 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0091284539999999994, 2023-11-05T08:33:35.089 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:35.089 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:35.089 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:35.089 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.089 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:35.090 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.365789+0000", 2023-11-05T08:33:35.090 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.090 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.090 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.090 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:35.091 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.365789+0000", 2023-11-05T08:33:35.091 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.091 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.091 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.091 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:35.092 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.365790+0000", 2023-11-05T08:33:35.092 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.2299999999999999e-07 2023-11-05T08:33:35.092 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.092 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.092 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:35.093 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.366363+0000", 2023-11-05T08:33:35.093 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00057346199999999995 2023-11-05T08:33:35.093 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.093 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.093 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:35.094 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.366389+0000", 2023-11-05T08:33:35.094 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6441999999999999e-05 2023-11-05T08:33:35.094 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.094 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.094 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:35.095 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.366406+0000", 2023-11-05T08:33:35.095 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6549999999999999e-05 2023-11-05T08:33:35.095 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.095 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.095 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:35.096 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.366590+0000", 2023-11-05T08:33:35.096 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00018348099999999999 2023-11-05T08:33:35.096 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.096 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.096 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:35.096 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.366623+0000", 2023-11-05T08:33:35.097 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.3822000000000002e-05 2023-11-05T08:33:35.097 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.097 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.097 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:35.097 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.374821+0000", 2023-11-05T08:33:35.098 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0081977999999999999 2023-11-05T08:33:35.098 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.098 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.098 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:35.098 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.374917+0000", 2023-11-05T08:33:35.099 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.5974e-05 2023-11-05T08:33:35.099 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.099 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:35.099 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.099 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.100 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.100 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.18s1 62/45 ECSubWrite(tid=217, reqid=client.4691.0:423, at_version=62'30, trim_to=0'0, roll_forward_to=52'28))", 2023-11-05T08:33:35.100 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:15.370741+0000", 2023-11-05T08:33:35.100 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 19.570415299, 2023-11-05T08:33:35.100 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0079249650000000008, 2023-11-05T08:33:35.101 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:35.101 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:35.101 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:35.101 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.101 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:35.101 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:15.370741+0000", 2023-11-05T08:33:35.102 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.102 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.102 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.102 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:35.102 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:15.370741+0000", 2023-11-05T08:33:35.103 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.103 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.103 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.103 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:35.103 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:15.370742+0000", 2023-11-05T08:33:35.104 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.1299999999999999e-07 2023-11-05T08:33:35.104 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.104 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.104 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:35.104 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:15.370976+0000", 2023-11-05T08:33:35.105 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000234236 2023-11-05T08:33:35.105 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.105 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.105 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:35.105 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:15.371018+0000", 2023-11-05T08:33:35.105 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.2608999999999998e-05 2023-11-05T08:33:35.106 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.106 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.106 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:35.106 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:15.371042+0000", 2023-11-05T08:33:35.107 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3949000000000001e-05 2023-11-05T08:33:35.107 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.107 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.107 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:35.107 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:15.371226+0000", 2023-11-05T08:33:35.108 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00018342400000000001 2023-11-05T08:33:35.108 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.108 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.108 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:35.108 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:15.371260+0000", 2023-11-05T08:33:35.108 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.4096e-05 2023-11-05T08:33:35.109 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.109 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.109 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:35.109 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:15.378600+0000", 2023-11-05T08:33:35.109 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0073403929999999997 2023-11-05T08:33:35.110 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.110 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.110 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:35.110 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:15.378666+0000", 2023-11-05T08:33:35.110 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.5544999999999997e-05 2023-11-05T08:33:35.111 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.111 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:35.111 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.111 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.111 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.111 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.2s3 65/45 ECSubWrite(tid=243, reqid=client.4691.0:470, at_version=65'18, trim_to=0'0, roll_forward_to=55'16))", 2023-11-05T08:33:35.112 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:18.556742+0000", 2023-11-05T08:33:35.112 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 16.384413989999999, 2023-11-05T08:33:35.112 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0064602469999999997, 2023-11-05T08:33:35.112 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:35.112 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:35.113 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:35.113 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.113 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:35.113 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.556742+0000", 2023-11-05T08:33:35.113 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.114 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.114 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.114 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:35.114 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.556742+0000", 2023-11-05T08:33:35.114 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.114 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.115 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.115 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:35.115 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.556743+0000", 2023-11-05T08:33:35.115 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.4899999999999995e-07 2023-11-05T08:33:35.115 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.116 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.116 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:35.116 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.557754+0000", 2023-11-05T08:33:35.116 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001010871 2023-11-05T08:33:35.116 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.117 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.117 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:35.117 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.557768+0000", 2023-11-05T08:33:35.117 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4092e-05 2023-11-05T08:33:35.117 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.117 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.118 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:35.118 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.557777+0000", 2023-11-05T08:33:35.118 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.7830000000000004e-06 2023-11-05T08:33:35.118 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.118 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.119 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:35.119 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.557848+0000", 2023-11-05T08:33:35.119 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.1762999999999997e-05 2023-11-05T08:33:35.119 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.119 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.120 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:35.120 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.557870+0000", 2023-11-05T08:33:35.120 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1999999999999999e-05 2023-11-05T08:33:35.120 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.120 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.121 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:35.121 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.563109+0000", 2023-11-05T08:33:35.121 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0052390780000000003 2023-11-05T08:33:35.121 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.121 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.121 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:35.122 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.563202+0000", 2023-11-05T08:33:35.122 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.3110999999999997e-05 2023-11-05T08:33:35.122 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.122 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:35.122 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.123 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.123 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.123 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.bs4 66/37 ECSubWrite(tid=108, reqid=client.4691.0:496, at_version=66'23, trim_to=0'0, roll_forward_to=65'21))", 2023-11-05T08:33:35.123 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:19.399298+0000", 2023-11-05T08:33:35.123 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 15.541857796, 2023-11-05T08:33:35.124 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.012016639000000001, 2023-11-05T08:33:35.124 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:35.124 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:35.124 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:35.124 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.125 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:35.125 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.399298+0000", 2023-11-05T08:33:35.125 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.125 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.125 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.125 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:35.126 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.399298+0000", 2023-11-05T08:33:35.126 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.126 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.126 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.126 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:35.127 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.399299+0000", 2023-11-05T08:33:35.127 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.4399999999999999e-07 2023-11-05T08:33:35.127 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.127 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.127 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:35.128 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.399993+0000", 2023-11-05T08:33:35.128 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00069384999999999998 2023-11-05T08:33:35.128 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.128 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.128 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:35.128 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.400030+0000", 2023-11-05T08:33:35.129 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.6508999999999999e-05 2023-11-05T08:33:35.129 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.129 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.129 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:35.129 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.400046+0000", 2023-11-05T08:33:35.130 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6266e-05 2023-11-05T08:33:35.130 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.130 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.130 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:35.130 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.400213+0000", 2023-11-05T08:33:35.131 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00016678099999999999 2023-11-05T08:33:35.131 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.131 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.131 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:35.131 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.400240+0000", 2023-11-05T08:33:35.132 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6979999999999999e-05 2023-11-05T08:33:35.132 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.132 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.132 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:35.132 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.411240+0000", 2023-11-05T08:33:35.133 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.010999978000000001 2023-11-05T08:33:35.133 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.133 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.133 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:35.133 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.411315+0000", 2023-11-05T08:33:35.134 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.5531000000000006e-05 2023-11-05T08:33:35.134 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.134 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:35.134 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.134 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.135 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.135 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.bs4 66/37 ECSubWrite(tid=109, reqid=client.4691.0:497, at_version=66'24, trim_to=0'0, roll_forward_to=65'21))", 2023-11-05T08:33:35.135 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:19.403912+0000", 2023-11-05T08:33:35.135 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 15.537244118, 2023-11-05T08:33:35.135 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0085903530000000002, 2023-11-05T08:33:35.135 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:35.136 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:35.136 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:35.136 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.136 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:35.136 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.403912+0000", 2023-11-05T08:33:35.137 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.137 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.137 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.137 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:35.137 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.403912+0000", 2023-11-05T08:33:35.138 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.138 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.138 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.138 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:35.138 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.403914+0000", 2023-11-05T08:33:35.139 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8500000000000001e-06 2023-11-05T08:33:35.139 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.139 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.139 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:35.139 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.404266+0000", 2023-11-05T08:33:35.140 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00035245300000000002 2023-11-05T08:33:35.140 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.140 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.140 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:35.140 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.404292+0000", 2023-11-05T08:33:35.141 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6049000000000001e-05 2023-11-05T08:33:35.141 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.141 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.141 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:35.141 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.404308+0000", 2023-11-05T08:33:35.141 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5807e-05 2023-11-05T08:33:35.142 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.142 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.142 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:35.142 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.404479+0000", 2023-11-05T08:33:35.142 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00017073500000000001 2023-11-05T08:33:35.143 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.143 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.143 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:35.143 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.404507+0000", 2023-11-05T08:33:35.143 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7614999999999999e-05 2023-11-05T08:33:35.144 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.144 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.144 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:35.144 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.412426+0000", 2023-11-05T08:33:35.144 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0079197099999999999 2023-11-05T08:33:35.145 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.145 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.145 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:35.145 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.412502+0000", 2023-11-05T08:33:35.145 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.6134e-05 2023-11-05T08:33:35.146 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.146 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:35.146 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.146 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.146 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.147 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.bs4 66/37 ECSubWrite(tid=111, reqid=client.4691.0:499, at_version=66'26, trim_to=0'0, roll_forward_to=65'21))", 2023-11-05T08:33:35.147 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:19.408909+0000", 2023-11-05T08:33:35.147 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 15.532247625, 2023-11-05T08:33:35.147 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0060916989999999999, 2023-11-05T08:33:35.147 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:35.148 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:35.148 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:35.148 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.148 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:35.148 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.408909+0000", 2023-11-05T08:33:35.149 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.149 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.149 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.149 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:35.149 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.408909+0000", 2023-11-05T08:33:35.150 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.150 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.150 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.150 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:35.150 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.408910+0000", 2023-11-05T08:33:35.151 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.083e-06 2023-11-05T08:33:35.151 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.151 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.151 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:35.151 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.409000+0000", 2023-11-05T08:33:35.152 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.0410999999999999e-05 2023-11-05T08:33:35.152 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.152 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.152 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:35.152 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.409028+0000", 2023-11-05T08:33:35.153 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8192e-05 2023-11-05T08:33:35.153 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.153 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.153 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:35.153 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.409048+0000", 2023-11-05T08:33:35.154 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0233000000000001e-05 2023-11-05T08:33:35.154 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.154 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.154 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:35.154 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.411432+0000", 2023-11-05T08:33:35.155 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0023832150000000002 2023-11-05T08:33:35.155 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.155 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.155 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:35.155 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.411457+0000", 2023-11-05T08:33:35.155 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5196000000000001e-05 2023-11-05T08:33:35.156 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.156 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.156 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:35.156 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.414903+0000", 2023-11-05T08:33:35.156 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0034460179999999999 2023-11-05T08:33:35.157 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.157 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.157 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:35.157 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.415000+0000", 2023-11-05T08:33:35.157 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.7350999999999997e-05 2023-11-05T08:33:35.158 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.158 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:35.158 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.158 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.158 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.159 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.10s1 76/45 ECSubWrite(tid=273, reqid=osd.13.0:273, at_version=76'36, trim_to=0'0, roll_forward_to=75'34))", 2023-11-05T08:33:35.159 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:29.502863+0000", 2023-11-05T08:33:35.159 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 5.4382934000000001, 2023-11-05T08:33:35.159 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0079313969999999998, 2023-11-05T08:33:35.159 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:35.160 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:35.160 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:35.160 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.160 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:35.160 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.502863+0000", 2023-11-05T08:33:35.161 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.161 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.161 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.161 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:35.161 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.502863+0000", 2023-11-05T08:33:35.161 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.162 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.162 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.162 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:35.162 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.502864+0000", 2023-11-05T08:33:35.162 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5099999999999999e-06 2023-11-05T08:33:35.163 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.163 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.163 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:35.163 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.502936+0000", 2023-11-05T08:33:35.163 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.1188999999999999e-05 2023-11-05T08:33:35.163 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.164 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.164 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:35.164 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.502968+0000", 2023-11-05T08:33:35.164 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2660999999999998e-05 2023-11-05T08:33:35.164 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.165 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.165 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:35.165 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.502992+0000", 2023-11-05T08:33:35.165 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3345000000000001e-05 2023-11-05T08:33:35.165 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.166 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.166 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:35.166 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.503162+0000", 2023-11-05T08:33:35.166 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00017057699999999999 2023-11-05T08:33:35.166 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.167 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.167 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:35.167 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.503200+0000", 2023-11-05T08:33:35.167 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.7800999999999999e-05 2023-11-05T08:33:35.167 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.168 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.168 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:35.168 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.510703+0000", 2023-11-05T08:33:35.168 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0075030330000000001 2023-11-05T08:33:35.168 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.168 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.169 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:35.169 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.510794+0000", 2023-11-05T08:33:35.169 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.1280999999999996e-05 2023-11-05T08:33:35.169 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.169 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:35.170 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.170 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.170 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.170 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.10s5 76/45 ECSubWrite(tid=273, reqid=osd.13.0:273, at_version=76'36, trim_to=0'0, roll_forward_to=75'34))", 2023-11-05T08:33:35.170 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:29.503048+0000", 2023-11-05T08:33:35.171 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 5.4381082540000003, 2023-11-05T08:33:35.171 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0078320760000000003, 2023-11-05T08:33:35.171 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:35.171 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:35.171 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:35.172 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.172 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:35.172 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.503048+0000", 2023-11-05T08:33:35.172 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.172 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.173 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.173 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:35.173 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.503048+0000", 2023-11-05T08:33:35.173 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.173 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.174 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.174 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:35.174 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.503049+0000", 2023-11-05T08:33:35.174 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.063e-06 2023-11-05T08:33:35.174 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.174 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.175 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:35.175 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.503142+0000", 2023-11-05T08:33:35.175 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.2664000000000006e-05 2023-11-05T08:33:35.175 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.175 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.176 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:35.176 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.503166+0000", 2023-11-05T08:33:35.176 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4819000000000001e-05 2023-11-05T08:33:35.176 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.176 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.177 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:35.177 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.503186+0000", 2023-11-05T08:33:35.177 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9018999999999999e-05 2023-11-05T08:33:35.177 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.177 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.178 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:35.178 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.503394+0000", 2023-11-05T08:33:35.178 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00020854399999999999 2023-11-05T08:33:35.178 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.178 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.179 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:35.179 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.503428+0000", 2023-11-05T08:33:35.179 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.4150999999999999e-05 2023-11-05T08:33:35.179 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.179 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.180 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:35.180 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.510803+0000", 2023-11-05T08:33:35.180 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0073745310000000001 2023-11-05T08:33:35.180 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.180 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.181 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:35.181 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.510880+0000", 2023-11-05T08:33:35.181 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.7285000000000003e-05 2023-11-05T08:33:35.181 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.181 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:35.181 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.182 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.182 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:35.182 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:33:35.184 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2023-11-05T08:33:35.185 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:33:35.180+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:33:35.284 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:33:35.280+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:33:35.359 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:33:35.359 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2023-11-05T08:33:35.359 INFO:teuthology.orchestra.run.smithi069.stdout: "num_ops": 0 2023-11-05T08:33:35.359 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:33:35.365 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 reset_purged_snaps_last 2023-11-05T08:33:35.373 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2023-11-05T08:33:35.423 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:33:35.421+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:33:35.486 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:35.483+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:35.495 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:33:35.496 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2023-11-05T08:33:35.496 INFO:teuthology.orchestra.run.smithi069.stdout: "complaint_time": 30, 2023-11-05T08:33:35.496 INFO:teuthology.orchestra.run.smithi069.stdout: "num_blocked_ops": 0 2023-11-05T08:33:35.496 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:33:35.510 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2023-11-05T08:33:35.527 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 reset_purged_snaps_last 2023-11-05T08:33:35.542 INFO:teuthology.orchestra.run.smithi002.stderr:noscrub is unset 2023-11-05T08:33:35.546 INFO:tasks.rados.rados.0.smithi090.stdout:262: done (2 left) 2023-11-05T08:33:35.547 INFO:tasks.rados.rados.0.smithi090.stdout:263: done (1 left) 2023-11-05T08:33:35.547 INFO:tasks.rados.rados.0.smithi090.stdout:264: read oid 12 snap 24 2023-11-05T08:33:35.547 INFO:tasks.rados.rados.0.smithi090.stdout:264: expect deleted 2023-11-05T08:33:35.547 INFO:tasks.rados.rados.0.smithi090.stdout:265: snap_remove snap 23 2023-11-05T08:33:35.587 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:33:35.584+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:33:35.678 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:33:35.678 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 20, 2023-11-05T08:33:35.678 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 600, 2023-11-05T08:33:35.679 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [ 2023-11-05T08:33:35.679 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.679 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.ds3 36/30 ECSubWrite(tid=20, reqid=client.4691.0:233, at_version=36'10, trim_to=0'0, roll_forward_to=34'8))", 2023-11-05T08:33:35.679 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.066903+0000", 2023-11-05T08:33:35.679 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 46.607550048999997, 2023-11-05T08:33:35.680 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.004733414, 2023-11-05T08:33:35.680 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:35.680 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:35.680 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:35.680 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.681 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:35.681 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.066903+0000", 2023-11-05T08:33:35.681 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.681 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.681 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.682 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:35.682 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.066903+0000", 2023-11-05T08:33:35.682 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.682 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.682 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.683 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:35.683 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.066903+0000", 2023-11-05T08:33:35.683 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.3999999999999997e-07 2023-11-05T08:33:35.683 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.683 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.684 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:35.684 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.066944+0000", 2023-11-05T08:33:35.684 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.0657000000000001e-05 2023-11-05T08:33:35.684 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.684 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.685 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:35.685 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.066968+0000", 2023-11-05T08:33:35.685 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3981999999999998e-05 2023-11-05T08:33:35.685 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.685 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.686 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:35.686 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.066985+0000", 2023-11-05T08:33:35.686 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6915999999999999e-05 2023-11-05T08:33:35.686 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.686 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.687 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:35.687 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.067184+0000", 2023-11-05T08:33:35.687 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000199507 2023-11-05T08:33:35.687 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.687 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.687 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:35.688 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.067213+0000", 2023-11-05T08:33:35.688 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8719000000000001e-05 2023-11-05T08:33:35.688 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.688 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.688 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:35.689 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.071537+0000", 2023-11-05T08:33:35.689 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0043238959999999998 2023-11-05T08:33:35.689 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.689 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.689 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:35.690 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.071636+0000", 2023-11-05T08:33:35.690 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.9396999999999997e-05 2023-11-05T08:33:35.690 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.690 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:35.690 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.691 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.691 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.691 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.ds3 36/30 ECSubWrite(tid=23, reqid=osd.7.0:23, at_version=36'14, trim_to=0'0, roll_forward_to=34'8))", 2023-11-05T08:33:35.691 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.077282+0000", 2023-11-05T08:33:35.691 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 46.597171277000001, 2023-11-05T08:33:35.691 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.005326842, 2023-11-05T08:33:35.692 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:35.692 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:35.692 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:35.692 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.692 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:35.693 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.077282+0000", 2023-11-05T08:33:35.693 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.693 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.693 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.693 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:35.693 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.077282+0000", 2023-11-05T08:33:35.694 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.694 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.694 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.694 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:35.694 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.077283+0000", 2023-11-05T08:33:35.695 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.1699999999999997e-07 2023-11-05T08:33:35.695 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.695 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.695 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:35.695 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.077374+0000", 2023-11-05T08:33:35.696 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.1181000000000007e-05 2023-11-05T08:33:35.696 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.696 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.696 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:35.696 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.077399+0000", 2023-11-05T08:33:35.696 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5452e-05 2023-11-05T08:33:35.697 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.697 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.697 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:35.697 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.077430+0000", 2023-11-05T08:33:35.698 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0768000000000001e-05 2023-11-05T08:33:35.698 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.698 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.698 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:35.698 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.077988+0000", 2023-11-05T08:33:35.698 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00055794899999999997 2023-11-05T08:33:35.699 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.699 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.699 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:35.699 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078010+0000", 2023-11-05T08:33:35.699 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2325000000000001e-05 2023-11-05T08:33:35.700 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.700 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.700 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:35.700 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.082479+0000", 2023-11-05T08:33:35.700 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0044689609999999996 2023-11-05T08:33:35.701 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.701 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.701 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:35.701 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.082608+0000", 2023-11-05T08:33:35.701 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00012928899999999999 2023-11-05T08:33:35.702 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.702 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:35.702 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.702 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.702 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.703 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.ds3 36/30 ECSubWrite(tid=24, reqid=osd.7.0:23, at_version=0'0, trim_to=0'0, roll_forward_to=36'14))", 2023-11-05T08:33:35.703 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.089743+0000", 2023-11-05T08:33:35.703 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 46.584709433, 2023-11-05T08:33:35.703 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0062502979999999996, 2023-11-05T08:33:35.703 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:35.704 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:35.704 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:35.704 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.704 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:35.704 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089743+0000", 2023-11-05T08:33:35.705 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.705 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.705 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.705 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:35.705 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089743+0000", 2023-11-05T08:33:35.705 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.706 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.706 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.706 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:35.706 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089745+0000", 2023-11-05T08:33:35.706 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.034e-06 2023-11-05T08:33:35.707 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.707 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.707 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:35.707 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089768+0000", 2023-11-05T08:33:35.707 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3195000000000001e-05 2023-11-05T08:33:35.708 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.708 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.708 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:35.708 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089791+0000", 2023-11-05T08:33:35.708 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2804999999999999e-05 2023-11-05T08:33:35.708 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.709 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.709 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:35.709 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089843+0000", 2023-11-05T08:33:35.709 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.27e-05 2023-11-05T08:33:35.709 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.710 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.710 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:35.710 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.090167+0000", 2023-11-05T08:33:35.710 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00032410700000000002 2023-11-05T08:33:35.710 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.711 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.711 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:35.711 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.090192+0000", 2023-11-05T08:33:35.711 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4819000000000001e-05 2023-11-05T08:33:35.711 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.712 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.712 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:35.712 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.095900+0000", 2023-11-05T08:33:35.712 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0057078540000000001 2023-11-05T08:33:35.712 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.712 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.713 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:35.713 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.095994+0000", 2023-11-05T08:33:35.713 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.3783999999999995e-05 2023-11-05T08:33:35.713 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.713 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:35.713 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.714 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.714 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.714 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.fs5 36/30 ECSubWrite(tid=30, reqid=osd.11.0:30, at_version=36'17, trim_to=0'0, roll_forward_to=35'15))", 2023-11-05T08:33:35.714 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.091636+0000", 2023-11-05T08:33:35.714 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 46.582816788999999, 2023-11-05T08:33:35.715 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0043942149999999999, 2023-11-05T08:33:35.715 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:35.715 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:35.715 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:35.715 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.715 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:35.716 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.091636+0000", 2023-11-05T08:33:35.716 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.716 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.716 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.716 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:35.717 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.091636+0000", 2023-11-05T08:33:35.717 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.717 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.717 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.717 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:35.718 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.091637+0000", 2023-11-05T08:33:35.718 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.37e-07 2023-11-05T08:33:35.718 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.718 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.718 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:35.719 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.091690+0000", 2023-11-05T08:33:35.719 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.3643000000000001e-05 2023-11-05T08:33:35.719 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.719 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.719 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:35.719 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.091713+0000", 2023-11-05T08:33:35.720 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2901999999999999e-05 2023-11-05T08:33:35.720 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.720 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.720 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:35.720 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.091734+0000", 2023-11-05T08:33:35.721 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0639e-05 2023-11-05T08:33:35.721 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.721 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.721 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:35.721 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.091916+0000", 2023-11-05T08:33:35.722 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00018221200000000001 2023-11-05T08:33:35.722 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.722 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.722 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:35.722 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.091940+0000", 2023-11-05T08:33:35.723 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4397999999999999e-05 2023-11-05T08:33:35.723 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.723 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.723 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:35.723 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.095937+0000", 2023-11-05T08:33:35.723 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0039963000000000004 2023-11-05T08:33:35.724 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.724 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.724 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:35.724 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.096030+0000", 2023-11-05T08:33:35.724 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.3584000000000004e-05 2023-11-05T08:33:35.725 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.725 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:35.725 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.725 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.725 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.726 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.11s4 41/40 ECSubWrite(tid=65, reqid=client.4691.0:287, at_version=41'13, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:35.726 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.173904+0000", 2023-11-05T08:33:35.726 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 41.500549219, 2023-11-05T08:33:35.726 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0062957869999999997, 2023-11-05T08:33:35.726 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:35.727 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:35.727 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:35.727 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.727 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:35.727 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.173904+0000", 2023-11-05T08:33:35.728 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.728 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.728 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.728 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:35.728 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.173904+0000", 2023-11-05T08:33:35.729 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.729 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.729 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.729 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:35.729 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.173905+0000", 2023-11-05T08:33:35.729 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.68e-06 2023-11-05T08:33:35.730 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.730 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.730 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:35.730 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.174701+0000", 2023-11-05T08:33:35.731 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00079553000000000002 2023-11-05T08:33:35.731 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.731 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.731 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:35.731 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.174739+0000", 2023-11-05T08:33:35.732 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.7605999999999999e-05 2023-11-05T08:33:35.732 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.732 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.732 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:35.732 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.174758+0000", 2023-11-05T08:33:35.733 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9403999999999999e-05 2023-11-05T08:33:35.733 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.733 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.733 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:35.733 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.174938+0000", 2023-11-05T08:33:35.733 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00018046800000000001 2023-11-05T08:33:35.734 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.734 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.734 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:35.734 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.174959+0000", 2023-11-05T08:33:35.734 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0471000000000001e-05 2023-11-05T08:33:35.735 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.735 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.735 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:35.735 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.180140+0000", 2023-11-05T08:33:35.735 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0051811080000000002 2023-11-05T08:33:35.735 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.736 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.736 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:35.736 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.180199+0000", 2023-11-05T08:33:35.736 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.9519999999999999e-05 2023-11-05T08:33:35.736 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.737 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:35.737 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.737 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.737 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.737 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpRead(3.9s6 41/40 ECSubRead(tid=69, to_read={3:96f693b2:::smithi09039252-13:head=0,266240,0}, subchunks={3:96f693b2:::smithi09039252-13:head=[0,1]}, attrs_to_read=))", 2023-11-05T08:33:35.738 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.184988+0000", 2023-11-05T08:33:35.738 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 41.489465234000001, 2023-11-05T08:33:35.738 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.004481428, 2023-11-05T08:33:35.738 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:35.738 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:35.738 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:35.739 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.739 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:35.739 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184988+0000", 2023-11-05T08:33:35.739 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.739 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.740 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.740 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:35.740 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184988+0000", 2023-11-05T08:33:35.740 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.740 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.740 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.741 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:35.741 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184989+0000", 2023-11-05T08:33:35.741 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.668e-06 2023-11-05T08:33:35.741 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.741 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.741 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:35.742 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.185009+0000", 2023-11-05T08:33:35.742 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9465000000000001e-05 2023-11-05T08:33:35.742 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.742 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.742 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:35.743 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.185053+0000", 2023-11-05T08:33:35.743 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.4529e-05 2023-11-05T08:33:35.743 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.743 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.743 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:35.743 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.185109+0000", 2023-11-05T08:33:35.744 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.524e-05 2023-11-05T08:33:35.744 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.744 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.744 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:35.744 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.185251+0000", 2023-11-05T08:33:35.745 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00014288699999999999 2023-11-05T08:33:35.745 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.745 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.745 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:35.745 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.189469+0000", 2023-11-05T08:33:35.745 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0042176389999999996 2023-11-05T08:33:35.746 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.746 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:35.746 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.746 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.746 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.746 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.ds3 52/45 ECSubWrite(tid=44, reqid=client.4691.0:337, at_version=0'0, trim_to=0'0, roll_forward_to=52'16))", 2023-11-05T08:33:35.747 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:05.250922+0000", 2023-11-05T08:33:35.747 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 30.423530783, 2023-11-05T08:33:35.747 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0048180039999999999, 2023-11-05T08:33:35.747 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:35.747 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:35.748 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:35.748 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.748 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:35.748 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.250922+0000", 2023-11-05T08:33:35.748 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.749 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.749 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.749 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:35.749 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.250922+0000", 2023-11-05T08:33:35.749 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.749 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.750 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.750 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:35.750 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.250924+0000", 2023-11-05T08:33:35.750 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.59e-06 2023-11-05T08:33:35.750 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.751 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.751 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:35.751 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.250950+0000", 2023-11-05T08:33:35.751 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6395999999999999e-05 2023-11-05T08:33:35.751 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.752 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.752 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:35.752 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.250988+0000", 2023-11-05T08:33:35.752 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.7858999999999998e-05 2023-11-05T08:33:35.752 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.752 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.753 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:35.753 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.251004+0000", 2023-11-05T08:33:35.753 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6192e-05 2023-11-05T08:33:35.753 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.753 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.754 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:35.754 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.251177+0000", 2023-11-05T08:33:35.754 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00017284000000000001 2023-11-05T08:33:35.754 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.754 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.755 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:35.755 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.251209+0000", 2023-11-05T08:33:35.755 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.1986000000000002e-05 2023-11-05T08:33:35.755 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.755 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.755 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:35.756 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255642+0000", 2023-11-05T08:33:35.756 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0044327890000000003 2023-11-05T08:33:35.756 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.756 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.756 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:35.757 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255740+0000", 2023-11-05T08:33:35.757 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.8351999999999996e-05 2023-11-05T08:33:35.757 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.757 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:35.757 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.758 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.758 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.758 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.9s6 52/45 ECSubWrite(tid=75, reqid=client.4691.0:341, at_version=52'24, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:35.758 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:05.258033+0000", 2023-11-05T08:33:35.758 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 30.416419555000001, 2023-11-05T08:33:35.759 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0062192000000000002, 2023-11-05T08:33:35.759 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:35.759 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:35.759 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:35.759 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.760 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:35.760 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.258033+0000", 2023-11-05T08:33:35.760 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.760 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.760 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.760 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:35.761 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.258033+0000", 2023-11-05T08:33:35.761 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.761 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.761 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.761 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:35.762 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.258034+0000", 2023-11-05T08:33:35.762 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.1299999999999999e-07 2023-11-05T08:33:35.762 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.762 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.762 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:35.763 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.258165+0000", 2023-11-05T08:33:35.763 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000131221 2023-11-05T08:33:35.763 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.763 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.763 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:35.764 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.258202+0000", 2023-11-05T08:33:35.764 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.6276000000000003e-05 2023-11-05T08:33:35.764 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.764 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.764 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:35.764 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.258225+0000", 2023-11-05T08:33:35.765 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3204999999999999e-05 2023-11-05T08:33:35.765 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.765 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.765 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:35.765 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.258480+0000", 2023-11-05T08:33:35.766 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00025507299999999998 2023-11-05T08:33:35.766 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.766 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.766 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:35.766 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.258512+0000", 2023-11-05T08:33:35.766 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2008999999999998e-05 2023-11-05T08:33:35.767 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.767 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.767 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:35.767 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.264164+0000", 2023-11-05T08:33:35.767 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0056518360000000004 2023-11-05T08:33:35.768 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.768 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.768 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:35.768 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.264253+0000", 2023-11-05T08:33:35.768 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.8766999999999999e-05 2023-11-05T08:33:35.769 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.769 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:35.769 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.769 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.770 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.770 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.9s6 52/45 ECSubWrite(tid=77, reqid=client.4691.0:344, at_version=52'26, trim_to=0'0, roll_forward_to=52'24))", 2023-11-05T08:33:35.770 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:05.280665+0000", 2023-11-05T08:33:35.770 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 30.393787474, 2023-11-05T08:33:35.770 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0052489859999999998, 2023-11-05T08:33:35.771 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:35.771 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:35.771 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:35.771 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.771 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:35.772 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.280665+0000", 2023-11-05T08:33:35.772 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.772 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.772 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.772 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:35.773 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.280665+0000", 2023-11-05T08:33:35.773 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.773 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.773 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.773 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:35.773 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.280668+0000", 2023-11-05T08:33:35.774 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0899999999999999e-06 2023-11-05T08:33:35.774 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.774 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.774 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:35.774 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.280940+0000", 2023-11-05T08:33:35.775 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000272386 2023-11-05T08:33:35.775 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.775 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.775 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:35.775 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.280988+0000", 2023-11-05T08:33:35.776 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.8177999999999998e-05 2023-11-05T08:33:35.776 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.776 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.776 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:35.776 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.281013+0000", 2023-11-05T08:33:35.776 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4915000000000001e-05 2023-11-05T08:33:35.777 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.777 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.777 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:35.777 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.281199+0000", 2023-11-05T08:33:35.777 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00018633300000000001 2023-11-05T08:33:35.778 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.778 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.778 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:35.778 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.281220+0000", 2023-11-05T08:33:35.778 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0268999999999999e-05 2023-11-05T08:33:35.779 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.779 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.779 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:35.779 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.285811+0000", 2023-11-05T08:33:35.779 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0045912629999999999 2023-11-05T08:33:35.779 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.780 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.780 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:35.780 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.285914+0000", 2023-11-05T08:33:35.780 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000103552 2023-11-05T08:33:35.780 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.781 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:35.781 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.781 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.781 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.781 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.1s4 62/40 ECSubWrite(tid=86, reqid=client.4691.0:416, at_version=62'13, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:35.782 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:15.360785+0000", 2023-11-05T08:33:35.782 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 20.313667641999999, 2023-11-05T08:33:35.782 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0050101479999999999, 2023-11-05T08:33:35.782 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:35.782 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:35.782 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:35.783 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.783 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:35.783 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:15.360785+0000", 2023-11-05T08:33:35.783 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.783 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.784 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.784 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:35.784 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:15.360785+0000", 2023-11-05T08:33:35.784 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.784 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.784 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.785 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:35.785 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:15.360786+0000", 2023-11-05T08:33:35.785 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.0299999999999997e-07 2023-11-05T08:33:35.785 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.785 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.786 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:35.786 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:15.361243+0000", 2023-11-05T08:33:35.786 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00045721200000000003 2023-11-05T08:33:35.786 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.786 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.786 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:35.787 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:15.361277+0000", 2023-11-05T08:33:35.787 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.3278999999999998e-05 2023-11-05T08:33:35.787 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.787 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.787 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:35.788 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:15.361293+0000", 2023-11-05T08:33:35.788 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6197000000000001e-05 2023-11-05T08:33:35.788 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.788 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.788 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:35.789 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:15.361461+0000", 2023-11-05T08:33:35.789 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00016777400000000001 2023-11-05T08:33:35.789 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.789 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.789 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:35.789 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:15.361491+0000", 2023-11-05T08:33:35.790 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0278999999999999e-05 2023-11-05T08:33:35.790 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.790 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.790 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:35.790 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:15.365730+0000", 2023-11-05T08:33:35.791 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0042391720000000003 2023-11-05T08:33:35.791 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.791 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.791 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:35.791 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:15.365795+0000", 2023-11-05T08:33:35.792 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.5332000000000004e-05 2023-11-05T08:33:35.792 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.792 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:35.792 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.792 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.792 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.793 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.1s4 62/40 ECSubWrite(tid=87, reqid=client.4691.0:417, at_version=62'14, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:35.793 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:15.368072+0000", 2023-11-05T08:33:35.793 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 20.30638098, 2023-11-05T08:33:35.793 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0043066170000000004, 2023-11-05T08:33:35.793 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:35.794 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:35.794 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:35.794 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.794 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:35.794 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:15.368072+0000", 2023-11-05T08:33:35.795 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.795 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.795 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.795 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:35.795 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:15.368072+0000", 2023-11-05T08:33:35.795 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.796 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.796 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.796 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:35.796 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:15.368074+0000", 2023-11-05T08:33:35.796 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.637e-06 2023-11-05T08:33:35.797 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.797 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.797 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:35.797 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:15.368576+0000", 2023-11-05T08:33:35.797 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000502444 2023-11-05T08:33:35.798 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.798 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.798 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:35.798 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:15.368607+0000", 2023-11-05T08:33:35.798 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0759000000000001e-05 2023-11-05T08:33:35.799 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.799 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.799 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:35.799 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:15.368622+0000", 2023-11-05T08:33:35.799 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.489e-05 2023-11-05T08:33:35.799 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.800 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.800 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:35.800 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:15.368773+0000", 2023-11-05T08:33:35.800 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00015161800000000001 2023-11-05T08:33:35.800 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.801 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.801 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:35.801 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:15.368807+0000", 2023-11-05T08:33:35.801 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.3853000000000003e-05 2023-11-05T08:33:35.801 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.802 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.802 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:35.802 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:15.372286+0000", 2023-11-05T08:33:35.802 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.003478489 2023-11-05T08:33:35.802 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.803 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.803 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:35.803 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:15.372379+0000", 2023-11-05T08:33:35.803 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.2927e-05 2023-11-05T08:33:35.803 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.804 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:35.804 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.804 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.804 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.804 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:481 3.11s0 3:88a306be:::smithi09039252-16:7 [read 0~262144,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2023-11-05T08:33:35.804 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:18.555634+0000", 2023-11-05T08:33:35.805 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 17.118818599000001, 2023-11-05T08:33:35.805 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0065275120000000001, 2023-11-05T08:33:35.805 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:35.805 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2023-11-05T08:33:35.805 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:33:35.806 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:33:35.806 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:35.806 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 481 2023-11-05T08:33:35.806 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.806 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:35.807 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.807 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:35.807 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555634+0000", 2023-11-05T08:33:35.807 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.807 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.808 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.808 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:35.808 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555634+0000", 2023-11-05T08:33:35.808 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.808 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.809 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.809 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:35.809 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555636+0000", 2023-11-05T08:33:35.809 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2860000000000001e-06 2023-11-05T08:33:35.809 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.809 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.810 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:35.810 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555642+0000", 2023-11-05T08:33:35.810 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.5810000000000002e-06 2023-11-05T08:33:35.810 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.810 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.811 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:35.811 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555658+0000", 2023-11-05T08:33:35.811 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5875999999999998e-05 2023-11-05T08:33:35.811 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.811 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.812 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:35.812 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555667+0000", 2023-11-05T08:33:35.812 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.9530000000000005e-06 2023-11-05T08:33:35.812 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.812 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.813 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:35.813 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555826+0000", 2023-11-05T08:33:35.813 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000158834 2023-11-05T08:33:35.813 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.813 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.813 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:33:35.814 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.556071+0000", 2023-11-05T08:33:35.814 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000245441 2023-11-05T08:33:35.814 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.814 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.814 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:35.815 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.562162+0000", 2023-11-05T08:33:35.815 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0060905409999999997 2023-11-05T08:33:35.815 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.815 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:35.815 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.816 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.816 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.816 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:482 3.11s0 3:88a306be:::smithi09039252-16:7 [read 0~262144,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2023-11-05T08:33:35.816 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:18.555691+0000", 2023-11-05T08:33:35.816 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 17.118761734, 2023-11-05T08:33:35.817 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.006823389, 2023-11-05T08:33:35.817 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:35.817 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2023-11-05T08:33:35.817 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:33:35.817 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:33:35.817 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:35.818 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 482 2023-11-05T08:33:35.818 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.818 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:35.818 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.818 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:35.819 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555691+0000", 2023-11-05T08:33:35.819 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.819 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.819 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.819 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:35.819 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555691+0000", 2023-11-05T08:33:35.820 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.820 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.820 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.820 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:35.820 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555692+0000", 2023-11-05T08:33:35.821 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.6199999999999998e-07 2023-11-05T08:33:35.821 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.821 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.821 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:35.821 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555695+0000", 2023-11-05T08:33:35.821 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.7220000000000001e-06 2023-11-05T08:33:35.822 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.822 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.822 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:35.822 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555703+0000", 2023-11-05T08:33:35.822 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.9689999999999994e-06 2023-11-05T08:33:35.823 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.823 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.823 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:35.823 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555709+0000", 2023-11-05T08:33:35.823 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.6420000000000004e-06 2023-11-05T08:33:35.824 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.824 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.824 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:35.824 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.556434+0000", 2023-11-05T08:33:35.824 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00072455700000000002 2023-11-05T08:33:35.824 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.825 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.825 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:33:35.825 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.556542+0000", 2023-11-05T08:33:35.825 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0001081 2023-11-05T08:33:35.825 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.826 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.826 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:35.826 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.562515+0000", 2023-11-05T08:33:35.826 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0059728369999999999 2023-11-05T08:33:35.826 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.826 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:35.827 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.827 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.827 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.827 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:483 3.11s0 3:88a306be:::smithi09039252-16:7 [read 0~262144,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2023-11-05T08:33:35.827 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:18.555880+0000", 2023-11-05T08:33:35.828 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 17.118572817, 2023-11-05T08:33:35.828 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0070107930000000004, 2023-11-05T08:33:35.828 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:35.828 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2023-11-05T08:33:35.828 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:33:35.828 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:33:35.829 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:35.829 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 483 2023-11-05T08:33:35.829 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.829 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:35.829 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.829 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:35.830 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555880+0000", 2023-11-05T08:33:35.830 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.830 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.830 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.830 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:35.831 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555880+0000", 2023-11-05T08:33:35.831 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.831 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.831 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.832 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:35.832 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555886+0000", 2023-11-05T08:33:35.832 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.1129999999999997e-06 2023-11-05T08:33:35.832 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.832 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.833 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:35.833 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555900+0000", 2023-11-05T08:33:35.833 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3769e-05 2023-11-05T08:33:35.833 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.833 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.833 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:35.834 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.556103+0000", 2023-11-05T08:33:35.834 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00020306400000000001 2023-11-05T08:33:35.834 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.834 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.834 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:35.835 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.556130+0000", 2023-11-05T08:33:35.835 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6466e-05 2023-11-05T08:33:35.835 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.835 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.835 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:35.836 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.558343+0000", 2023-11-05T08:33:35.836 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.002213207 2023-11-05T08:33:35.836 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.836 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.836 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:33:35.837 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.558458+0000", 2023-11-05T08:33:35.837 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000115773 2023-11-05T08:33:35.837 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.837 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.837 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:35.837 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.562891+0000", 2023-11-05T08:33:35.838 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0044324009999999999 2023-11-05T08:33:35.838 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.838 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:35.838 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.838 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.839 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.839 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpRead(3.11s0 65/45 ECSubRead(tid=8, to_read={3:88a306be:::smithi09039252-16:8=0,65536,0}, subchunks={3:88a306be:::smithi09039252-16:8=[0,1]}, attrs_to_read=))", 2023-11-05T08:33:35.839 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:18.558101+0000", 2023-11-05T08:33:35.839 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 17.11635193, 2023-11-05T08:33:35.839 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0053927030000000004, 2023-11-05T08:33:35.840 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:35.840 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:35.840 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:35.840 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.840 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:35.841 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.558101+0000", 2023-11-05T08:33:35.841 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.841 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.841 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.841 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:35.841 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.558101+0000", 2023-11-05T08:33:35.842 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.842 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.842 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.842 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:35.842 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.558101+0000", 2023-11-05T08:33:35.843 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.843 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.843 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.843 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:35.843 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.558101+0000", 2023-11-05T08:33:35.843 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.844 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.844 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.844 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:35.844 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.558161+0000", 2023-11-05T08:33:35.844 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.0037999999999997e-05 2023-11-05T08:33:35.845 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.845 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.845 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:35.845 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.558171+0000", 2023-11-05T08:33:35.845 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.9070000000000002e-06 2023-11-05T08:33:35.845 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.846 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.846 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:35.846 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.560070+0000", 2023-11-05T08:33:35.846 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001898601 2023-11-05T08:33:35.846 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.847 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.847 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:35.847 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.563494+0000", 2023-11-05T08:33:35.847 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0034241570000000002 2023-11-05T08:33:35.847 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.847 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:35.848 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.848 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.848 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.848 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.16s6 66/45 ECSubWrite(tid=112, reqid=osd.11.0:112, at_version=66'34, trim_to=0'0, roll_forward_to=65'32))", 2023-11-05T08:33:35.848 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:19.401233+0000", 2023-11-05T08:33:35.848 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 16.273219655999998, 2023-11-05T08:33:35.849 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.013005030000000001, 2023-11-05T08:33:35.849 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:35.849 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:35.849 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:35.849 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.850 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:35.850 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.401233+0000", 2023-11-05T08:33:35.850 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.850 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.850 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.850 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:35.851 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.401233+0000", 2023-11-05T08:33:35.851 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.851 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.851 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.851 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:35.851 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.401234+0000", 2023-11-05T08:33:35.852 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.1999999999999999e-07 2023-11-05T08:33:35.852 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.852 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.852 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:35.852 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.401290+0000", 2023-11-05T08:33:35.852 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.5544999999999998e-05 2023-11-05T08:33:35.853 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.853 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.853 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:35.853 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.401316+0000", 2023-11-05T08:33:35.853 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6889e-05 2023-11-05T08:33:35.854 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.854 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.854 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:35.854 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.401343+0000", 2023-11-05T08:33:35.854 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6239000000000001e-05 2023-11-05T08:33:35.855 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.855 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.855 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:35.855 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.401551+0000", 2023-11-05T08:33:35.857 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00020812399999999999 2023-11-05T08:33:35.857 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.857 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.857 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:35.858 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.401581+0000", 2023-11-05T08:33:35.858 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0508999999999999e-05 2023-11-05T08:33:35.858 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.858 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.858 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:35.859 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.414133+0000", 2023-11-05T08:33:35.859 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.012551639999999999 2023-11-05T08:33:35.859 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.859 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.859 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:35.860 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.414238+0000", 2023-11-05T08:33:35.860 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000105364 2023-11-05T08:33:35.860 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.860 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:35.860 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.861 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.861 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.861 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.ds3 76/45 ECSubWrite(tid=75, reqid=client.4691.0:572, at_version=76'21, trim_to=0'0, roll_forward_to=68'18))", 2023-11-05T08:33:35.861 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:29.500087+0000", 2023-11-05T08:33:35.861 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 6.1743659119999998, 2023-11-05T08:33:35.862 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0053297700000000002, 2023-11-05T08:33:35.862 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:35.862 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:35.862 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:35.862 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.863 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:35.863 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.500087+0000", 2023-11-05T08:33:35.863 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.863 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.863 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.863 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:35.864 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.500087+0000", 2023-11-05T08:33:35.864 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.864 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.864 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.864 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:35.865 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.500089+0000", 2023-11-05T08:33:35.865 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6029999999999999e-06 2023-11-05T08:33:35.865 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.865 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.865 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:35.866 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.500175+0000", 2023-11-05T08:33:35.866 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.6174999999999994e-05 2023-11-05T08:33:35.866 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.866 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.866 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:35.867 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.500205+0000", 2023-11-05T08:33:35.867 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0582e-05 2023-11-05T08:33:35.867 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.867 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.867 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:35.868 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.500227+0000", 2023-11-05T08:33:35.868 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1682e-05 2023-11-05T08:33:35.868 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.868 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.868 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:35.869 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.500524+0000", 2023-11-05T08:33:35.869 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000296769 2023-11-05T08:33:35.869 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.869 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.869 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:35.869 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.500548+0000", 2023-11-05T08:33:35.870 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4185e-05 2023-11-05T08:33:35.870 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.870 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.870 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:35.871 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.505328+0000", 2023-11-05T08:33:35.871 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0047798160000000001 2023-11-05T08:33:35.871 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.871 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.871 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:35.872 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.505417+0000", 2023-11-05T08:33:35.872 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.8957999999999998e-05 2023-11-05T08:33:35.872 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.872 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:35.872 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.873 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.873 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.873 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.ds3 76/45 ECSubWrite(tid=76, reqid=client.4691.0:572, at_version=0'0, trim_to=0'0, roll_forward_to=76'21))", 2023-11-05T08:33:35.873 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:29.507204+0000", 2023-11-05T08:33:35.873 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 6.1672484169999997, 2023-11-05T08:33:35.873 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0046404339999999997, 2023-11-05T08:33:35.874 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:35.874 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:35.874 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:35.874 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.874 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:35.875 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.507204+0000", 2023-11-05T08:33:35.875 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.875 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.875 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.875 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:35.875 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.507204+0000", 2023-11-05T08:33:35.876 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.876 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.876 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.876 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:35.876 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.507206+0000", 2023-11-05T08:33:35.877 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9139999999999998e-06 2023-11-05T08:33:35.877 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.877 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.877 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:35.877 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.507239+0000", 2023-11-05T08:33:35.878 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2503999999999999e-05 2023-11-05T08:33:35.878 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.878 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.878 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:35.878 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.507277+0000", 2023-11-05T08:33:35.879 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.8191000000000001e-05 2023-11-05T08:33:35.879 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.879 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.879 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:35.879 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.507301+0000", 2023-11-05T08:33:35.879 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3578999999999999e-05 2023-11-05T08:33:35.880 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.880 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.880 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:35.880 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.507526+0000", 2023-11-05T08:33:35.880 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00022541299999999999 2023-11-05T08:33:35.881 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.881 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.881 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:35.881 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.507553+0000", 2023-11-05T08:33:35.881 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6497999999999999e-05 2023-11-05T08:33:35.881 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.882 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.882 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:35.882 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.511752+0000", 2023-11-05T08:33:35.882 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0041991980000000003 2023-11-05T08:33:35.882 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.883 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.883 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:35.883 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:29.511845+0000", 2023-11-05T08:33:35.883 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.3137e-05 2023-11-05T08:33:35.883 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.884 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:35.884 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.884 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.884 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.884 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.fs5 77/37 ECSubWrite(tid=130, reqid=client.4691.0:579, at_version=77'44, trim_to=0'0, roll_forward_to=70'42))", 2023-11-05T08:33:35.885 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:30.539706+0000", 2023-11-05T08:33:35.885 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 5.1347468860000003, 2023-11-05T08:33:35.885 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0061240360000000002, 2023-11-05T08:33:35.885 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:35.885 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:35.885 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:35.886 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.886 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:35.886 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.539706+0000", 2023-11-05T08:33:35.886 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.886 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.887 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.887 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:35.887 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.539706+0000", 2023-11-05T08:33:35.887 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.887 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.888 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.888 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:35.888 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.539706+0000", 2023-11-05T08:33:35.888 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.77e-07 2023-11-05T08:33:35.888 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.889 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.889 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:35.889 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.540581+0000", 2023-11-05T08:33:35.889 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00087454000000000004 2023-11-05T08:33:35.889 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.889 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.890 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:35.890 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.540619+0000", 2023-11-05T08:33:35.890 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.7561000000000002e-05 2023-11-05T08:33:35.890 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.890 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.891 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:35.891 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.540641+0000", 2023-11-05T08:33:35.891 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2881999999999999e-05 2023-11-05T08:33:35.891 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.891 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.892 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:35.892 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.540815+0000", 2023-11-05T08:33:35.892 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000173377 2023-11-05T08:33:35.892 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.892 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.893 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:35.893 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.540842+0000", 2023-11-05T08:33:35.893 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6755999999999999e-05 2023-11-05T08:33:35.893 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.893 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.893 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:35.894 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.545733+0000", 2023-11-05T08:33:35.894 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0048915590000000002 2023-11-05T08:33:35.894 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.894 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.894 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:35.895 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.545830+0000", 2023-11-05T08:33:35.895 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.6984000000000005e-05 2023-11-05T08:33:35.895 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.895 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:35.895 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.895 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.896 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.896 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.fs5 77/37 ECSubWrite(tid=133, reqid=client.4691.0:582, at_version=77'47, trim_to=0'0, roll_forward_to=77'45))", 2023-11-05T08:33:35.896 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:30.555660+0000", 2023-11-05T08:33:35.896 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 5.1187928879999998, 2023-11-05T08:33:35.896 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0042342370000000001, 2023-11-05T08:33:35.897 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:35.897 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:35.897 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:35.897 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.897 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:35.898 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.555660+0000", 2023-11-05T08:33:35.898 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.898 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.898 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.898 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:35.899 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.555660+0000", 2023-11-05T08:33:35.899 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:35.899 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.899 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.899 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:35.899 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.555662+0000", 2023-11-05T08:33:35.900 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.86e-06 2023-11-05T08:33:35.900 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.900 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.900 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:35.900 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.555771+0000", 2023-11-05T08:33:35.900 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000109304 2023-11-05T08:33:35.901 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.901 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.901 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:35.901 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.555799+0000", 2023-11-05T08:33:35.901 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7421999999999999e-05 2023-11-05T08:33:35.901 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.902 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.902 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:35.902 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.555822+0000", 2023-11-05T08:33:35.902 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2912e-05 2023-11-05T08:33:35.902 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.903 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.903 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:35.903 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.555952+0000", 2023-11-05T08:33:35.903 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00013035299999999999 2023-11-05T08:33:35.903 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.903 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.904 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:35.904 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.555968+0000", 2023-11-05T08:33:35.904 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6011999999999998e-05 2023-11-05T08:33:35.904 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.904 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.905 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:35.905 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.559826+0000", 2023-11-05T08:33:35.905 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0038579249999999999 2023-11-05T08:33:35.905 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:35.905 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:35.906 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:35.906 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.559894+0000", 2023-11-05T08:33:35.906 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.8448999999999995e-05 2023-11-05T08:33:35.906 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.906 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:35.907 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.907 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:35.907 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:35.907 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:33:35.908 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-05T08:33:35.909 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:33:35.904+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:33:35.910 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2023-11-05T08:33:36.010 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:33:36.008+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:33:36.040 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:33:36.041 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2023-11-05T08:33:36.041 INFO:teuthology.orchestra.run.smithi069.stdout: "num_ops": 0 2023-11-05T08:33:36.041 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:33:36.055 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2023-11-05T08:33:36.098 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-11-05T08:33:36.110 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:33:36.105+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:33:36.120 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1: {} 2023-11-05T08:33:36.120 INFO:teuthology.orchestra.run.smithi002.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-11-05T08:33:36.137 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2: {} 2023-11-05T08:33:36.138 INFO:teuthology.orchestra.run.smithi002.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-11-05T08:33:36.138 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-11-05T08:33:36.154 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4: {} 2023-11-05T08:33:36.154 INFO:teuthology.orchestra.run.smithi002.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-11-05T08:33:36.174 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5: {} 2023-11-05T08:33:36.174 INFO:teuthology.orchestra.run.smithi002.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-11-05T08:33:36.177 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:33:36.177 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2023-11-05T08:33:36.177 INFO:teuthology.orchestra.run.smithi069.stdout: "complaint_time": 30, 2023-11-05T08:33:36.177 INFO:teuthology.orchestra.run.smithi069.stdout: "num_blocked_ops": 0 2023-11-05T08:33:36.177 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:33:36.191 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2023-11-05T08:33:36.192 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6: {} 2023-11-05T08:33:36.192 INFO:teuthology.orchestra.run.smithi002.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-11-05T08:33:36.210 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:33:36.207+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:33:36.213 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7: {} 2023-11-05T08:33:36.214 INFO:teuthology.orchestra.run.smithi002.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-11-05T08:33:36.234 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8: {} 2023-11-05T08:33:36.234 INFO:teuthology.orchestra.run.smithi002.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-11-05T08:33:36.256 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9: {} 2023-11-05T08:33:36.257 INFO:teuthology.orchestra.run.smithi002.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-11-05T08:33:36.276 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10: {} 2023-11-05T08:33:36.277 INFO:teuthology.orchestra.run.smithi002.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-11-05T08:33:36.299 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11: {} 2023-11-05T08:33:36.299 INFO:teuthology.orchestra.run.smithi002.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-11-05T08:33:36.311 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:36.307+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:36.322 INFO:teuthology.orchestra.run.smithi002.stdout:osd.12: {} 2023-11-05T08:33:36.322 INFO:teuthology.orchestra.run.smithi002.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-11-05T08:33:36.345 INFO:teuthology.orchestra.run.smithi002.stdout:osd.13: {} 2023-11-05T08:33:36.345 INFO:teuthology.orchestra.run.smithi002.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-11-05T08:33:36.356 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:33:36.356 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 20, 2023-11-05T08:33:36.356 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 600, 2023-11-05T08:33:36.357 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [ 2023-11-05T08:33:36.357 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.357 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.ds2 36/30 ECSubWrite(tid=23, reqid=osd.7.0:23, at_version=36'14, trim_to=0'0, roll_forward_to=34'8))", 2023-11-05T08:33:36.357 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.078426+0000", 2023-11-05T08:33:36.358 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 47.274841117000001, 2023-11-05T08:33:36.358 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0071789940000000002, 2023-11-05T08:33:36.358 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:36.358 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:36.358 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:36.359 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.359 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:36.359 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078426+0000", 2023-11-05T08:33:36.359 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.359 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.360 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.360 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:36.360 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078426+0000", 2023-11-05T08:33:36.360 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.360 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.361 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.361 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:36.361 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078427+0000", 2023-11-05T08:33:36.361 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.093e-06 2023-11-05T08:33:36.361 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.362 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.362 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:36.362 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078484+0000", 2023-11-05T08:33:36.362 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.7240000000000001e-05 2023-11-05T08:33:36.362 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.363 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.363 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:36.363 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078508+0000", 2023-11-05T08:33:36.363 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4032e-05 2023-11-05T08:33:36.363 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.364 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.364 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:36.364 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078527+0000", 2023-11-05T08:33:36.364 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8754999999999999e-05 2023-11-05T08:33:36.364 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.364 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.365 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:36.365 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079037+0000", 2023-11-05T08:33:36.365 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000509722 2023-11-05T08:33:36.365 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.365 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.366 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:36.366 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079059+0000", 2023-11-05T08:33:36.366 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2467999999999999e-05 2023-11-05T08:33:36.366 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.366 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.367 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:36.367 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.085512+0000", 2023-11-05T08:33:36.367 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0064529059999999996 2023-11-05T08:33:36.367 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.367 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.368 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:36.368 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.085605+0000", 2023-11-05T08:33:36.368 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.2777999999999998e-05 2023-11-05T08:33:36.368 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.368 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:36.369 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.369 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.369 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.369 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.4s5 36/30 ECSubWrite(tid=10, reqid=osd.8.0:10, at_version=36'8, trim_to=0'0, roll_forward_to=34'6))", 2023-11-05T08:33:36.369 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.078716+0000", 2023-11-05T08:33:36.370 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 47.274551332000001, 2023-11-05T08:33:36.370 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.006931782, 2023-11-05T08:33:36.370 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:36.370 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:36.370 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:36.371 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.371 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:36.371 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078716+0000", 2023-11-05T08:33:36.371 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.371 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.372 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.372 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:36.372 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078716+0000", 2023-11-05T08:33:36.372 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.372 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.373 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.373 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:36.373 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078716+0000", 2023-11-05T08:33:36.373 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.0399999999999995e-07 2023-11-05T08:33:36.373 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.373 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.374 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:36.374 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078773+0000", 2023-11-05T08:33:36.374 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.6149000000000001e-05 2023-11-05T08:33:36.374 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.374 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.375 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:36.375 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078804+0000", 2023-11-05T08:33:36.375 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.1677999999999997e-05 2023-11-05T08:33:36.375 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.375 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.376 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:36.376 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078827+0000", 2023-11-05T08:33:36.376 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2469000000000001e-05 2023-11-05T08:33:36.376 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.376 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.377 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:36.377 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079007+0000", 2023-11-05T08:33:36.377 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000180242 2023-11-05T08:33:36.377 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.377 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.378 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:36.378 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079032+0000", 2023-11-05T08:33:36.378 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4915000000000001e-05 2023-11-05T08:33:36.378 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.378 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.379 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:36.379 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.085566+0000", 2023-11-05T08:33:36.379 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0065336140000000001 2023-11-05T08:33:36.379 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.379 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.379 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:36.380 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.085648+0000", 2023-11-05T08:33:36.380 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.2010999999999998e-05 2023-11-05T08:33:36.380 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.380 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:36.380 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.381 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.381 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.381 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.8s4 36/30 ECSubWrite(tid=23, reqid=osd.5.0:23, at_version=36'15, trim_to=0'0, roll_forward_to=34'13))", 2023-11-05T08:33:36.381 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.079985+0000", 2023-11-05T08:33:36.381 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 47.273282317000003, 2023-11-05T08:33:36.381 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.007626023, 2023-11-05T08:33:36.382 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:36.382 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:36.382 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:36.382 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.382 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:36.383 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079985+0000", 2023-11-05T08:33:36.383 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.383 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.383 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.383 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:36.384 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079985+0000", 2023-11-05T08:33:36.384 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.384 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.384 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.384 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:36.385 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079985+0000", 2023-11-05T08:33:36.385 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.6300000000000005e-07 2023-11-05T08:33:36.385 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.385 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.385 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:36.385 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.080040+0000", 2023-11-05T08:33:36.386 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.4796999999999997e-05 2023-11-05T08:33:36.386 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.386 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.386 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:36.386 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.080073+0000", 2023-11-05T08:33:36.387 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.3034999999999998e-05 2023-11-05T08:33:36.387 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.387 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.387 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:36.387 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.080095+0000", 2023-11-05T08:33:36.388 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1841999999999999e-05 2023-11-05T08:33:36.388 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.388 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.388 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:36.388 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.080255+0000", 2023-11-05T08:33:36.389 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00016012700000000001 2023-11-05T08:33:36.389 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.389 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.389 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:36.389 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.080280+0000", 2023-11-05T08:33:36.389 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4953999999999999e-05 2023-11-05T08:33:36.390 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.390 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.390 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:36.390 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.087517+0000", 2023-11-05T08:33:36.390 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0072370799999999999 2023-11-05T08:33:36.391 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.391 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.391 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:36.391 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.087611+0000", 2023-11-05T08:33:36.391 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.3524999999999996e-05 2023-11-05T08:33:36.392 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.392 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:36.392 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.392 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.392 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.393 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.ds2 36/30 ECSubWrite(tid=24, reqid=osd.7.0:23, at_version=0'0, trim_to=0'0, roll_forward_to=36'14))", 2023-11-05T08:33:36.393 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.089710+0000", 2023-11-05T08:33:36.393 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 47.263556987000001, 2023-11-05T08:33:36.393 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0067357069999999996, 2023-11-05T08:33:36.393 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:36.394 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:36.394 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:36.394 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.394 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:36.394 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089710+0000", 2023-11-05T08:33:36.395 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.395 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.395 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.395 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:36.395 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089710+0000", 2023-11-05T08:33:36.395 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.396 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.396 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.396 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:36.396 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089711+0000", 2023-11-05T08:33:36.396 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1429999999999999e-06 2023-11-05T08:33:36.397 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.397 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.397 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:36.397 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089733+0000", 2023-11-05T08:33:36.397 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1559000000000001e-05 2023-11-05T08:33:36.397 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.398 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.398 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:36.398 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089757+0000", 2023-11-05T08:33:36.398 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4515000000000002e-05 2023-11-05T08:33:36.398 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.399 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.399 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:36.399 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089780+0000", 2023-11-05T08:33:36.399 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2789000000000002e-05 2023-11-05T08:33:36.399 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.400 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.400 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:36.400 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.090049+0000", 2023-11-05T08:33:36.400 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000269363 2023-11-05T08:33:36.400 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.400 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.401 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:36.401 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.090074+0000", 2023-11-05T08:33:36.401 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4071999999999999e-05 2023-11-05T08:33:36.401 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.401 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.402 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:36.402 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.096355+0000", 2023-11-05T08:33:36.402 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0062814489999999997 2023-11-05T08:33:36.402 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.402 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.403 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:36.403 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.096446+0000", 2023-11-05T08:33:36.403 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.0816999999999995e-05 2023-11-05T08:33:36.403 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.403 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:36.404 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.404 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.404 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.404 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.8s4 36/30 ECSubWrite(tid=24, reqid=osd.5.0:23, at_version=0'0, trim_to=0'0, roll_forward_to=36'15))", 2023-11-05T08:33:36.404 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.089889+0000", 2023-11-05T08:33:36.404 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 47.263378578000001, 2023-11-05T08:33:36.405 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0065355869999999998, 2023-11-05T08:33:36.405 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:36.405 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:36.405 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:36.405 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.405 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:36.406 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089889+0000", 2023-11-05T08:33:36.406 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.406 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.406 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.406 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:36.407 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089889+0000", 2023-11-05T08:33:36.407 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.407 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.407 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.407 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:36.408 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089890+0000", 2023-11-05T08:33:36.408 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1599999999999999e-06 2023-11-05T08:33:36.408 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.408 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.408 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:36.409 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089908+0000", 2023-11-05T08:33:36.409 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8448999999999999e-05 2023-11-05T08:33:36.409 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.409 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.409 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:36.410 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089929+0000", 2023-11-05T08:33:36.410 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0534999999999999e-05 2023-11-05T08:33:36.410 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.410 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.410 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:36.411 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089943+0000", 2023-11-05T08:33:36.411 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4469e-05 2023-11-05T08:33:36.411 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.411 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.411 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:36.411 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.090098+0000", 2023-11-05T08:33:36.412 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00015513699999999999 2023-11-05T08:33:36.412 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.412 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.412 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:36.412 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.090114+0000", 2023-11-05T08:33:36.413 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5962999999999999e-05 2023-11-05T08:33:36.413 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.413 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.413 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:36.413 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.096334+0000", 2023-11-05T08:33:36.414 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0062193320000000002 2023-11-05T08:33:36.414 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.414 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.414 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:36.414 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.096424+0000", 2023-11-05T08:33:36.415 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.0542000000000001e-05 2023-11-05T08:33:36.415 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.415 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:36.415 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.415 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.415 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.416 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.as4 41/33 ECSubWrite(tid=35, reqid=osd.7.0:35, at_version=41'12, trim_to=0'0, roll_forward_to=40'10))", 2023-11-05T08:33:36.416 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.128988+0000", 2023-11-05T08:33:36.416 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 42.224278976000001, 2023-11-05T08:33:36.416 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0097032620000000007, 2023-11-05T08:33:36.416 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:36.417 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:36.417 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:36.417 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.417 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:36.417 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.128988+0000", 2023-11-05T08:33:36.418 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.418 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.418 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.418 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:36.418 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.128988+0000", 2023-11-05T08:33:36.419 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.419 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.419 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.419 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:36.419 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.128989+0000", 2023-11-05T08:33:36.420 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.3600000000000003e-07 2023-11-05T08:33:36.420 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.420 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.420 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:36.420 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.129043+0000", 2023-11-05T08:33:36.421 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.4163999999999998e-05 2023-11-05T08:33:36.421 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.421 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.421 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:36.421 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.129080+0000", 2023-11-05T08:33:36.422 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.7144999999999998e-05 2023-11-05T08:33:36.422 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.422 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.422 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:36.422 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.129104+0000", 2023-11-05T08:33:36.423 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3538999999999999e-05 2023-11-05T08:33:36.423 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.423 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.423 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:36.423 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.129248+0000", 2023-11-05T08:33:36.424 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00014404500000000001 2023-11-05T08:33:36.424 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.424 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.424 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:36.424 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.129273+0000", 2023-11-05T08:33:36.425 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5814999999999999e-05 2023-11-05T08:33:36.425 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.425 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.425 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:36.425 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.138576+0000", 2023-11-05T08:33:36.426 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0093020840000000004 2023-11-05T08:33:36.426 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.426 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.426 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:36.426 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.138691+0000", 2023-11-05T08:33:36.427 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000115834 2023-11-05T08:33:36.427 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.427 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:36.427 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.427 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.427 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.428 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.as4 41/33 ECSubWrite(tid=36, reqid=osd.7.0:36, at_version=41'14, trim_to=0'0, roll_forward_to=40'10))", 2023-11-05T08:33:36.428 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.131320+0000", 2023-11-05T08:33:36.428 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 42.221947094999997, 2023-11-05T08:33:36.428 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0086819870000000004, 2023-11-05T08:33:36.428 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:36.429 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:36.429 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:36.429 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.429 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:36.429 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131320+0000", 2023-11-05T08:33:36.430 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.430 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.430 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.430 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:36.430 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131320+0000", 2023-11-05T08:33:36.431 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.431 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.431 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.431 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:36.431 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131322+0000", 2023-11-05T08:33:36.432 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2500000000000001e-06 2023-11-05T08:33:36.432 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.432 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.432 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:36.432 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131375+0000", 2023-11-05T08:33:36.433 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.3041000000000003e-05 2023-11-05T08:33:36.433 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.433 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.433 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:36.433 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131422+0000", 2023-11-05T08:33:36.434 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.6526999999999997e-05 2023-11-05T08:33:36.434 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.434 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.434 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:36.434 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131445+0000", 2023-11-05T08:33:36.435 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3672e-05 2023-11-05T08:33:36.435 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.435 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.435 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:36.435 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131641+0000", 2023-11-05T08:33:36.436 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00019531299999999999 2023-11-05T08:33:36.436 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.436 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.436 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:36.436 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131666+0000", 2023-11-05T08:33:36.437 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5122000000000001e-05 2023-11-05T08:33:36.437 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.437 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.437 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:36.437 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.139945+0000", 2023-11-05T08:33:36.438 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0082790889999999999 2023-11-05T08:33:36.438 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.438 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.438 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:36.438 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.140002+0000", 2023-11-05T08:33:36.439 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.6972999999999998e-05 2023-11-05T08:33:36.439 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.439 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:36.439 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.439 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.439 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.440 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpReadReply(3.19s0 41/40 ECSubReadReply(tid=67, attrs_read=0))", 2023-11-05T08:33:36.440 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.176501+0000", 2023-11-05T08:33:36.440 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 42.176766577999999, 2023-11-05T08:33:36.440 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.006463318, 2023-11-05T08:33:36.440 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:36.441 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:36.441 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:36.441 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.441 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:36.441 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.176501+0000", 2023-11-05T08:33:36.442 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.442 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.442 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.442 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:36.442 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.176501+0000", 2023-11-05T08:33:36.443 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.443 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.443 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.443 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:36.443 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.176501+0000", 2023-11-05T08:33:36.444 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.5199999999999995e-07 2023-11-05T08:33:36.444 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.444 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.444 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:36.444 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.176802+0000", 2023-11-05T08:33:36.445 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00030095700000000003 2023-11-05T08:33:36.445 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.445 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.445 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:36.445 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.176822+0000", 2023-11-05T08:33:36.446 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9179000000000001e-05 2023-11-05T08:33:36.446 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.446 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.446 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:36.446 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.176834+0000", 2023-11-05T08:33:36.447 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2152e-05 2023-11-05T08:33:36.447 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.447 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.447 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:36.447 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.181163+0000", 2023-11-05T08:33:36.448 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0043295790000000001 2023-11-05T08:33:36.448 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.448 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.448 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:36.448 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.182964+0000", 2023-11-05T08:33:36.449 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0018004989999999999 2023-11-05T08:33:36.449 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.449 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:36.449 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.449 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.449 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.450 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:293 3.9s0 3:96f693b2:::smithi09039252-13:head [read 0~1064960,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2023-11-05T08:33:36.450 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.184012+0000", 2023-11-05T08:33:36.450 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 42.169255253999999, 2023-11-05T08:33:36.450 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0082043189999999998, 2023-11-05T08:33:36.450 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:36.451 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2023-11-05T08:33:36.451 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:33:36.451 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:33:36.451 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:36.451 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 293 2023-11-05T08:33:36.452 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.452 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:36.452 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.452 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:36.452 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184012+0000", 2023-11-05T08:33:36.453 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.453 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.453 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.453 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:36.453 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184012+0000", 2023-11-05T08:33:36.454 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.454 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.454 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.454 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:36.454 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184014+0000", 2023-11-05T08:33:36.455 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4930000000000001e-06 2023-11-05T08:33:36.455 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.455 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.455 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:36.455 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184024+0000", 2023-11-05T08:33:36.455 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0114e-05 2023-11-05T08:33:36.456 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.456 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.456 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:36.456 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184047+0000", 2023-11-05T08:33:36.456 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2463000000000002e-05 2023-11-05T08:33:36.457 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.457 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.457 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:36.457 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184063+0000", 2023-11-05T08:33:36.457 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6212e-05 2023-11-05T08:33:36.458 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.458 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.458 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:36.458 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184172+0000", 2023-11-05T08:33:36.458 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000108501 2023-11-05T08:33:36.459 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.459 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.459 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:33:36.459 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184445+0000", 2023-11-05T08:33:36.459 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00027307699999999999 2023-11-05T08:33:36.460 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.460 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.460 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:36.460 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.192216+0000", 2023-11-05T08:33:36.460 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0077714589999999997 2023-11-05T08:33:36.461 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.461 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:36.461 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.461 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.461 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.462 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:294 3.9s0 3:96f693b2:::smithi09039252-13:head [read 0~1064960,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2023-11-05T08:33:36.462 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.184100+0000", 2023-11-05T08:33:36.462 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 42.169166939, 2023-11-05T08:33:36.462 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0099915090000000008, 2023-11-05T08:33:36.462 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:36.463 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2023-11-05T08:33:36.463 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:33:36.463 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:33:36.463 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:36.463 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 294 2023-11-05T08:33:36.464 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.464 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:36.464 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.464 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:36.464 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184100+0000", 2023-11-05T08:33:36.464 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.465 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.465 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.465 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:36.465 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184100+0000", 2023-11-05T08:33:36.465 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.466 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.466 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.466 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:36.466 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184101+0000", 2023-11-05T08:33:36.466 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2100000000000001e-06 2023-11-05T08:33:36.467 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.469 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.470 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:36.470 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184109+0000", 2023-11-05T08:33:36.470 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.5630000000000003e-06 2023-11-05T08:33:36.470 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.470 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.471 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:36.471 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184126+0000", 2023-11-05T08:33:36.471 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6674e-05 2023-11-05T08:33:36.471 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.471 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.472 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:36.472 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184138+0000", 2023-11-05T08:33:36.472 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2306e-05 2023-11-05T08:33:36.472 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.472 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.473 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:36.473 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.189352+0000", 2023-11-05T08:33:36.473 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0052140629999999997 2023-11-05T08:33:36.473 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.473 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.474 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:33:36.474 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.189596+0000", 2023-11-05T08:33:36.474 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00024365899999999999 2023-11-05T08:33:36.474 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.474 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.475 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:36.475 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.194092+0000", 2023-11-05T08:33:36.475 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0044960340000000003 2023-11-05T08:33:36.475 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.475 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:36.475 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.476 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.476 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.476 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:295 3.9s0 3:96f693b2:::smithi09039252-13:head [read 0~1064960,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2023-11-05T08:33:36.476 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.184155+0000", 2023-11-05T08:33:36.476 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 42.169112134000002, 2023-11-05T08:33:36.477 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.011309207999999999, 2023-11-05T08:33:36.477 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:36.477 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2023-11-05T08:33:36.477 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:33:36.477 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:33:36.478 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:36.478 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 295 2023-11-05T08:33:36.478 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.478 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:36.478 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.479 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:36.479 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184155+0000", 2023-11-05T08:33:36.479 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.479 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.479 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.480 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:36.480 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184155+0000", 2023-11-05T08:33:36.480 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.480 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.480 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.480 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:36.481 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184156+0000", 2023-11-05T08:33:36.481 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0139999999999999e-06 2023-11-05T08:33:36.481 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.481 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.481 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:36.481 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184162+0000", 2023-11-05T08:33:36.482 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.5330000000000002e-06 2023-11-05T08:33:36.482 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.482 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.482 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:36.482 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184176+0000", 2023-11-05T08:33:36.483 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.486e-05 2023-11-05T08:33:36.483 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.483 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.483 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:36.483 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184188+0000", 2023-11-05T08:33:36.484 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1246e-05 2023-11-05T08:33:36.484 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.484 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.484 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:36.484 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.190012+0000", 2023-11-05T08:33:36.484 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0058240920000000003 2023-11-05T08:33:36.485 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.485 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.485 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:33:36.485 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.190143+0000", 2023-11-05T08:33:36.485 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000130984 2023-11-05T08:33:36.486 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.486 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.486 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:36.486 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.195464+0000", 2023-11-05T08:33:36.486 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0053214789999999996 2023-11-05T08:33:36.487 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.487 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:36.487 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.487 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.487 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.487 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.14s5 41/40 ECSubWrite(tid=17, reqid=client.4691.0:278, at_version=41'14, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:36.488 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.185798+0000", 2023-11-05T08:33:36.488 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 42.167468857000003, 2023-11-05T08:33:36.488 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0063439719999999998, 2023-11-05T08:33:36.488 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:36.488 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:36.489 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:36.489 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.489 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:36.489 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.185798+0000", 2023-11-05T08:33:36.489 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.489 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.490 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.490 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:36.490 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.185798+0000", 2023-11-05T08:33:36.490 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.490 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.491 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.491 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:36.491 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.185800+0000", 2023-11-05T08:33:36.491 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.689e-06 2023-11-05T08:33:36.491 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.491 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.492 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:36.492 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.186569+0000", 2023-11-05T08:33:36.492 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00076927199999999999 2023-11-05T08:33:36.492 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.492 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.493 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:36.493 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.186598+0000", 2023-11-05T08:33:36.493 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8776999999999999e-05 2023-11-05T08:33:36.493 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.493 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.493 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:36.494 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.186615+0000", 2023-11-05T08:33:36.494 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6759e-05 2023-11-05T08:33:36.494 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.494 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.494 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:36.495 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.186763+0000", 2023-11-05T08:33:36.495 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000148646 2023-11-05T08:33:36.495 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.495 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.495 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:36.496 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.186784+0000", 2023-11-05T08:33:36.496 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0709000000000001e-05 2023-11-05T08:33:36.496 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.496 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.496 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:36.497 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.192059+0000", 2023-11-05T08:33:36.497 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0052751400000000002 2023-11-05T08:33:36.497 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.497 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.497 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:36.498 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.192142+0000", 2023-11-05T08:33:36.498 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.2979999999999995e-05 2023-11-05T08:33:36.498 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.498 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:36.498 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.499 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.499 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.499 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.as4 42/33 ECSubWrite(tid=38, reqid=osd.7.0:38, at_version=42'16, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:33:36.499 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:55.127593+0000", 2023-11-05T08:33:36.499 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 41.225673915999998, 2023-11-05T08:33:36.499 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.014145875, 2023-11-05T08:33:36.500 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:36.500 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:36.500 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:36.500 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.500 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:36.500 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127593+0000", 2023-11-05T08:33:36.501 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.501 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.501 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.501 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:36.501 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127593+0000", 2023-11-05T08:33:36.502 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.502 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.502 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.502 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:36.502 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127594+0000", 2023-11-05T08:33:36.503 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.1399999999999996e-07 2023-11-05T08:33:36.503 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.503 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.503 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:36.503 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127658+0000", 2023-11-05T08:33:36.503 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.3590000000000006e-05 2023-11-05T08:33:36.504 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.504 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.504 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:36.504 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127690+0000", 2023-11-05T08:33:36.504 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2447999999999998e-05 2023-11-05T08:33:36.505 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.505 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.505 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:36.505 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127716+0000", 2023-11-05T08:33:36.505 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5931999999999998e-05 2023-11-05T08:33:36.505 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.506 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.506 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:36.506 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127955+0000", 2023-11-05T08:33:36.506 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00023882100000000001 2023-11-05T08:33:36.506 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.506 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.507 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:36.507 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127983+0000", 2023-11-05T08:33:36.507 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7928000000000001e-05 2023-11-05T08:33:36.507 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.507 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.508 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:36.508 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.141638+0000", 2023-11-05T08:33:36.508 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.013655244 2023-11-05T08:33:36.508 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.508 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.509 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:36.509 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.141739+0000", 2023-11-05T08:33:36.509 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000101098 2023-11-05T08:33:36.509 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.509 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:36.509 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.510 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.510 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.510 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.as4 42/33 ECSubWrite(tid=39, reqid=osd.7.0:39, at_version=42'18, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:33:36.510 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:55.127804+0000", 2023-11-05T08:33:36.510 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 41.225463093000002, 2023-11-05T08:33:36.511 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.017037818, 2023-11-05T08:33:36.511 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:36.511 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:36.511 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:36.511 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.512 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:36.512 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127804+0000", 2023-11-05T08:33:36.512 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.512 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.512 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.513 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:36.513 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127804+0000", 2023-11-05T08:33:36.513 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.513 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.513 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.514 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:36.514 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127805+0000", 2023-11-05T08:33:36.514 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.026e-06 2023-11-05T08:33:36.514 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.514 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.515 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:36.515 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127848+0000", 2023-11-05T08:33:36.515 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.3195000000000002e-05 2023-11-05T08:33:36.515 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.515 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.515 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:36.516 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.133513+0000", 2023-11-05T08:33:36.516 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.005664741 2023-11-05T08:33:36.516 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.516 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.516 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:36.517 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.133539+0000", 2023-11-05T08:33:36.517 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6389000000000001e-05 2023-11-05T08:33:36.517 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.517 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.517 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:36.518 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.135275+0000", 2023-11-05T08:33:36.518 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001735349 2023-11-05T08:33:36.518 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.518 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.518 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:36.519 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.135312+0000", 2023-11-05T08:33:36.519 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.6965000000000003e-05 2023-11-05T08:33:36.519 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.519 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.519 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:36.520 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.144746+0000", 2023-11-05T08:33:36.520 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0094338319999999996 2023-11-05T08:33:36.520 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.520 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.520 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:36.521 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.144842+0000", 2023-11-05T08:33:36.521 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.6321000000000002e-05 2023-11-05T08:33:36.521 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.521 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:36.521 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.522 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.522 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.522 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:341 3.9s0 3:96f693b2:::smithi09039252-13:head [create,setxattr ]ScjNOsReWmcrsXtQckiNfVsraic]liXUgaZfrPb (12280) in=40b,setxattr RP^fql]PXPbbV]`\\UXhZUcc\\s\\eVVh]On^oUhNgZ (593) in=40b,setxattr ejajVrghj]sk_[VhasYoUSNhlO_PNopbXh\\iYad\\ (1963) in=40b,setxattr ocopbYhpUShgXjpVTtXhgrZ^fhTrWUZnka_sOlQ] (1506) in=40b,setxattr drSfWQp[TZZaai[S^\\ecpepRjetPTZ[Qn[slVhsn (1437) in=40b,setxattr qfWkbgV_hlqXWl`QOe]pOPUPZspbtU__NVPUUltj (877) in=40b,setxattr ]Vrng_\\anafncgktlQh_okb\\i]chR]P]jeoaR\\h] (1858) in=40b,setxattr ]W_abhUq]ocdkOt[ZkpjgPX^drZi]OcUWNRonRa\\ (455) in=40b,setxattr WiZeSWPfsqgYXk`d\\gOeSecbpakdsPTXZfZhn_k^ (875) in=40b,setxattr kqn^akbbamUTRXRhQS^i`lQYomhV_P`j\\ahTj]dk (1894) in=40b,setxattr TUcrapOXk_aZjTSdOT`rYcU_YYstnVgNTojk`lXZ (805) in=40b,setxattr a`RfekoeadQgiYossXre`rV^qNeUNiNf\\lSXhohq (15648) in=40b,setxattr VigYdf[rkVcbX`_aXcZrljeilZZV\\O[gpOhof]\\P (1473) in=40b,setxattr Qrfl^hc[_sp[p`bcN[Ne\\tg^]rgrOlRqfQ_VWNt\\ (1261) in=40b,setxattr iQVqR\\bssbQRQf^a`UrVROfgpPZlnstjPUaUnWWt (665) in=40b,setxattr aO^mjdTNk[qYOgYsgTigbbP`]mhe`WmaNUO[^ele (979) in=40b,setxattr cfQeZUernkePqgPcSrokkcnPqZ^Osc^^ng[ilga_ (16840) in=40b,setxattr fdbVa_]fdVq_`kcpaWlQSgY^VgakSqVhj`qqdj[m (1858) in=40b,setxattr tXieTi_TgP_dVipjmQcU_QVRqNdboYhRWgNrpVhm (1858) in=40b,setxattr ]YTcidkg\\][ZtajYPqmOji_UPVPZ`j^]tnNRefcd (2876) in=40b] snapc 10=[7] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:36.522 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:05.255132+0000", 2023-11-05T08:33:36.522 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 31.09813467, 2023-11-05T08:33:36.522 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.01486664, 2023-11-05T08:33:36.523 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:36.523 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:36.523 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:33:36.523 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:33:36.523 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:36.524 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 341 2023-11-05T08:33:36.524 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.524 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:36.524 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.524 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:36.524 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255132+0000", 2023-11-05T08:33:36.525 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.525 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.525 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.525 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:36.525 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255132+0000", 2023-11-05T08:33:36.526 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.526 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.526 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.526 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:36.526 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255136+0000", 2023-11-05T08:33:36.527 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.5159999999999999e-06 2023-11-05T08:33:36.527 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.527 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.527 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:36.527 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255423+0000", 2023-11-05T08:33:36.528 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00028735799999999998 2023-11-05T08:33:36.528 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.528 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.528 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:36.528 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255462+0000", 2023-11-05T08:33:36.528 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.8512e-05 2023-11-05T08:33:36.529 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.529 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.529 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:36.529 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255499+0000", 2023-11-05T08:33:36.529 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.7531999999999999e-05 2023-11-05T08:33:36.530 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.530 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.530 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:36.530 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255651+0000", 2023-11-05T08:33:36.530 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00015121700000000001 2023-11-05T08:33:36.531 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.531 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.531 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:33:36.531 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.256051+0000", 2023-11-05T08:33:36.531 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00040081699999999998 2023-11-05T08:33:36.532 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.532 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.532 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:36.532 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.257968+0000", 2023-11-05T08:33:36.532 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001917052 2023-11-05T08:33:36.532 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.533 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.533 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:36.533 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.265314+0000", 2023-11-05T08:33:36.533 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0073454469999999997 2023-11-05T08:33:36.533 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.534 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.534 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:33:36.534 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.269385+0000", 2023-11-05T08:33:36.534 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0040709839999999997 2023-11-05T08:33:36.534 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.534 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.535 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:36.535 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.269999+0000", 2023-11-05T08:33:36.535 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00061420500000000005 2023-11-05T08:33:36.535 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.535 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:36.536 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.536 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.536 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.536 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:344 3.9s0 3:936041dc:::smithi09039252-3:head [copy-from ver 4] snapc 10=[7] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:36.536 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:05.255538+0000", 2023-11-05T08:33:36.536 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 31.097728660000001, 2023-11-05T08:33:36.537 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.032906886000000003, 2023-11-05T08:33:36.537 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:36.537 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:36.537 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:33:36.537 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:33:36.538 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:36.538 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 344 2023-11-05T08:33:36.538 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.538 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:36.538 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.539 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:36.539 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255538+0000", 2023-11-05T08:33:36.539 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.539 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.539 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.540 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:36.540 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255538+0000", 2023-11-05T08:33:36.540 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.540 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.540 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.541 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:36.541 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255540+0000", 2023-11-05T08:33:36.541 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5060000000000001e-06 2023-11-05T08:33:36.541 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.541 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.542 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:36.542 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255550+0000", 2023-11-05T08:33:36.542 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0063e-05 2023-11-05T08:33:36.542 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.542 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.543 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:36.543 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255589+0000", 2023-11-05T08:33:36.543 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.9245000000000001e-05 2023-11-05T08:33:36.543 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.543 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.544 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:36.544 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255606+0000", 2023-11-05T08:33:36.544 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6307000000000002e-05 2023-11-05T08:33:36.544 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.544 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.545 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:36.545 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.259641+0000", 2023-11-05T08:33:36.545 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.004035421 2023-11-05T08:33:36.545 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.545 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.545 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:33:36.546 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.260112+0000", 2023-11-05T08:33:36.546 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00047097799999999999 2023-11-05T08:33:36.546 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.546 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.546 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:36.547 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.280466+0000", 2023-11-05T08:33:36.547 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.020353850999999999 2023-11-05T08:33:36.547 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.547 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.547 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:36.548 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.286378+0000", 2023-11-05T08:33:36.548 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0059124490000000002 2023-11-05T08:33:36.548 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.548 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.548 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:33:36.549 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.288092+0000", 2023-11-05T08:33:36.549 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0017133300000000001 2023-11-05T08:33:36.549 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.549 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.549 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:36.550 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.288445+0000", 2023-11-05T08:33:36.550 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000353736 2023-11-05T08:33:36.550 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.550 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:36.550 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.551 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.551 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.551 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:345 3.9s0 3:936041dc:::smithi09039252-3:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:36.551 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:05.255630+0000", 2023-11-05T08:33:36.551 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 31.097637200000001, 2023-11-05T08:33:36.552 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.029547922000000001, 2023-11-05T08:33:36.552 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:36.552 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2023-11-05T08:33:36.552 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:33:36.552 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:33:36.553 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:36.553 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 345 2023-11-05T08:33:36.553 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.553 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:36.553 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.554 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:36.554 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255630+0000", 2023-11-05T08:33:36.554 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.554 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.554 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.555 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:36.555 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255630+0000", 2023-11-05T08:33:36.555 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.555 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.556 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.556 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:36.556 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255631+0000", 2023-11-05T08:33:36.556 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.519e-06 2023-11-05T08:33:36.556 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.557 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.557 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:36.557 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255640+0000", 2023-11-05T08:33:36.557 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.9360000000000005e-06 2023-11-05T08:33:36.557 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.557 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.558 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:36.558 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255659+0000", 2023-11-05T08:33:36.558 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9089999999999998e-05 2023-11-05T08:33:36.558 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.558 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.559 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:36.559 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255675+0000", 2023-11-05T08:33:36.559 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5133000000000001e-05 2023-11-05T08:33:36.559 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.559 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.560 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:36.560 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.260602+0000", 2023-11-05T08:33:36.560 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0049270449999999997 2023-11-05T08:33:36.560 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.560 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.561 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for blocked object", 2023-11-05T08:33:36.561 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.260717+0000", 2023-11-05T08:33:36.561 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000115182 2023-11-05T08:33:36.561 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.561 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.562 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:36.562 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.284014+0000", 2023-11-05T08:33:36.562 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.023297577999999999 2023-11-05T08:33:36.562 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.562 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.563 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:33:36.563 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.284370+0000", 2023-11-05T08:33:36.563 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000355306 2023-11-05T08:33:36.563 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.563 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.564 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:36.564 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.285178+0000", 2023-11-05T08:33:36.564 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00080813300000000005 2023-11-05T08:33:36.564 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.564 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:36.565 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.565 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.565 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.565 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.as4 52/45 ECSubWrite(tid=45, reqid=client.4691.0:349, at_version=52'21, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:36.565 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:05.264746+0000", 2023-11-05T08:33:36.565 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 31.088521139000001, 2023-11-05T08:33:36.566 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0072026099999999999, 2023-11-05T08:33:36.566 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:36.566 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:36.566 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:36.566 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.567 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:36.567 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.264746+0000", 2023-11-05T08:33:36.567 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.567 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.567 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.568 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:36.568 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.264746+0000", 2023-11-05T08:33:36.568 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.568 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.568 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.569 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:36.569 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.264748+0000", 2023-11-05T08:33:36.569 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.677e-06 2023-11-05T08:33:36.569 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.569 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.570 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:36.570 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.264893+0000", 2023-11-05T08:33:36.570 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000145067 2023-11-05T08:33:36.570 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.570 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.571 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:36.571 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.264921+0000", 2023-11-05T08:33:36.571 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8328999999999999e-05 2023-11-05T08:33:36.571 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.571 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.572 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:36.572 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.264942+0000", 2023-11-05T08:33:36.572 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1046000000000001e-05 2023-11-05T08:33:36.572 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.572 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.572 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:36.573 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.265123+0000", 2023-11-05T08:33:36.573 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00018091299999999999 2023-11-05T08:33:36.573 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.573 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.573 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:36.574 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.265150+0000", 2023-11-05T08:33:36.574 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6645e-05 2023-11-05T08:33:36.574 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.574 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.574 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:36.575 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.271806+0000", 2023-11-05T08:33:36.575 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0066566919999999996 2023-11-05T08:33:36.575 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.575 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.575 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:36.576 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.271949+0000", 2023-11-05T08:33:36.576 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00014224100000000001 2023-11-05T08:33:36.576 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.576 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:36.576 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.577 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.577 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.577 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.as4 61/45 ECSubWrite(tid=57, reqid=client.4691.0:405, at_version=61'23, trim_to=0'0, roll_forward_to=52'21))", 2023-11-05T08:33:36.577 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:14.366205+0000", 2023-11-05T08:33:36.577 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 21.987061719, 2023-11-05T08:33:36.577 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0087065779999999995, 2023-11-05T08:33:36.578 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:36.578 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:36.578 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:36.578 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.578 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:36.579 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.366205+0000", 2023-11-05T08:33:36.579 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.579 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.579 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.579 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:36.580 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.366205+0000", 2023-11-05T08:33:36.580 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.580 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.580 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.580 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:36.581 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.366206+0000", 2023-11-05T08:33:36.581 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.3000000000000001e-07 2023-11-05T08:33:36.581 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.581 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.581 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:36.581 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.366951+0000", 2023-11-05T08:33:36.582 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00074537699999999998 2023-11-05T08:33:36.582 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.582 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.582 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:36.582 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.366978+0000", 2023-11-05T08:33:36.583 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6721999999999999e-05 2023-11-05T08:33:36.583 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.583 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.583 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:36.583 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.366994+0000", 2023-11-05T08:33:36.584 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6189999999999999e-05 2023-11-05T08:33:36.584 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.584 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.584 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:36.584 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.367186+0000", 2023-11-05T08:33:36.584 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000191387 2023-11-05T08:33:36.585 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.585 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.585 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:36.585 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.367213+0000", 2023-11-05T08:33:36.585 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7169e-05 2023-11-05T08:33:36.585 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.586 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.586 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:36.586 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.374835+0000", 2023-11-05T08:33:36.586 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0076217589999999997 2023-11-05T08:33:36.586 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.587 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.587 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:36.587 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.374912+0000", 2023-11-05T08:33:36.587 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.7443999999999999e-05 2023-11-05T08:33:36.587 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.588 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:36.588 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.588 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.588 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.588 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.fs2 77/37 ECSubWrite(tid=130, reqid=client.4691.0:579, at_version=77'44, trim_to=0'0, roll_forward_to=70'42))", 2023-11-05T08:33:36.588 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:30.540041+0000", 2023-11-05T08:33:36.589 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 5.8132265050000003, 2023-11-05T08:33:36.589 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0083478800000000002, 2023-11-05T08:33:36.589 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:36.589 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:36.589 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:36.590 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.590 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:36.590 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.540041+0000", 2023-11-05T08:33:36.590 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.590 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.591 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.591 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:36.591 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.540041+0000", 2023-11-05T08:33:36.591 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:36.591 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.591 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.592 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:36.592 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.540042+0000", 2023-11-05T08:33:36.592 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0130000000000001e-06 2023-11-05T08:33:36.592 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.592 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.592 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:36.593 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.540574+0000", 2023-11-05T08:33:36.593 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00053246000000000001 2023-11-05T08:33:36.593 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.593 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.593 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:36.593 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.540600+0000", 2023-11-05T08:33:36.594 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6007999999999999e-05 2023-11-05T08:33:36.594 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.594 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.594 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:36.594 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.540623+0000", 2023-11-05T08:33:36.595 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3416000000000001e-05 2023-11-05T08:33:36.595 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.595 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.595 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:36.595 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.540798+0000", 2023-11-05T08:33:36.596 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000174897 2023-11-05T08:33:36.596 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.596 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.596 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:36.596 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.540830+0000", 2023-11-05T08:33:36.596 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.1943999999999998e-05 2023-11-05T08:33:36.597 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.597 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.597 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:36.597 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.548328+0000", 2023-11-05T08:33:36.597 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0074971990000000004 2023-11-05T08:33:36.598 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:36.598 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:36.598 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:36.598 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.548388+0000", 2023-11-05T08:33:36.598 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.0943000000000002e-05 2023-11-05T08:33:36.599 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.599 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:36.599 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.599 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:36.599 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:36.599 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:33:36.600 INFO:tasks.rados.rados.0.smithi090.stdout:265: done (2 left) 2023-11-05T08:33:36.601 INFO:tasks.rados.rados.0.smithi090.stdout:266: snap_remove snap 22 2023-11-05T08:33:36.601 INFO:teuthology.orchestra.run.smithi002.stdout:osd.14: {} 2023-11-05T08:33:36.602 INFO:teuthology.orchestra.run.smithi002.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-11-05T08:33:36.602 INFO:teuthology.orchestra.run.smithi002.stdout:osd.15: {} 2023-11-05T08:33:36.603 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-11-05T08:33:36.604 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-05T08:33:36.604 INFO:teuthology.orchestra.run.smithi002.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-11-05T08:33:36.604 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-05T08:33:36.605 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:36.599+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:36.701 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-05T08:33:36.741 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:36.741 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:36.755 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-11-05T08:33:36.803 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:33:36.797+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:33:36.876 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:36.876 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:36.890 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-11-05T08:33:36.904 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:36.900+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:37.005 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:33:37.000+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:33:37.009 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:37.009 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:37.023 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2023-11-05T08:33:37.105 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:33:37.101+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:33:37.144 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:37.145 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:37.158 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2023-11-05T08:33:37.206 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:33:37.205+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:33:37.282 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:37.285 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:37.297 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2023-11-05T08:33:37.308 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:33:37.305+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:33:37.407 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-05T08:33:37.420 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:37.420 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:37.434 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2023-11-05T08:33:37.510 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:33:37.508+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:33:37.552 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:37.552 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:37.564 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-11-05T08:33:37.566 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2023-11-05T08:33:37.576 INFO:tasks.rados.rados.0.smithi090.stdout:266: done (2 left) 2023-11-05T08:33:37.576 INFO:tasks.rados.rados.0.smithi090.stdout:267: snap_remove snap 25 2023-11-05T08:33:37.612 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:33:37.609+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:33:37.687 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:37.688 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:37.702 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2023-11-05T08:33:37.711 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:33:37.709+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:33:37.813 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:33:37.808+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:33:37.864 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:37.864 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:37.879 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-11-05T08:33:37.913 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:33:37.909+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:33:37.984 INFO:teuthology.orchestra.run.smithi002.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-05T08:33:37.987 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-05T08:33:37.987 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-11-05T08:33:38.036 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:33:38.033+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:33:38.097 INFO:teuthology.orchestra.run.smithi002.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-05T08:33:38.100 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-05T08:33:38.100 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-11-05T08:33:38.115 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:38.112+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:38.215 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:33:38.212+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:33:38.255 INFO:teuthology.orchestra.run.smithi002.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-05T08:33:38.258 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-05T08:33:38.258 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-11-05T08:33:38.315 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-05T08:33:38.387 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:38.387 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:38.402 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-11-05T08:33:38.417 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-05T08:33:38.520 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:33:38.517+0000 7fdc5cd58640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:33:38.522 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:38.522 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:38.536 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-11-05T08:33:38.579 INFO:teuthology.orchestra.run.smithi002.stderr:nodeep-scrub is unset 2023-11-05T08:33:38.588 INFO:tasks.rados.rados.0.smithi090.stdout:267: done (2 left) 2023-11-05T08:33:38.588 INFO:tasks.rados.rados.0.smithi090.stdout:268: read oid 11 snap -1 2023-11-05T08:33:38.588 INFO:tasks.rados.rados.0.smithi090.stdout:268: expect (ObjNum 75 snap 9 seq_num 75) 2023-11-05T08:33:38.588 INFO:tasks.rados.rados.0.smithi090.stdout:269: rollback oid 21 current snap is 26 2023-11-05T08:33:38.589 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 21 to 4 2023-11-05T08:33:38.589 INFO:tasks.rados.rados.0.smithi090.stdout:264: done (3 left) 2023-11-05T08:33:38.589 INFO:tasks.rados.rados.0.smithi090.stdout:270: copy_from oid 34 from oid 23 current snap is 26 2023-11-05T08:33:38.589 INFO:tasks.rados.rados.0.smithi090.stdout:271: rollback oid 39 current snap is 26 2023-11-05T08:33:38.589 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 39 to 24 2023-11-05T08:33:38.589 INFO:tasks.rados.rados.0.smithi090.stdout:272: snap_create 2023-11-05T08:33:38.606 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-05T08:33:38.608 INFO:tasks.rados.rados.0.smithi090.stdout:271: finishing rollback tid 0 to smithi09039252-39 2023-11-05T08:33:38.609 INFO:tasks.rados.rados.0.smithi090.stdout:269: finishing rollback tid 1 to smithi09039252-21 2023-11-05T08:33:38.610 INFO:tasks.rados.rados.0.smithi090.stdout:269: finishing rollback tid 2 to smithi09039252-21 2023-11-05T08:33:38.611 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 21 v 22 (ObjNum 20 snap 0 seq_num 20) dirty exists 2023-11-05T08:33:38.611 INFO:tasks.rados.rados.0.smithi090.stdout:271: finishing rollback tid 1 to smithi09039252-39 2023-11-05T08:33:38.613 INFO:tasks.rados.rados.0.smithi090.stdout:271: finishing rollback tid 2 to smithi09039252-39 2023-11-05T08:33:38.613 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 39 v 52 (ObjNum 89 snap 16 seq_num 89) dirty exists 2023-11-05T08:33:38.616 INFO:tasks.rados.rados.0.smithi090.stdout:268: expect (ObjNum 75 snap 9 seq_num 75) 2023-11-05T08:33:38.620 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:33:38.617+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:33:38.695 INFO:tasks.rados.rados.0.smithi090.stdout:270: finishing copy_from racing read to smithi09039252-34 2023-11-05T08:33:38.696 INFO:tasks.rados.rados.0.smithi090.stdout:270: finishing copy_from to smithi09039252-34 2023-11-05T08:33:38.696 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 34 v 21 (ObjNum 60 snap 1 seq_num 60) dirty exists 2023-11-05T08:33:38.714 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:38.714 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:38.720 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:33:38.717+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:33:38.730 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2023-11-05T08:33:38.781 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-11-05T08:33:38.803 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1: {} 2023-11-05T08:33:38.804 INFO:teuthology.orchestra.run.smithi002.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-11-05T08:33:38.821 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:33:38.816+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:33:38.823 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2: {} 2023-11-05T08:33:38.824 INFO:teuthology.orchestra.run.smithi002.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-11-05T08:33:38.824 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-11-05T08:33:38.841 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4: {} 2023-11-05T08:33:38.841 INFO:teuthology.orchestra.run.smithi002.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-11-05T08:33:38.861 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5: {} 2023-11-05T08:33:38.861 INFO:teuthology.orchestra.run.smithi002.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-11-05T08:33:38.883 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6: {} 2023-11-05T08:33:38.883 INFO:teuthology.orchestra.run.smithi002.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-11-05T08:33:38.887 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:38.887 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:38.902 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2023-11-05T08:33:38.905 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7: {} 2023-11-05T08:33:38.906 INFO:teuthology.orchestra.run.smithi002.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-11-05T08:33:38.922 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:33:38.919+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:33:38.924 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8: {} 2023-11-05T08:33:38.924 INFO:teuthology.orchestra.run.smithi002.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-11-05T08:33:38.944 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9: {} 2023-11-05T08:33:38.945 INFO:teuthology.orchestra.run.smithi002.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-11-05T08:33:38.966 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10: {} 2023-11-05T08:33:38.966 INFO:teuthology.orchestra.run.smithi002.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-11-05T08:33:38.987 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11: {} 2023-11-05T08:33:38.987 INFO:teuthology.orchestra.run.smithi002.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-11-05T08:33:39.004 INFO:teuthology.orchestra.run.smithi002.stdout:osd.12: {} 2023-11-05T08:33:39.004 INFO:teuthology.orchestra.run.smithi002.stderr:osd.12: osd_enable_op_tracker = 'true' 2023-11-05T08:33:39.023 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:33:39.023 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [], 2023-11-05T08:33:39.023 INFO:teuthology.orchestra.run.smithi002.stdout: "complaint_time": 30, 2023-11-05T08:33:39.023 INFO:teuthology.orchestra.run.smithi002.stdout: "num_blocked_ops": 0 2023-11-05T08:33:39.023 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:33:39.024 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:33:39.019+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:33:39.025 INFO:teuthology.orchestra.run.smithi002.stdout:osd.13: {} 2023-11-05T08:33:39.025 INFO:teuthology.orchestra.run.smithi002.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-11-05T08:33:39.037 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2023-11-05T08:33:39.046 INFO:teuthology.orchestra.run.smithi002.stdout:osd.14: {} 2023-11-05T08:33:39.046 INFO:teuthology.orchestra.run.smithi002.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-11-05T08:33:39.068 INFO:teuthology.orchestra.run.smithi002.stdout:osd.15: {} 2023-11-05T08:33:39.068 INFO:teuthology.orchestra.run.smithi002.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-11-05T08:33:39.072 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-05T08:33:39.073 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-05T08:33:39.123 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:33:39.121+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:33:39.166 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:33:39.166 INFO:teuthology.orchestra.run.smithi002.stdout: "size": 20, 2023-11-05T08:33:39.167 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 600, 2023-11-05T08:33:39.167 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [ 2023-11-05T08:33:39.167 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.167 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.ds5 36/30 ECSubWrite(tid=23, reqid=osd.7.0:23, at_version=36'14, trim_to=0'0, roll_forward_to=34'8))", 2023-11-05T08:33:39.167 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:49.078802+0000", 2023-11-05T08:33:39.168 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 50.085220552000003, 2023-11-05T08:33:39.168 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0035119359999999998, 2023-11-05T08:33:39.168 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:39.168 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:39.168 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:39.169 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.169 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:39.169 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.078802+0000", 2023-11-05T08:33:39.169 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.169 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.170 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.170 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:39.170 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.078802+0000", 2023-11-05T08:33:39.170 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.170 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.171 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.171 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:39.171 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.078802+0000", 2023-11-05T08:33:39.171 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.5899999999999996e-07 2023-11-05T08:33:39.171 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.171 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.172 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:39.172 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.078834+0000", 2023-11-05T08:33:39.172 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.1696999999999998e-05 2023-11-05T08:33:39.172 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.172 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.173 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:39.173 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.078845+0000", 2023-11-05T08:33:39.173 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0992000000000001e-05 2023-11-05T08:33:39.173 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.173 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.174 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:39.174 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.078851+0000", 2023-11-05T08:33:39.174 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.1360000000000002e-06 2023-11-05T08:33:39.174 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.174 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.174 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:39.175 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.079095+0000", 2023-11-05T08:33:39.175 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00024419600000000002 2023-11-05T08:33:39.175 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.175 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.175 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:39.176 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.079105+0000", 2023-11-05T08:33:39.176 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0341e-05 2023-11-05T08:33:39.176 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.176 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.176 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:39.177 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.082270+0000", 2023-11-05T08:33:39.177 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0031646719999999999 2023-11-05T08:33:39.177 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.177 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.177 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:39.178 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:49.082313+0000", 2023-11-05T08:33:39.178 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.3343000000000002e-05 2023-11-05T08:33:39.178 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.178 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:39.178 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.179 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.179 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.179 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:278 3.14s0 3:292b6c2e:::smithi09039252-40:head [copy-from ver 10] snapc 8=[8,7,6,5,3] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:33:39.179 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:53.143346+0000", 2023-11-05T08:33:39.179 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 46.020676280000004, 2023-11-05T08:33:39.180 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0524983919999999, 2023-11-05T08:33:39.180 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:39.180 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:39.180 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:39.180 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:39.181 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:39.181 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 278 2023-11-05T08:33:39.181 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.181 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:39.181 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.181 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:39.182 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143346+0000", 2023-11-05T08:33:39.182 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.182 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.182 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.182 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:39.182 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143346+0000", 2023-11-05T08:33:39.183 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.183 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.183 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.183 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:39.183 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143347+0000", 2023-11-05T08:33:39.184 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4470000000000001e-06 2023-11-05T08:33:39.184 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.184 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.184 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:39.184 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143353+0000", 2023-11-05T08:33:39.185 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.7790000000000001e-06 2023-11-05T08:33:39.185 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.185 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.185 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:39.185 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143367+0000", 2023-11-05T08:33:39.185 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4321000000000001e-05 2023-11-05T08:33:39.186 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.186 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.186 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:39.186 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143374+0000", 2023-11-05T08:33:39.186 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.8029999999999999e-06 2023-11-05T08:33:39.187 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.187 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.187 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:39.187 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143446+0000", 2023-11-05T08:33:39.187 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.2080000000000001e-05 2023-11-05T08:33:39.188 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.188 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.188 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for peered", 2023-11-05T08:33:39.188 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143450+0000", 2023-11-05T08:33:39.188 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.5449999999999999e-06 2023-11-05T08:33:39.189 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.189 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.189 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:39.189 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.169316+0000", 2023-11-05T08:33:39.189 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.025866494 2023-11-05T08:33:39.190 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.190 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.190 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:39.190 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.169711+0000", 2023-11-05T08:33:39.190 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00039465700000000003 2023-11-05T08:33:39.191 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.191 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.191 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:39.191 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.185825+0000", 2023-11-05T08:33:39.191 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.016113624 2023-11-05T08:33:39.192 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.192 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.192 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:39.192 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.190579+0000", 2023-11-05T08:33:39.192 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0047547370000000002 2023-11-05T08:33:39.193 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.193 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.193 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:33:39.193 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.195484+0000", 2023-11-05T08:33:39.193 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0049050200000000004 2023-11-05T08:33:39.193 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.194 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.194 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:39.194 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.195844+0000", 2023-11-05T08:33:39.194 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000359885 2023-11-05T08:33:39.194 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.195 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:39.195 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.195 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.195 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.195 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:279 3.14s0 3:292b6c2e:::smithi09039252-40:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:33:39.196 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:53.143390+0000", 2023-11-05T08:33:39.196 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 46.020632204000002, 2023-11-05T08:33:39.196 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0469768799999999, 2023-11-05T08:33:39.196 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:39.196 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "started", 2023-11-05T08:33:39.197 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:39.197 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:39.197 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:39.197 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 279 2023-11-05T08:33:39.197 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.198 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:39.198 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.198 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:39.198 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143390+0000", 2023-11-05T08:33:39.198 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.199 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.199 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.199 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:39.199 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143390+0000", 2023-11-05T08:33:39.199 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.200 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.200 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.200 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:39.200 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143390+0000", 2023-11-05T08:33:39.200 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.5999999999999999e-07 2023-11-05T08:33:39.200 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.201 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.201 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:39.201 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143392+0000", 2023-11-05T08:33:39.201 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.0209999999999998e-06 2023-11-05T08:33:39.201 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.202 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.202 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:39.202 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143399+0000", 2023-11-05T08:33:39.202 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.6869999999999997e-06 2023-11-05T08:33:39.203 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.203 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.203 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:39.203 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143405+0000", 2023-11-05T08:33:39.203 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.0619999999999997e-06 2023-11-05T08:33:39.204 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.204 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.204 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:39.204 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143483+0000", 2023-11-05T08:33:39.204 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.8268000000000003e-05 2023-11-05T08:33:39.205 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.205 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.205 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for peered", 2023-11-05T08:33:39.205 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143486+0000", 2023-11-05T08:33:39.205 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.322e-06 2023-11-05T08:33:39.206 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.206 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.206 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:39.206 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.172078+0000", 2023-11-05T08:33:39.206 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0285927589999999 2023-11-05T08:33:39.207 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.207 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.207 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for blocked object", 2023-11-05T08:33:39.207 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.172240+0000", 2023-11-05T08:33:39.207 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00016178899999999999 2023-11-05T08:33:39.207 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.208 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.208 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:39.208 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.189404+0000", 2023-11-05T08:33:39.208 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.017164220000000001 2023-11-05T08:33:39.208 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.209 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.209 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:39.209 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.189710+0000", 2023-11-05T08:33:39.209 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000305617 2023-11-05T08:33:39.209 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.210 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.210 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:39.210 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.190367+0000", 2023-11-05T08:33:39.210 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00065677499999999996 2023-11-05T08:33:39.210 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.211 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:39.211 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.211 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.211 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.211 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es4 41/30 ECSubWrite(tid=40, reqid=osd.2.0:40, at_version=41'27, trim_to=0'0, roll_forward_to=40'25))", 2023-11-05T08:33:39.212 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:54.120687+0000", 2023-11-05T08:33:39.212 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 45.043335419999998, 2023-11-05T08:33:39.212 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0097705859999999995, 2023-11-05T08:33:39.212 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:39.212 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:39.212 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:39.213 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.213 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:39.213 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120687+0000", 2023-11-05T08:33:39.213 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.213 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.214 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.214 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:39.214 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120687+0000", 2023-11-05T08:33:39.214 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.214 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.215 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.215 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:39.215 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120688+0000", 2023-11-05T08:33:39.215 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.64e-06 2023-11-05T08:33:39.215 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.215 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.216 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:39.216 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120741+0000", 2023-11-05T08:33:39.216 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.2274000000000001e-05 2023-11-05T08:33:39.216 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.216 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.217 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:39.217 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120770+0000", 2023-11-05T08:33:39.217 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.9768999999999999e-05 2023-11-05T08:33:39.217 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.217 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.218 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:39.218 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120795+0000", 2023-11-05T08:33:39.218 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.4805000000000001e-05 2023-11-05T08:33:39.218 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.218 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.219 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:39.219 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.124189+0000", 2023-11-05T08:33:39.219 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0033935279999999998 2023-11-05T08:33:39.219 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.219 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.220 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:39.220 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.124214+0000", 2023-11-05T08:33:39.220 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.5823e-05 2023-11-05T08:33:39.220 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.221 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.221 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:39.221 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.130330+0000", 2023-11-05T08:33:39.221 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0061156190000000001 2023-11-05T08:33:39.221 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.221 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.222 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:39.222 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.130457+0000", 2023-11-05T08:33:39.222 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00012712800000000001 2023-11-05T08:33:39.222 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.222 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:39.223 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.223 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.223 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.223 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es4 42/30 ECSubWrite(tid=43, reqid=osd.2.0:43, at_version=42'29, trim_to=0'0, roll_forward_to=41'27))", 2023-11-05T08:33:39.223 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:55.140218+0000", 2023-11-05T08:33:39.224 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 44.023803579000003, 2023-11-05T08:33:39.224 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0038969510000000001, 2023-11-05T08:33:39.224 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:39.224 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:39.224 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:39.225 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.225 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:39.225 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.140218+0000", 2023-11-05T08:33:39.225 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.225 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.226 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.226 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:39.226 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.140218+0000", 2023-11-05T08:33:39.226 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.226 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.226 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.227 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:39.227 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.140219+0000", 2023-11-05T08:33:39.227 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.7000000000000005e-07 2023-11-05T08:33:39.227 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.227 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.228 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:39.228 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.140283+0000", 2023-11-05T08:33:39.228 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.3738000000000005e-05 2023-11-05T08:33:39.228 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.228 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.229 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:39.229 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.140317+0000", 2023-11-05T08:33:39.229 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.4498000000000001e-05 2023-11-05T08:33:39.229 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.229 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.230 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:39.230 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.140342+0000", 2023-11-05T08:33:39.230 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.4742000000000001e-05 2023-11-05T08:33:39.230 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.230 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.231 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:39.231 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.140849+0000", 2023-11-05T08:33:39.231 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00050743999999999995 2023-11-05T08:33:39.231 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.231 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.231 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:39.232 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.140876+0000", 2023-11-05T08:33:39.232 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.6798999999999999e-05 2023-11-05T08:33:39.232 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.232 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.232 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:39.233 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.144042+0000", 2023-11-05T08:33:39.233 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0031655870000000001 2023-11-05T08:33:39.233 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.233 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.233 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:39.234 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.144115+0000", 2023-11-05T08:33:39.234 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.3577000000000004e-05 2023-11-05T08:33:39.234 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.234 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:39.234 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.235 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.235 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.235 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpReadReply(3.14s0 47/46 ECSubReadReply(tid=22, attrs_read=0))", 2023-11-05T08:33:39.235 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:00.232364+0000", 2023-11-05T08:33:39.235 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 38.931657606000002, 2023-11-05T08:33:39.235 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0035804690000000002, 2023-11-05T08:33:39.236 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:39.236 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:39.236 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:39.236 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.236 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:39.237 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.232364+0000", 2023-11-05T08:33:39.237 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.237 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.237 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.237 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:39.238 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.232364+0000", 2023-11-05T08:33:39.238 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.238 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.238 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.238 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:39.238 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.232366+0000", 2023-11-05T08:33:39.239 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.393e-06 2023-11-05T08:33:39.239 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.239 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.239 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:39.239 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.232910+0000", 2023-11-05T08:33:39.240 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000543938 2023-11-05T08:33:39.240 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.240 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.240 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:39.240 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.232931+0000", 2023-11-05T08:33:39.241 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.1599000000000001e-05 2023-11-05T08:33:39.241 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.241 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.241 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:39.241 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.232961+0000", 2023-11-05T08:33:39.241 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.9949000000000001e-05 2023-11-05T08:33:39.242 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.242 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.242 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:39.242 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.233087+0000", 2023-11-05T08:33:39.242 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00012517499999999999 2023-11-05T08:33:39.243 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.243 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.243 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:39.243 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.235945+0000", 2023-11-05T08:33:39.243 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.002858415 2023-11-05T08:33:39.244 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.244 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:39.244 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.244 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.244 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.245 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDPGPushReply(3.14s0 47/46 [PushReplyOp(3:292b6c2e:::smithi09039252-40:7)])", 2023-11-05T08:33:39.245 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:00.249065+0000", 2023-11-05T08:33:39.245 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 38.914957338999997, 2023-11-05T08:33:39.245 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0037023899999999998, 2023-11-05T08:33:39.245 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:39.245 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:39.246 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:39.246 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.246 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:39.246 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.249065+0000", 2023-11-05T08:33:39.246 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.247 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.247 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.247 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:39.247 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.249065+0000", 2023-11-05T08:33:39.247 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.248 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.248 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.248 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:39.248 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.249066+0000", 2023-11-05T08:33:39.248 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.73e-06 2023-11-05T08:33:39.249 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.249 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.249 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:39.249 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.249080+0000", 2023-11-05T08:33:39.249 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.3196e-05 2023-11-05T08:33:39.250 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.250 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.250 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:39.250 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.249117+0000", 2023-11-05T08:33:39.251 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.7419000000000002e-05 2023-11-05T08:33:39.251 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.251 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.251 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:39.251 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.249134+0000", 2023-11-05T08:33:39.251 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.7079e-05 2023-11-05T08:33:39.252 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.252 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.252 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:39.252 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.249248+0000", 2023-11-05T08:33:39.252 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000113929 2023-11-05T08:33:39.253 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.253 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.253 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:39.253 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.252767+0000", 2023-11-05T08:33:39.253 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.003519037 2023-11-05T08:33:39.254 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.254 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:39.254 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.254 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.254 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.255 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDPGPushReply(3.14s0 47/46 [PushReplyOp(3:292b6c2e:::smithi09039252-40:head)])", 2023-11-05T08:33:39.255 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:00.292862+0000", 2023-11-05T08:33:39.255 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 38.871160513, 2023-11-05T08:33:39.255 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0063631419999999996, 2023-11-05T08:33:39.255 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:39.255 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:39.256 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:39.256 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.256 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:39.256 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.292862+0000", 2023-11-05T08:33:39.256 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.257 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.257 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.257 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:39.257 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.292862+0000", 2023-11-05T08:33:39.257 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.258 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.258 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.258 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:39.258 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.292862+0000", 2023-11-05T08:33:39.258 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.0200000000000001e-07 2023-11-05T08:33:39.259 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.259 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.259 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:39.259 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.292867+0000", 2023-11-05T08:33:39.259 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.7829999999999998e-06 2023-11-05T08:33:39.260 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.260 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.260 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:39.260 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.292877+0000", 2023-11-05T08:33:39.260 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0312000000000001e-05 2023-11-05T08:33:39.260 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.261 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.261 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:39.261 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.292889+0000", 2023-11-05T08:33:39.261 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.2019e-05 2023-11-05T08:33:39.261 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.262 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.262 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:39.262 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.293399+0000", 2023-11-05T08:33:39.262 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00050963099999999997 2023-11-05T08:33:39.262 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.263 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.263 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:39.263 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.299225+0000", 2023-11-05T08:33:39.263 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0058256949999999997 2023-11-05T08:33:39.263 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.264 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:39.264 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.264 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.264 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.264 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.16s3 51/45 ECSubWrite(tid=58, reqid=osd.11.0:58, at_version=51'19, trim_to=0'0, roll_forward_to=49'17))", 2023-11-05T08:33:39.265 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:04.227640+0000", 2023-11-05T08:33:39.265 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 34.936381722999997, 2023-11-05T08:33:39.265 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.003497839, 2023-11-05T08:33:39.265 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:39.265 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:39.265 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:39.266 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.266 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:39.266 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.227640+0000", 2023-11-05T08:33:39.266 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.266 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.267 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.267 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:39.267 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.227640+0000", 2023-11-05T08:33:39.267 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.267 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.268 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.268 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:39.268 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.227643+0000", 2023-11-05T08:33:39.268 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.2129999999999998e-06 2023-11-05T08:33:39.268 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.268 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.269 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:39.269 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.227704+0000", 2023-11-05T08:33:39.269 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.1748000000000006e-05 2023-11-05T08:33:39.269 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.269 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.270 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:39.270 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.227744+0000", 2023-11-05T08:33:39.270 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.9598999999999998e-05 2023-11-05T08:33:39.270 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.270 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.271 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:39.271 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.227769+0000", 2023-11-05T08:33:39.271 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.5182999999999999e-05 2023-11-05T08:33:39.271 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.271 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.271 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:39.272 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.227954+0000", 2023-11-05T08:33:39.272 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000185167 2023-11-05T08:33:39.272 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.272 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.272 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:39.273 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.227977+0000", 2023-11-05T08:33:39.273 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.2586e-05 2023-11-05T08:33:39.273 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.273 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.273 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:39.273 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.231028+0000", 2023-11-05T08:33:39.274 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0030515400000000002 2023-11-05T08:33:39.274 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.274 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.274 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:39.274 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.231138+0000", 2023-11-05T08:33:39.275 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00010980300000000001 2023-11-05T08:33:39.275 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.275 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:39.275 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.275 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.276 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.276 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.16s3 56/45 ECSubWrite(tid=63, reqid=client.4691.0:369, at_version=56'20, trim_to=0'0, roll_forward_to=51'19))", 2023-11-05T08:33:39.276 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:09.293412+0000", 2023-11-05T08:33:39.276 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 29.870609618, 2023-11-05T08:33:39.276 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0049197820000000001, 2023-11-05T08:33:39.276 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:39.277 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:39.277 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:39.277 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.277 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:39.277 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.293412+0000", 2023-11-05T08:33:39.278 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.278 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.278 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.278 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:39.278 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.293412+0000", 2023-11-05T08:33:39.279 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.279 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.279 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.279 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:39.279 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.293413+0000", 2023-11-05T08:33:39.279 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.0299999999999998e-07 2023-11-05T08:33:39.280 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.280 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.280 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:39.280 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.293961+0000", 2023-11-05T08:33:39.280 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00054824799999999999 2023-11-05T08:33:39.281 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.281 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.281 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:39.281 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.293997+0000", 2023-11-05T08:33:39.281 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.5991999999999997e-05 2023-11-05T08:33:39.282 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.282 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.282 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:39.282 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.294022+0000", 2023-11-05T08:33:39.282 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.4865999999999999e-05 2023-11-05T08:33:39.282 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.283 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.283 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:39.283 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.294234+0000", 2023-11-05T08:33:39.283 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00021178399999999999 2023-11-05T08:33:39.283 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.284 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.284 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:39.284 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.294264+0000", 2023-11-05T08:33:39.284 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.0309000000000001e-05 2023-11-05T08:33:39.284 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.285 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.285 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:39.285 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.298238+0000", 2023-11-05T08:33:39.285 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0039737959999999999 2023-11-05T08:33:39.285 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.285 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.286 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:39.286 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.298332+0000", 2023-11-05T08:33:39.286 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.3984e-05 2023-11-05T08:33:39.286 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.286 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:39.287 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.287 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.287 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.287 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:374 3.14s0 3:292b6c2e:::smithi09039252-40:head [copy-from ver 2] snapc 15=[7] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2023-11-05T08:33:39.287 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:09.297419+0000", 2023-11-05T08:33:39.288 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 29.866602870000001, 2023-11-05T08:33:39.288 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.03004254, 2023-11-05T08:33:39.288 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:39.288 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:39.288 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:39.288 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:39.289 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:39.289 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 374 2023-11-05T08:33:39.289 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.289 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:39.289 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.290 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:39.290 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297419+0000", 2023-11-05T08:33:39.290 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.290 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.290 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.291 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:39.291 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297419+0000", 2023-11-05T08:33:39.291 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.291 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.291 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.292 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:39.292 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297422+0000", 2023-11-05T08:33:39.292 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.717e-06 2023-11-05T08:33:39.292 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.292 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.293 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:39.293 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297437+0000", 2023-11-05T08:33:39.293 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4793e-05 2023-11-05T08:33:39.293 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.293 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.293 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:39.294 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297468+0000", 2023-11-05T08:33:39.294 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.1519000000000001e-05 2023-11-05T08:33:39.294 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.294 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.294 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:39.295 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297490+0000", 2023-11-05T08:33:39.295 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.1651999999999999e-05 2023-11-05T08:33:39.295 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.295 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.295 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:39.296 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.300337+0000", 2023-11-05T08:33:39.296 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0028475219999999999 2023-11-05T08:33:39.296 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.296 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.297 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:39.297 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.300717+0000", 2023-11-05T08:33:39.297 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00037937600000000001 2023-11-05T08:33:39.297 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.297 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.298 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:39.298 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.319194+0000", 2023-11-05T08:33:39.298 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.018477303 2023-11-05T08:33:39.298 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.298 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.299 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:39.299 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.322493+0000", 2023-11-05T08:33:39.299 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0032987250000000002 2023-11-05T08:33:39.299 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.299 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.300 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:33:39.300 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.326714+0000", 2023-11-05T08:33:39.300 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0042208319999999999 2023-11-05T08:33:39.300 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.300 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.300 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:39.301 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.327462+0000", 2023-11-05T08:33:39.301 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00074810100000000004 2023-11-05T08:33:39.301 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.301 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:39.301 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.302 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.302 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.302 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:375 3.14s0 3:292b6c2e:::smithi09039252-40:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e56)", 2023-11-05T08:33:39.302 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:09.297530+0000", 2023-11-05T08:33:39.302 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 29.866492043000001, 2023-11-05T08:33:39.303 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.024433857999999999, 2023-11-05T08:33:39.303 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:39.303 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "started", 2023-11-05T08:33:39.303 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:39.303 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:39.304 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:39.304 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 375 2023-11-05T08:33:39.304 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.304 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:39.304 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.304 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:39.305 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297530+0000", 2023-11-05T08:33:39.305 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.305 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.305 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.305 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:39.306 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297530+0000", 2023-11-05T08:33:39.306 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.306 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.306 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.306 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:39.307 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297531+0000", 2023-11-05T08:33:39.307 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.48e-06 2023-11-05T08:33:39.307 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.307 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.307 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:39.307 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297540+0000", 2023-11-05T08:33:39.308 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.49e-06 2023-11-05T08:33:39.308 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.308 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.308 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:39.308 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297566+0000", 2023-11-05T08:33:39.309 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.6152e-05 2023-11-05T08:33:39.309 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.309 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.309 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:39.309 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297583+0000", 2023-11-05T08:33:39.310 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.6676000000000001e-05 2023-11-05T08:33:39.310 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.310 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.310 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:39.310 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.301232+0000", 2023-11-05T08:33:39.311 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.003649209 2023-11-05T08:33:39.311 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.311 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.311 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for blocked object", 2023-11-05T08:33:39.311 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.301367+0000", 2023-11-05T08:33:39.312 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00013454600000000001 2023-11-05T08:33:39.312 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.312 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.312 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:39.312 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.321294+0000", 2023-11-05T08:33:39.313 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.019927261000000002 2023-11-05T08:33:39.313 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.313 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.313 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:39.313 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.321567+0000", 2023-11-05T08:33:39.313 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00027297299999999998 2023-11-05T08:33:39.314 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.314 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.314 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:39.314 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.321964+0000", 2023-11-05T08:33:39.314 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00039707099999999998 2023-11-05T08:33:39.315 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.315 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:39.315 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.315 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.315 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.315 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.14s5 56/48 ECSubWrite(tid=32, reqid=client.4691.0:374, at_version=56'19, trim_to=0'0, roll_forward_to=49'18))", 2023-11-05T08:33:39.316 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:09.318983+0000", 2023-11-05T08:33:39.316 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 29.845039335999999, 2023-11-05T08:33:39.316 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00432556, 2023-11-05T08:33:39.316 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:39.316 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:39.317 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:39.317 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.317 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:39.317 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.318983+0000", 2023-11-05T08:33:39.317 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.318 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.318 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.318 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:39.318 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.318983+0000", 2023-11-05T08:33:39.318 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.318 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.319 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.319 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:39.319 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.318983+0000", 2023-11-05T08:33:39.319 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.319 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.320 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.320 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:39.320 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.318983+0000", 2023-11-05T08:33:39.320 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.320 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.320 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.321 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:39.321 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.319029+0000", 2023-11-05T08:33:39.321 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.6675999999999998e-05 2023-11-05T08:33:39.321 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.321 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.322 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:39.322 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.319054+0000", 2023-11-05T08:33:39.322 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.4669e-05 2023-11-05T08:33:39.322 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.322 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.323 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:39.323 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.319246+0000", 2023-11-05T08:33:39.323 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00019165499999999999 2023-11-05T08:33:39.323 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.323 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.323 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:39.324 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.319278+0000", 2023-11-05T08:33:39.324 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.2744999999999999e-05 2023-11-05T08:33:39.324 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.324 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.324 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:39.325 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.323256+0000", 2023-11-05T08:33:39.325 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0039773589999999998 2023-11-05T08:33:39.325 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.325 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.325 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:39.325 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.323308+0000", 2023-11-05T08:33:39.326 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.2456000000000001e-05 2023-11-05T08:33:39.326 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.326 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:39.326 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.326 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.327 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.327 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es4 57/30 ECSubWrite(tid=60, reqid=client.4691.0:380, at_version=57'45, trim_to=0'0, roll_forward_to=49'43))", 2023-11-05T08:33:39.327 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:10.285514+0000", 2023-11-05T08:33:39.327 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 28.878508468, 2023-11-05T08:33:39.327 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.003439372, 2023-11-05T08:33:39.328 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:39.328 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:39.328 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:39.328 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.328 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:39.328 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285514+0000", 2023-11-05T08:33:39.329 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.329 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.329 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.329 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:39.329 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285514+0000", 2023-11-05T08:33:39.330 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.330 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.330 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.330 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:39.330 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285514+0000", 2023-11-05T08:33:39.331 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.9000000000000002e-07 2023-11-05T08:33:39.331 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.331 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.331 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:39.331 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285558+0000", 2023-11-05T08:33:39.332 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.4229e-05 2023-11-05T08:33:39.332 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.332 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.332 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:39.332 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285584+0000", 2023-11-05T08:33:39.333 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.5706000000000001e-05 2023-11-05T08:33:39.333 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.333 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.333 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:39.333 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285602+0000", 2023-11-05T08:33:39.334 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.7863000000000002e-05 2023-11-05T08:33:39.334 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.334 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.334 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:39.334 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285821+0000", 2023-11-05T08:33:39.335 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000219416 2023-11-05T08:33:39.335 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.335 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.335 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:39.335 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285862+0000", 2023-11-05T08:33:39.336 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.0923000000000001e-05 2023-11-05T08:33:39.336 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.336 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.336 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:39.336 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.288883+0000", 2023-11-05T08:33:39.336 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0030210430000000002 2023-11-05T08:33:39.337 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.337 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.337 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:39.337 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.288953+0000", 2023-11-05T08:33:39.337 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.9801999999999993e-05 2023-11-05T08:33:39.338 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.338 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:39.338 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.338 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.338 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.339 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.10s4 62/45 ECSubWrite(tid=219, reqid=client.4691.0:424, at_version=62'11, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:39.339 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:15.391016+0000", 2023-11-05T08:33:39.339 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 23.773005884, 2023-11-05T08:33:39.339 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0049263429999999997, 2023-11-05T08:33:39.339 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:39.340 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:39.340 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:39.340 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.340 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:39.340 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.391016+0000", 2023-11-05T08:33:39.341 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.341 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.341 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.341 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:39.341 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.391016+0000", 2023-11-05T08:33:39.341 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.342 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.342 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.342 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:39.342 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.391017+0000", 2023-11-05T08:33:39.342 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.7e-07 2023-11-05T08:33:39.343 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.343 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.343 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:39.343 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.391866+0000", 2023-11-05T08:33:39.343 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00084949500000000002 2023-11-05T08:33:39.344 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.344 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.344 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:39.344 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.391906+0000", 2023-11-05T08:33:39.344 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.0302000000000001e-05 2023-11-05T08:33:39.345 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.345 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.345 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:39.345 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.391932+0000", 2023-11-05T08:33:39.345 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.5439000000000002e-05 2023-11-05T08:33:39.345 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.346 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.346 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:39.346 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.392053+0000", 2023-11-05T08:33:39.346 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00012140400000000001 2023-11-05T08:33:39.346 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.347 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.347 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:39.347 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.392067+0000", 2023-11-05T08:33:39.347 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4051e-05 2023-11-05T08:33:39.347 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.348 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.348 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:39.348 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.395819+0000", 2023-11-05T08:33:39.348 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0037515489999999999 2023-11-05T08:33:39.348 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.349 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.349 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:39.349 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.395943+0000", 2023-11-05T08:33:39.349 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000123733 2023-11-05T08:33:39.349 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.350 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:39.350 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.350 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.350 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.350 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.10s4 62/45 ECSubWrite(tid=220, reqid=client.4691.0:425, at_version=62'12, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:39.351 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:15.397304+0000", 2023-11-05T08:33:39.351 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 23.766717686, 2023-11-05T08:33:39.351 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0042497419999999999, 2023-11-05T08:33:39.351 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:39.351 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:39.352 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:39.352 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.352 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:39.352 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.397304+0000", 2023-11-05T08:33:39.352 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.352 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.353 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.353 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:39.353 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.397304+0000", 2023-11-05T08:33:39.353 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.353 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.354 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.354 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:39.354 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.397306+0000", 2023-11-05T08:33:39.354 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.5290000000000001e-06 2023-11-05T08:33:39.354 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.355 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.355 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:39.355 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.397686+0000", 2023-11-05T08:33:39.355 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00038012700000000001 2023-11-05T08:33:39.355 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.356 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.356 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:39.356 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.397726+0000", 2023-11-05T08:33:39.356 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.9842000000000002e-05 2023-11-05T08:33:39.356 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.357 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.357 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:39.357 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.397748+0000", 2023-11-05T08:33:39.357 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.2079e-05 2023-11-05T08:33:39.357 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.358 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.358 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:39.358 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.397923+0000", 2023-11-05T08:33:39.358 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000174637 2023-11-05T08:33:39.358 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.359 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.359 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:39.359 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.397959+0000", 2023-11-05T08:33:39.359 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.6885999999999999e-05 2023-11-05T08:33:39.359 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.359 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.360 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:39.360 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.401476+0000", 2023-11-05T08:33:39.360 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0035168809999999999 2023-11-05T08:33:39.360 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.360 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.361 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:39.361 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.401554+0000", 2023-11-05T08:33:39.361 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.7761000000000003e-05 2023-11-05T08:33:39.361 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.361 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:39.362 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.362 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.362 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.362 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.10s4 62/45 ECSubWrite(tid=221, reqid=client.4691.0:426, at_version=62'13, trim_to=0'0, roll_forward_to=62'11))", 2023-11-05T08:33:39.362 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:15.398990+0000", 2023-11-05T08:33:39.363 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 23.765031769, 2023-11-05T08:33:39.363 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0037293809999999999, 2023-11-05T08:33:39.363 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:39.363 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:39.363 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:39.364 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.364 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:39.364 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.398990+0000", 2023-11-05T08:33:39.364 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.364 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.365 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.365 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:39.365 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.398990+0000", 2023-11-05T08:33:39.365 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.365 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.365 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.366 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:39.366 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.398999+0000", 2023-11-05T08:33:39.366 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.7660000000000004e-06 2023-11-05T08:33:39.366 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.366 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.367 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:39.367 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.399128+0000", 2023-11-05T08:33:39.367 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00012880699999999999 2023-11-05T08:33:39.367 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.367 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.367 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:39.368 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.399160+0000", 2023-11-05T08:33:39.368 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.1761999999999998e-05 2023-11-05T08:33:39.368 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.368 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.369 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:39.369 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.399191+0000", 2023-11-05T08:33:39.369 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.1053000000000002e-05 2023-11-05T08:33:39.369 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.369 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.369 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:39.370 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.400107+0000", 2023-11-05T08:33:39.370 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00091625699999999997 2023-11-05T08:33:39.370 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.370 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.371 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:39.371 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.400140+0000", 2023-11-05T08:33:39.371 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.3385e-05 2023-11-05T08:33:39.371 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.371 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.371 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:39.372 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.402645+0000", 2023-11-05T08:33:39.372 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0025045129999999999 2023-11-05T08:33:39.372 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.372 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.372 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:39.373 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.402720+0000", 2023-11-05T08:33:39.373 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.4838000000000004e-05 2023-11-05T08:33:39.373 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.373 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:39.373 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.374 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.374 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.374 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.10s4 62/45 ECSubWrite(tid=222, reqid=client.4691.0:427, at_version=62'14, trim_to=0'0, roll_forward_to=62'11))", 2023-11-05T08:33:39.374 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:15.399828+0000", 2023-11-05T08:33:39.374 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 23.764194543999999, 2023-11-05T08:33:39.375 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0036514389999999998, 2023-11-05T08:33:39.375 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:39.375 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:39.375 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:39.375 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.375 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:39.376 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.399828+0000", 2023-11-05T08:33:39.376 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.376 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.376 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.376 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:39.377 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.399828+0000", 2023-11-05T08:33:39.377 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.377 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.377 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.377 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:39.377 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.399829+0000", 2023-11-05T08:33:39.378 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4300000000000001e-06 2023-11-05T08:33:39.378 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.378 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.378 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:39.378 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.399871+0000", 2023-11-05T08:33:39.379 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.1848999999999999e-05 2023-11-05T08:33:39.379 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.379 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.379 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:39.379 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.399902+0000", 2023-11-05T08:33:39.380 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.0985999999999998e-05 2023-11-05T08:33:39.380 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.380 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.380 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:39.380 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.399928+0000", 2023-11-05T08:33:39.380 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.6426000000000001e-05 2023-11-05T08:33:39.381 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.381 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.381 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:39.381 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.401704+0000", 2023-11-05T08:33:39.381 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001775694 2023-11-05T08:33:39.382 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.382 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.382 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:39.382 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.401736+0000", 2023-11-05T08:33:39.382 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.2511000000000001e-05 2023-11-05T08:33:39.383 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.383 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.383 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:39.383 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.403426+0000", 2023-11-05T08:33:39.383 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0016898390000000001 2023-11-05T08:33:39.383 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.384 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.384 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:39.384 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.403479+0000", 2023-11-05T08:33:39.384 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.2704000000000002e-05 2023-11-05T08:33:39.384 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.385 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:39.385 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.385 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.385 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.385 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:573 3.19s0 3:99e86b02:::smithi09039252-36:head [] snapc 27=[7] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2023-11-05T08:33:39.386 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:29.488425+0000", 2023-11-05T08:33:39.386 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 9.6755970920000003, 2023-11-05T08:33:39.386 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.003519749, 2023-11-05T08:33:39.386 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:39.386 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:39.387 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:33:39.387 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:33:39.387 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:39.387 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 573 2023-11-05T08:33:39.387 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.388 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:39.388 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.388 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:39.388 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:29.488425+0000", 2023-11-05T08:33:39.388 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.389 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.389 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.389 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:39.389 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:29.488425+0000", 2023-11-05T08:33:39.389 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.390 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.390 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.390 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:39.390 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:29.488427+0000", 2023-11-05T08:33:39.391 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.4770000000000002e-06 2023-11-05T08:33:39.391 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.391 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.391 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:39.391 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:29.488438+0000", 2023-11-05T08:33:39.392 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0668e-05 2023-11-05T08:33:39.392 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.392 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.392 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:39.392 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:29.488464+0000", 2023-11-05T08:33:39.392 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.6380000000000002e-05 2023-11-05T08:33:39.393 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.393 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.393 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:39.393 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:29.488477+0000", 2023-11-05T08:33:39.393 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.2425999999999999e-05 2023-11-05T08:33:39.394 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.394 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.394 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:39.394 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:29.488579+0000", 2023-11-05T08:33:39.394 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000102174 2023-11-05T08:33:39.395 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.395 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.395 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "op must wait for map", 2023-11-05T08:33:39.395 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:29.488592+0000", 2023-11-05T08:33:39.395 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.3062e-05 2023-11-05T08:33:39.396 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.396 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.396 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:39.396 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:29.491866+0000", 2023-11-05T08:33:39.396 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0032740550000000001 2023-11-05T08:33:39.397 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.397 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.397 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:39.397 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:29.491945+0000", 2023-11-05T08:33:39.397 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.8506999999999999e-05 2023-11-05T08:33:39.397 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.398 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:39.398 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.398 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.398 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.398 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.fs4 77/37 ECSubWrite(tid=130, reqid=client.4691.0:579, at_version=77'44, trim_to=0'0, roll_forward_to=70'42))", 2023-11-05T08:33:39.399 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:30.540343+0000", 2023-11-05T08:33:39.399 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 8.6236792310000006, 2023-11-05T08:33:39.399 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0057579060000000001, 2023-11-05T08:33:39.399 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:39.399 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:39.400 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:39.400 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.400 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:39.400 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:30.540343+0000", 2023-11-05T08:33:39.400 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.400 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.401 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.401 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:39.401 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:30.540343+0000", 2023-11-05T08:33:39.401 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.401 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.402 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.402 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:39.402 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:30.540344+0000", 2023-11-05T08:33:39.402 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.5999999999999991e-07 2023-11-05T08:33:39.402 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.403 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.403 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:39.403 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:30.540957+0000", 2023-11-05T08:33:39.403 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00061291099999999995 2023-11-05T08:33:39.403 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.404 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.404 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:39.404 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:30.540985+0000", 2023-11-05T08:33:39.404 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.7999000000000001e-05 2023-11-05T08:33:39.404 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.405 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.405 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:39.405 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:30.541002+0000", 2023-11-05T08:33:39.405 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.7105999999999999e-05 2023-11-05T08:33:39.405 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.406 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.406 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:39.406 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:30.541176+0000", 2023-11-05T08:33:39.406 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00017453100000000001 2023-11-05T08:33:39.406 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.407 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.407 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:39.407 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:30.541198+0000", 2023-11-05T08:33:39.407 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.1773e-05 2023-11-05T08:33:39.407 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.407 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.408 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:39.408 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:30.546033+0000", 2023-11-05T08:33:39.408 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.004835092 2023-11-05T08:33:39.408 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.408 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.409 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:39.409 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:30.546101+0000", 2023-11-05T08:33:39.409 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.7533999999999994e-05 2023-11-05T08:33:39.409 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.409 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:39.409 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.410 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.410 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:39.410 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:33:39.411 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~44s 2023-11-05T08:33:39.412 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2023-11-05T08:33:39.413 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:39.408+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:39.513 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:33:39.511+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:33:39.597 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:33:39.597 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [], 2023-11-05T08:33:39.598 INFO:teuthology.orchestra.run.smithi002.stdout: "num_ops": 0 2023-11-05T08:33:39.598 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:33:39.599 INFO:tasks.rados.rados.0.smithi090.stdout:268: done (5 left) 2023-11-05T08:33:39.599 INFO:tasks.rados.rados.0.smithi090.stdout:269: done (4 left) 2023-11-05T08:33:39.599 INFO:tasks.rados.rados.0.smithi090.stdout:270: done (3 left) 2023-11-05T08:33:39.599 INFO:tasks.rados.rados.0.smithi090.stdout:271: done (2 left) 2023-11-05T08:33:39.600 INFO:tasks.rados.rados.0.smithi090.stdout:272: done (1 left) 2023-11-05T08:33:39.600 INFO:tasks.rados.rados.0.smithi090.stdout:273: copy_from oid 36 from oid 22 current snap is 27 2023-11-05T08:33:39.600 INFO:tasks.rados.rados.0.smithi090.stdout:274: snap_remove snap 26 2023-11-05T08:33:39.614 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2023-11-05T08:33:39.615 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:33:39.608+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:33:39.715 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:39.711+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:39.741 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:33:39.742 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [], 2023-11-05T08:33:39.742 INFO:teuthology.orchestra.run.smithi002.stdout: "complaint_time": 30, 2023-11-05T08:33:39.742 INFO:teuthology.orchestra.run.smithi002.stdout: "num_blocked_ops": 0 2023-11-05T08:33:39.742 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:33:39.756 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2023-11-05T08:33:39.815 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:39.811+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:39.886 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:33:39.887 INFO:teuthology.orchestra.run.smithi002.stdout: "size": 20, 2023-11-05T08:33:39.887 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 600, 2023-11-05T08:33:39.887 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [ 2023-11-05T08:33:39.887 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.888 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es6 41/30 ECSubWrite(tid=40, reqid=osd.2.0:40, at_version=41'27, trim_to=0'0, roll_forward_to=40'25))", 2023-11-05T08:33:39.888 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:54.120785+0000", 2023-11-05T08:33:39.888 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 45.762990371999997, 2023-11-05T08:33:39.888 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0057023600000000001, 2023-11-05T08:33:39.889 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:39.889 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:39.889 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:39.889 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.890 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:39.890 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120785+0000", 2023-11-05T08:33:39.890 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.890 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.890 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.891 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:39.891 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120785+0000", 2023-11-05T08:33:39.891 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.891 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.891 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.892 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:39.892 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120786+0000", 2023-11-05T08:33:39.892 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.0999999999999998e-07 2023-11-05T08:33:39.892 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.892 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.892 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:39.893 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120850+0000", 2023-11-05T08:33:39.893 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.4727000000000006e-05 2023-11-05T08:33:39.893 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.893 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.893 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:39.894 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120899+0000", 2023-11-05T08:33:39.894 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.8905e-05 2023-11-05T08:33:39.894 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.894 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.894 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:39.895 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120925+0000", 2023-11-05T08:33:39.895 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.5831999999999999e-05 2023-11-05T08:33:39.895 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.895 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.895 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:39.896 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.121107+0000", 2023-11-05T08:33:39.896 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00018210899999999999 2023-11-05T08:33:39.896 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.896 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.896 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:39.896 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.121136+0000", 2023-11-05T08:33:39.897 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.9116e-05 2023-11-05T08:33:39.897 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.897 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.897 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:39.897 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.126383+0000", 2023-11-05T08:33:39.898 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0052464579999999999 2023-11-05T08:33:39.898 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.898 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.898 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:39.898 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.126487+0000", 2023-11-05T08:33:39.898 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000104503 2023-11-05T08:33:39.899 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.899 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:39.899 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.899 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.899 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.900 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es6 41/30 ECSubWrite(tid=41, reqid=osd.2.0:40, at_version=0'0, trim_to=0'0, roll_forward_to=41'27))", 2023-11-05T08:33:39.900 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:54.150296+0000", 2023-11-05T08:33:39.900 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 45.733479035999999, 2023-11-05T08:33:39.900 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.002204733, 2023-11-05T08:33:39.900 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:39.901 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:39.901 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:39.901 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.901 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:39.901 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.150296+0000", 2023-11-05T08:33:39.902 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.902 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.902 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.902 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:39.903 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.150296+0000", 2023-11-05T08:33:39.903 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.903 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.903 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.903 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:39.904 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.150298+0000", 2023-11-05T08:33:39.904 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.9530000000000002e-06 2023-11-05T08:33:39.904 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.904 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.905 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:39.905 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.150330+0000", 2023-11-05T08:33:39.905 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.1483000000000003e-05 2023-11-05T08:33:39.905 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.905 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.906 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:39.906 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.150368+0000", 2023-11-05T08:33:39.906 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.8194000000000001e-05 2023-11-05T08:33:39.906 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.906 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.907 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:39.907 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.150395+0000", 2023-11-05T08:33:39.907 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.6806000000000001e-05 2023-11-05T08:33:39.907 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.907 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.908 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:39.908 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.150620+0000", 2023-11-05T08:33:39.908 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00022512399999999999 2023-11-05T08:33:39.908 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.908 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.908 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:39.909 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.150652+0000", 2023-11-05T08:33:39.909 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.1912000000000002e-05 2023-11-05T08:33:39.909 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.909 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.909 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:39.910 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.152424+0000", 2023-11-05T08:33:39.910 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0017722339999999999 2023-11-05T08:33:39.910 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.910 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.910 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:39.911 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.152501+0000", 2023-11-05T08:33:39.911 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.7027000000000006e-05 2023-11-05T08:33:39.911 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.911 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:39.911 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.912 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.912 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.912 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es6 42/30 ECSubWrite(tid=43, reqid=osd.2.0:43, at_version=42'29, trim_to=0'0, roll_forward_to=41'27))", 2023-11-05T08:33:39.912 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:55.137493+0000", 2023-11-05T08:33:39.912 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 44.746282448999999, 2023-11-05T08:33:39.913 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0031512160000000001, 2023-11-05T08:33:39.913 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:39.913 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:39.913 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:39.913 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.914 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:39.914 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.137493+0000", 2023-11-05T08:33:39.914 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.914 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.914 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.914 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:39.915 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.137493+0000", 2023-11-05T08:33:39.915 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.915 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.915 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.915 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:39.916 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.137493+0000", 2023-11-05T08:33:39.916 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.7999999999999995e-07 2023-11-05T08:33:39.916 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.916 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.916 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:39.917 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.137559+0000", 2023-11-05T08:33:39.917 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.6098000000000003e-05 2023-11-05T08:33:39.917 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.917 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.917 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:39.918 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.137594+0000", 2023-11-05T08:33:39.918 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.4082000000000003e-05 2023-11-05T08:33:39.918 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.918 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.918 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:39.919 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.137620+0000", 2023-11-05T08:33:39.919 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.6271999999999999e-05 2023-11-05T08:33:39.919 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.919 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.919 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:39.920 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.137815+0000", 2023-11-05T08:33:39.920 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000195229 2023-11-05T08:33:39.920 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.920 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.920 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:39.921 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.137848+0000", 2023-11-05T08:33:39.921 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.2611999999999999e-05 2023-11-05T08:33:39.921 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.921 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.921 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:39.922 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.140476+0000", 2023-11-05T08:33:39.922 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0026284860000000002 2023-11-05T08:33:39.922 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.922 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.923 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:39.923 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.140644+0000", 2023-11-05T08:33:39.923 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00016785699999999999 2023-11-05T08:33:39.923 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.923 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:39.923 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.924 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.924 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.924 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.18s6 46/45 ECSubWrite(tid=195, reqid=osd.13.0:195, at_version=46'19, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:39.924 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:59.244820+0000", 2023-11-05T08:33:39.924 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 40.638955142999997, 2023-11-05T08:33:39.925 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0041897130000000003, 2023-11-05T08:33:39.925 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:39.925 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:39.925 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:39.925 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.926 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:39.926 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.244820+0000", 2023-11-05T08:33:39.926 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.926 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.926 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.927 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:39.927 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.244820+0000", 2023-11-05T08:33:39.927 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.927 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.927 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.928 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:39.928 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.244829+0000", 2023-11-05T08:33:39.928 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.8559999999999996e-06 2023-11-05T08:33:39.928 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.928 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.928 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:39.929 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.244875+0000", 2023-11-05T08:33:39.929 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.5837999999999997e-05 2023-11-05T08:33:39.929 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.929 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.929 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:39.930 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.244903+0000", 2023-11-05T08:33:39.930 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.8123000000000002e-05 2023-11-05T08:33:39.930 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.930 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.930 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:39.931 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.244919+0000", 2023-11-05T08:33:39.931 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.5973000000000001e-05 2023-11-05T08:33:39.931 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.931 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.931 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:39.932 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.245039+0000", 2023-11-05T08:33:39.932 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000120289 2023-11-05T08:33:39.932 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.932 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.932 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:39.933 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.245054+0000", 2023-11-05T08:33:39.933 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.491e-05 2023-11-05T08:33:39.933 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.933 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.933 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:39.934 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.248956+0000", 2023-11-05T08:33:39.934 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0039018859999999998 2023-11-05T08:33:39.934 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.934 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.934 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:39.935 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.249010+0000", 2023-11-05T08:33:39.935 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.3838000000000002e-05 2023-11-05T08:33:39.935 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.935 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:39.936 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.936 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.936 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.936 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.18s6 51/45 ECSubWrite(tid=203, reqid=client.4691.0:335, at_version=0'0, trim_to=0'0, roll_forward_to=51'23))", 2023-11-05T08:33:39.936 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:04.255817+0000", 2023-11-05T08:33:39.937 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 35.627957985999998, 2023-11-05T08:33:39.937 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0018718389999999999, 2023-11-05T08:33:39.937 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:39.937 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:39.937 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:39.938 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.938 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:39.938 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.255817+0000", 2023-11-05T08:33:39.938 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.938 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.939 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.939 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:39.939 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.255817+0000", 2023-11-05T08:33:39.939 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.939 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.940 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.940 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:39.940 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.255818+0000", 2023-11-05T08:33:39.940 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.8999999999999995e-07 2023-11-05T08:33:39.940 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.941 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.941 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:39.941 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.255834+0000", 2023-11-05T08:33:39.941 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.6240999999999999e-05 2023-11-05T08:33:39.941 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.942 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.942 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:39.942 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.255851+0000", 2023-11-05T08:33:39.942 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.6427999999999999e-05 2023-11-05T08:33:39.942 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.943 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.943 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:39.943 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.255865+0000", 2023-11-05T08:33:39.943 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4290999999999999e-05 2023-11-05T08:33:39.944 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.944 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.944 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:39.944 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.256066+0000", 2023-11-05T08:33:39.944 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00020114199999999999 2023-11-05T08:33:39.944 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.945 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.945 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:39.945 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.256100+0000", 2023-11-05T08:33:39.945 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.4159000000000003e-05 2023-11-05T08:33:39.945 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.946 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.946 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:39.946 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.257640+0000", 2023-11-05T08:33:39.946 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001539913 2023-11-05T08:33:39.946 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.947 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.947 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:39.947 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.257689+0000", 2023-11-05T08:33:39.947 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.8875000000000002e-05 2023-11-05T08:33:39.947 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.948 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:39.948 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.948 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.948 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.948 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.18s6 52/45 ECSubWrite(tid=205, reqid=client.4691.0:351, at_version=52'24, trim_to=0'0, roll_forward_to=51'23))", 2023-11-05T08:33:39.949 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:05.281319+0000", 2023-11-05T08:33:39.949 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 34.602455712999998, 2023-11-05T08:33:39.949 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0018246200000000001, 2023-11-05T08:33:39.949 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:39.949 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:39.950 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:39.950 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.950 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:39.950 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.281319+0000", 2023-11-05T08:33:39.950 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.951 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.951 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.951 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:39.951 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.281319+0000", 2023-11-05T08:33:39.951 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.951 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.952 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.952 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:39.952 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.281320+0000", 2023-11-05T08:33:39.952 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.9500000000000002e-07 2023-11-05T08:33:39.952 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.953 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.953 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:39.953 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.281768+0000", 2023-11-05T08:33:39.953 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00044751799999999999 2023-11-05T08:33:39.953 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.954 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.954 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:39.954 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.281785+0000", 2023-11-05T08:33:39.954 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.6993000000000001e-05 2023-11-05T08:33:39.954 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.955 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.955 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:39.955 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.281798+0000", 2023-11-05T08:33:39.955 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.3733999999999999e-05 2023-11-05T08:33:39.955 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.956 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.956 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:39.956 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.281884+0000", 2023-11-05T08:33:39.956 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.5649999999999995e-05 2023-11-05T08:33:39.956 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.956 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.957 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:39.957 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.281899+0000", 2023-11-05T08:33:39.957 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.5258e-05 2023-11-05T08:33:39.957 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.957 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.958 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:39.958 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.283079+0000", 2023-11-05T08:33:39.958 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0011801559999999999 2023-11-05T08:33:39.958 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.959 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.959 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:39.959 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.283144+0000", 2023-11-05T08:33:39.959 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.4616000000000006e-05 2023-11-05T08:33:39.959 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.959 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:39.960 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.960 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.960 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.960 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.18s6 52/45 ECSubWrite(tid=206, reqid=client.4691.0:352, at_version=52'25, trim_to=0'0, roll_forward_to=51'23))", 2023-11-05T08:33:39.960 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:05.287173+0000", 2023-11-05T08:33:39.961 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 34.596602339999997, 2023-11-05T08:33:39.961 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0019672029999999998, 2023-11-05T08:33:39.961 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:39.962 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:39.962 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:39.962 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.962 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:39.962 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.287173+0000", 2023-11-05T08:33:39.963 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.963 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.963 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.963 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:39.963 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.287173+0000", 2023-11-05T08:33:39.964 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.964 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.964 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.964 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:39.965 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.287174+0000", 2023-11-05T08:33:39.965 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.1200000000000002e-07 2023-11-05T08:33:39.965 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.965 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.965 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:39.966 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.287701+0000", 2023-11-05T08:33:39.966 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00052763599999999995 2023-11-05T08:33:39.966 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.966 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.966 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:39.967 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.287719+0000", 2023-11-05T08:33:39.967 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.7937000000000001e-05 2023-11-05T08:33:39.967 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.967 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.967 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:39.968 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.287732+0000", 2023-11-05T08:33:39.968 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.2289999999999999e-05 2023-11-05T08:33:39.968 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.968 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.969 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:39.969 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.287801+0000", 2023-11-05T08:33:39.969 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.9029000000000006e-05 2023-11-05T08:33:39.969 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.969 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.970 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:39.970 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.287809+0000", 2023-11-05T08:33:39.970 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.4530000000000002e-06 2023-11-05T08:33:39.970 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.971 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.971 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:39.971 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.289058+0000", 2023-11-05T08:33:39.971 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001248678 2023-11-05T08:33:39.971 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.972 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.972 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:39.972 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:05.289140+0000", 2023-11-05T08:33:39.972 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.2367999999999995e-05 2023-11-05T08:33:39.972 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.973 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:39.973 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.973 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.973 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.974 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es6 57/30 ECSubWrite(tid=60, reqid=client.4691.0:380, at_version=57'45, trim_to=0'0, roll_forward_to=49'43))", 2023-11-05T08:33:39.974 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:10.285649+0000", 2023-11-05T08:33:39.974 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 29.598125786000001, 2023-11-05T08:33:39.974 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0029323890000000001, 2023-11-05T08:33:39.974 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:39.975 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:39.975 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:39.975 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.975 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:39.975 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285649+0000", 2023-11-05T08:33:39.976 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.976 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.976 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.976 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:39.976 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285649+0000", 2023-11-05T08:33:39.977 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.977 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.977 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.977 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:39.977 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285650+0000", 2023-11-05T08:33:39.978 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.4300000000000003e-07 2023-11-05T08:33:39.978 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.978 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.978 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:39.978 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285727+0000", 2023-11-05T08:33:39.978 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.7088000000000005e-05 2023-11-05T08:33:39.979 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.979 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.979 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:39.979 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285762+0000", 2023-11-05T08:33:39.979 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.4808999999999998e-05 2023-11-05T08:33:39.980 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.980 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.980 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:39.980 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285788+0000", 2023-11-05T08:33:39.980 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.6075999999999999e-05 2023-11-05T08:33:39.981 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.981 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.981 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:39.981 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285958+0000", 2023-11-05T08:33:39.981 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00016993499999999999 2023-11-05T08:33:39.981 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.982 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.982 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:39.982 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285977+0000", 2023-11-05T08:33:39.982 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.8692999999999999e-05 2023-11-05T08:33:39.982 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.983 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.983 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:39.983 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.288484+0000", 2023-11-05T08:33:39.983 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.002507085 2023-11-05T08:33:39.983 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.984 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.984 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:39.984 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.288582+0000", 2023-11-05T08:33:39.984 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.8060000000000006e-05 2023-11-05T08:33:39.984 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.985 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:39.985 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.985 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.985 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.985 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es6 57/30 ECSubWrite(tid=61, reqid=client.4691.0:380, at_version=0'0, trim_to=0'0, roll_forward_to=57'45))", 2023-11-05T08:33:39.986 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:10.290231+0000", 2023-11-05T08:33:39.986 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 29.59354454, 2023-11-05T08:33:39.986 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0024684110000000002, 2023-11-05T08:33:39.986 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:39.986 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:39.986 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:39.987 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.987 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:39.987 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.290231+0000", 2023-11-05T08:33:39.987 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.987 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.988 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.988 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:39.988 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.290231+0000", 2023-11-05T08:33:39.988 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.988 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.989 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.989 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:39.989 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.290233+0000", 2023-11-05T08:33:39.989 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.9829999999999999e-06 2023-11-05T08:33:39.989 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.990 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.990 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:39.990 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.290260+0000", 2023-11-05T08:33:39.990 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.6889e-05 2023-11-05T08:33:39.990 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.991 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.991 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:39.991 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.290304+0000", 2023-11-05T08:33:39.991 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.4589000000000003e-05 2023-11-05T08:33:39.991 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.992 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.992 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:39.992 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.290327+0000", 2023-11-05T08:33:39.992 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.3302000000000002e-05 2023-11-05T08:33:39.992 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.992 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.993 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:39.993 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.290608+0000", 2023-11-05T08:33:39.993 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00028071599999999997 2023-11-05T08:33:39.993 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.993 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.994 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:39.994 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.290656+0000", 2023-11-05T08:33:39.994 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.8257999999999998e-05 2023-11-05T08:33:39.994 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.994 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.995 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:39.995 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.292606+0000", 2023-11-05T08:33:39.995 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0019496100000000001 2023-11-05T08:33:39.995 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.995 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.996 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:39.996 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.292699+0000", 2023-11-05T08:33:39.996 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.3064000000000002e-05 2023-11-05T08:33:39.996 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.996 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:39.997 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:39.997 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.997 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.997 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDScrubReserve(3.18s6 REQUEST e60)", 2023-11-05T08:33:39.997 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:13.953181+0000", 2023-11-05T08:33:39.997 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 25.930594374000002, 2023-11-05T08:33:39.998 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0021641680000000002, 2023-11-05T08:33:39.998 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:39.998 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "started", 2023-11-05T08:33:39.998 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:39.998 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.999 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:39.999 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:13.953181+0000", 2023-11-05T08:33:39.999 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:39.999 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:39.999 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:39.999 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:40.000 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:13.953181+0000", 2023-11-05T08:33:40.000 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:40.000 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.000 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.000 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:40.000 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:13.953181+0000", 2023-11-05T08:33:40.001 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.5000000000000003e-07 2023-11-05T08:33:40.001 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.001 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.001 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:40.002 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:13.953187+0000", 2023-11-05T08:33:40.002 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.1e-06 2023-11-05T08:33:40.002 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.002 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.002 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:40.003 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:13.953215+0000", 2023-11-05T08:33:40.003 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.7959000000000001e-05 2023-11-05T08:33:40.003 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.003 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.003 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:40.004 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:13.953233+0000", 2023-11-05T08:33:40.004 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.7402999999999999e-05 2023-11-05T08:33:40.004 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.004 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.004 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:40.005 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:13.953326+0000", 2023-11-05T08:33:40.005 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.3616999999999995e-05 2023-11-05T08:33:40.005 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.005 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.005 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "op must wait for map", 2023-11-05T08:33:40.006 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:13.953334+0000", 2023-11-05T08:33:40.006 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.131e-06 2023-11-05T08:33:40.006 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.006 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.006 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:40.007 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:13.955269+0000", 2023-11-05T08:33:40.007 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0019357179999999999 2023-11-05T08:33:40.007 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.007 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.007 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:33:40.007 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:13.955288+0000", 2023-11-05T08:33:40.008 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.8533e-05 2023-11-05T08:33:40.008 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.008 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.008 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:40.008 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:13.955345+0000", 2023-11-05T08:33:40.009 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.7157000000000002e-05 2023-11-05T08:33:40.009 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:40.009 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:40.009 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:40.009 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.010 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.010 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.18s6 62/45 ECSubWrite(tid=217, reqid=client.4691.0:423, at_version=62'30, trim_to=0'0, roll_forward_to=52'28))", 2023-11-05T08:33:40.010 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:15.372664+0000", 2023-11-05T08:33:40.010 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 24.511111174, 2023-11-05T08:33:40.010 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0040353610000000003, 2023-11-05T08:33:40.010 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:40.011 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:40.011 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:40.011 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.011 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:40.012 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.372664+0000", 2023-11-05T08:33:40.012 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:40.012 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.012 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.012 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:40.013 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.372664+0000", 2023-11-05T08:33:40.013 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:40.013 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.013 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.013 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:40.014 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.372664+0000", 2023-11-05T08:33:40.014 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.3099999999999999e-07 2023-11-05T08:33:40.014 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.014 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.014 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:40.015 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.372840+0000", 2023-11-05T08:33:40.015 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00017561599999999999 2023-11-05T08:33:40.015 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.015 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.016 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:40.016 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.372857+0000", 2023-11-05T08:33:40.016 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.7169000000000001e-05 2023-11-05T08:33:40.016 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.016 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.017 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:40.017 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.372870+0000", 2023-11-05T08:33:40.017 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.2847000000000001e-05 2023-11-05T08:33:40.017 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.018 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.018 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:40.018 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.373057+0000", 2023-11-05T08:33:40.018 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000186742 2023-11-05T08:33:40.018 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.019 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.019 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:40.019 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.373091+0000", 2023-11-05T08:33:40.019 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.4186000000000001e-05 2023-11-05T08:33:40.019 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.020 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.020 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:40.020 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.376614+0000", 2023-11-05T08:33:40.020 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0035234469999999999 2023-11-05T08:33:40.020 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.021 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.021 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:40.021 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.376699+0000", 2023-11-05T08:33:40.021 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.5023000000000002e-05 2023-11-05T08:33:40.021 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:40.021 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:40.022 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:40.022 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.022 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.022 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.bs3 65/37 ECSubWrite(tid=105, reqid=client.4691.0:488, at_version=65'21, trim_to=0'0, roll_forward_to=65'19))", 2023-11-05T08:33:40.023 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:18.605266+0000", 2023-11-05T08:33:40.023 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 21.278508797000001, 2023-11-05T08:33:40.023 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00218206, 2023-11-05T08:33:40.023 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:40.023 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:40.024 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:40.024 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.024 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:40.024 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.605266+0000", 2023-11-05T08:33:40.024 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:40.024 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.025 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.025 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:40.025 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.605266+0000", 2023-11-05T08:33:40.025 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:40.025 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.026 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.026 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:40.026 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.605268+0000", 2023-11-05T08:33:40.026 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.122e-06 2023-11-05T08:33:40.026 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.027 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.027 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:40.027 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.605361+0000", 2023-11-05T08:33:40.027 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.3486999999999994e-05 2023-11-05T08:33:40.027 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.028 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.028 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:40.028 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.605470+0000", 2023-11-05T08:33:40.028 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00010883999999999999 2023-11-05T08:33:40.028 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.029 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.029 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:40.029 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.605518+0000", 2023-11-05T08:33:40.029 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.7932000000000001e-05 2023-11-05T08:33:40.029 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.030 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.030 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:40.030 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.605673+0000", 2023-11-05T08:33:40.030 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00015474400000000001 2023-11-05T08:33:40.030 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.031 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.031 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:40.031 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.605697+0000", 2023-11-05T08:33:40.031 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.4054999999999999e-05 2023-11-05T08:33:40.031 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.032 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.032 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:40.032 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.607374+0000", 2023-11-05T08:33:40.032 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0016771220000000001 2023-11-05T08:33:40.032 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.033 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.033 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:40.033 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:18.607448+0000", 2023-11-05T08:33:40.033 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.4758000000000005e-05 2023-11-05T08:33:40.033 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:40.034 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:40.034 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:40.034 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.034 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.034 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.bs3 66/37 ECSubWrite(tid=108, reqid=client.4691.0:496, at_version=66'23, trim_to=0'0, roll_forward_to=65'21))", 2023-11-05T08:33:40.035 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:19.400396+0000", 2023-11-05T08:33:40.035 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 20.483379518, 2023-11-05T08:33:40.035 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0042162620000000001, 2023-11-05T08:33:40.035 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:40.035 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:40.036 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:40.036 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.036 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:40.036 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.400396+0000", 2023-11-05T08:33:40.036 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:40.037 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.037 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.037 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:40.037 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.400396+0000", 2023-11-05T08:33:40.037 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:40.037 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.038 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.038 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:40.038 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.400397+0000", 2023-11-05T08:33:40.038 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.8700000000000004e-07 2023-11-05T08:33:40.038 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.039 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.039 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:40.039 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.400694+0000", 2023-11-05T08:33:40.039 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00029789299999999998 2023-11-05T08:33:40.039 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.040 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.040 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:40.040 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.400711+0000", 2023-11-05T08:33:40.040 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.6909999999999999e-05 2023-11-05T08:33:40.040 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.041 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.041 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:40.041 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.400722+0000", 2023-11-05T08:33:40.041 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0892e-05 2023-11-05T08:33:40.041 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.042 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.042 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:40.042 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.400819+0000", 2023-11-05T08:33:40.042 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.7171999999999997e-05 2023-11-05T08:33:40.042 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.043 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.043 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:40.043 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.400830+0000", 2023-11-05T08:33:40.043 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0825999999999999e-05 2023-11-05T08:33:40.043 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.043 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.044 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:40.044 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.404534+0000", 2023-11-05T08:33:40.044 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0037035570000000001 2023-11-05T08:33:40.044 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.045 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.045 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:40.045 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.404612+0000", 2023-11-05T08:33:40.045 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.8125000000000002e-05 2023-11-05T08:33:40.045 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:40.046 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:40.046 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:40.046 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.046 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.046 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.bs3 66/37 ECSubWrite(tid=109, reqid=client.4691.0:497, at_version=66'24, trim_to=0'0, roll_forward_to=65'21))", 2023-11-05T08:33:40.047 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:19.404473+0000", 2023-11-05T08:33:40.047 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 20.479302673999999, 2023-11-05T08:33:40.047 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0021785720000000001, 2023-11-05T08:33:40.047 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:40.047 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:40.048 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:40.048 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.048 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:40.048 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.404473+0000", 2023-11-05T08:33:40.048 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:40.049 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.049 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.049 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:40.049 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.404473+0000", 2023-11-05T08:33:40.049 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:40.050 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.050 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.050 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:40.050 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.404474+0000", 2023-11-05T08:33:40.050 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4410000000000001e-06 2023-11-05T08:33:40.051 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.051 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.051 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:40.051 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.404694+0000", 2023-11-05T08:33:40.051 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000219574 2023-11-05T08:33:40.052 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.052 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.052 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:40.052 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.404707+0000", 2023-11-05T08:33:40.052 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.3433e-05 2023-11-05T08:33:40.053 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.053 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.053 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:40.053 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.404717+0000", 2023-11-05T08:33:40.053 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0047000000000001e-05 2023-11-05T08:33:40.054 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.054 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.054 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:40.054 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.404817+0000", 2023-11-05T08:33:40.054 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.9486000000000003e-05 2023-11-05T08:33:40.055 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.055 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.055 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:40.055 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.404830+0000", 2023-11-05T08:33:40.055 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.3474e-05 2023-11-05T08:33:40.056 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.056 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.056 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:40.056 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.406599+0000", 2023-11-05T08:33:40.056 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0017685769999999999 2023-11-05T08:33:40.057 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.057 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.057 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:40.057 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.406651+0000", 2023-11-05T08:33:40.057 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.2540000000000002e-05 2023-11-05T08:33:40.057 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:40.058 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:40.058 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:40.058 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.058 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.058 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es6 66/30 ECSubWrite(tid=87, reqid=osd.2.0:87, at_version=66'65, trim_to=0'0, roll_forward_to=65'63))", 2023-11-05T08:33:40.059 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:19.412990+0000", 2023-11-05T08:33:40.059 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 20.470784924, 2023-11-05T08:33:40.059 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001938132, 2023-11-05T08:33:40.059 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:40.059 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:40.060 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:40.060 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.060 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:40.060 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.412990+0000", 2023-11-05T08:33:40.060 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:40.061 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.061 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.061 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:40.061 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.412990+0000", 2023-11-05T08:33:40.061 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:40.062 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.062 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.062 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:40.062 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.412990+0000", 2023-11-05T08:33:40.062 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.8e-07 2023-11-05T08:33:40.063 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.063 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.063 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:40.063 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.413014+0000", 2023-11-05T08:33:40.063 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.3339000000000001e-05 2023-11-05T08:33:40.064 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.064 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.064 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:40.064 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.413030+0000", 2023-11-05T08:33:40.064 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.5945000000000001e-05 2023-11-05T08:33:40.065 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.065 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.065 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:40.065 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.413043+0000", 2023-11-05T08:33:40.065 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.3199e-05 2023-11-05T08:33:40.066 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.066 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.066 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:40.066 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.413155+0000", 2023-11-05T08:33:40.066 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000111766 2023-11-05T08:33:40.067 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.067 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.067 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:40.067 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.413173+0000", 2023-11-05T08:33:40.067 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.7966999999999999e-05 2023-11-05T08:33:40.068 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.068 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.068 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:40.068 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.414854+0000", 2023-11-05T08:33:40.069 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001681487 2023-11-05T08:33:40.069 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.069 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.069 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:40.069 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.414928+0000", 2023-11-05T08:33:40.070 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.4249e-05 2023-11-05T08:33:40.070 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:40.070 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:40.070 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:40.071 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.071 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.071 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es6 66/30 ECSubWrite(tid=88, reqid=osd.2.0:88, at_version=66'67, trim_to=0'0, roll_forward_to=65'63))", 2023-11-05T08:33:40.071 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:19.416196+0000", 2023-11-05T08:33:40.071 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 20.467578854999999, 2023-11-05T08:33:40.072 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0032309859999999999, 2023-11-05T08:33:40.072 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:40.072 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:40.072 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:40.072 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.073 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:40.073 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.416196+0000", 2023-11-05T08:33:40.073 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:40.073 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.073 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.073 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:40.074 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.416196+0000", 2023-11-05T08:33:40.074 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:40.074 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.074 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.075 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:40.075 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.416197+0000", 2023-11-05T08:33:40.075 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0079999999999999e-06 2023-11-05T08:33:40.075 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.075 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.076 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:40.076 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.416228+0000", 2023-11-05T08:33:40.076 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.0837000000000003e-05 2023-11-05T08:33:40.076 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.076 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.076 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:40.077 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.416246+0000", 2023-11-05T08:33:40.077 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.8289999999999999e-05 2023-11-05T08:33:40.077 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.077 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.077 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:40.078 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.416280+0000", 2023-11-05T08:33:40.078 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.3791000000000002e-05 2023-11-05T08:33:40.078 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.078 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.078 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:40.078 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.416373+0000", 2023-11-05T08:33:40.079 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.2650000000000002e-05 2023-11-05T08:33:40.079 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.079 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.079 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:40.079 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.416388+0000", 2023-11-05T08:33:40.079 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.471e-05 2023-11-05T08:33:40.080 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.080 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.080 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:40.080 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.419374+0000", 2023-11-05T08:33:40.080 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.002986388 2023-11-05T08:33:40.081 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.081 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.081 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:40.081 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.419427+0000", 2023-11-05T08:33:40.081 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.3312000000000001e-05 2023-11-05T08:33:40.082 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:40.082 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:40.082 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:40.082 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.082 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.082 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es6 66/30 ECSubWrite(tid=89, reqid=osd.2.0:88, at_version=0'0, trim_to=0'0, roll_forward_to=66'67))", 2023-11-05T08:33:40.083 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:19.426569+0000", 2023-11-05T08:33:40.083 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 20.457206652, 2023-11-05T08:33:40.083 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.002038223, 2023-11-05T08:33:40.083 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:40.083 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:40.084 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:40.084 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.084 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:40.084 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.426569+0000", 2023-11-05T08:33:40.084 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:40.085 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.085 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.085 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:40.085 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.426569+0000", 2023-11-05T08:33:40.085 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:40.086 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.086 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.086 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:40.086 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.426570+0000", 2023-11-05T08:33:40.086 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.249e-06 2023-11-05T08:33:40.087 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.087 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.087 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:40.087 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.426585+0000", 2023-11-05T08:33:40.087 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.5099000000000001e-05 2023-11-05T08:33:40.088 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.088 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.088 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:40.088 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.426648+0000", 2023-11-05T08:33:40.088 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.2908000000000001e-05 2023-11-05T08:33:40.088 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.089 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.089 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:40.089 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.426663+0000", 2023-11-05T08:33:40.089 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4718999999999999e-05 2023-11-05T08:33:40.089 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.089 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.090 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:40.090 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.426748+0000", 2023-11-05T08:33:40.090 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.5960999999999999e-05 2023-11-05T08:33:40.090 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.090 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.091 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:40.091 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.426762+0000", 2023-11-05T08:33:40.091 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.3141e-05 2023-11-05T08:33:40.091 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.091 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.092 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:40.092 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.428548+0000", 2023-11-05T08:33:40.092 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001786274 2023-11-05T08:33:40.092 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.092 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.092 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:40.093 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.428607+0000", 2023-11-05T08:33:40.093 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.8872000000000001e-05 2023-11-05T08:33:40.093 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:40.093 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:40.093 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:40.094 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.094 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.094 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.18s6 66/45 ECSubWrite(tid=252, reqid=osd.13.0:250, at_version=0'0, trim_to=0'0, roll_forward_to=66'38))", 2023-11-05T08:33:40.094 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:19.429793+0000", 2023-11-05T08:33:40.094 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 20.453982412999999, 2023-11-05T08:33:40.095 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0018166860000000001, 2023-11-05T08:33:40.095 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:40.095 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:40.095 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:40.095 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.096 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:40.096 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.429793+0000", 2023-11-05T08:33:40.096 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:40.096 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.096 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.096 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:40.097 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.429793+0000", 2023-11-05T08:33:40.097 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:40.097 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.097 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.097 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:40.098 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.429794+0000", 2023-11-05T08:33:40.098 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.3399999999999998e-07 2023-11-05T08:33:40.098 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.098 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.098 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:40.099 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.429813+0000", 2023-11-05T08:33:40.099 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.9293999999999999e-05 2023-11-05T08:33:40.099 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.099 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.099 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:40.100 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.429829+0000", 2023-11-05T08:33:40.100 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.6130999999999999e-05 2023-11-05T08:33:40.100 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.100 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.100 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:40.101 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.429841+0000", 2023-11-05T08:33:40.101 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.1588e-05 2023-11-05T08:33:40.101 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.101 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.101 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:40.102 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.429962+0000", 2023-11-05T08:33:40.102 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00012110800000000001 2023-11-05T08:33:40.102 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.102 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.102 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:40.103 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.429980+0000", 2023-11-05T08:33:40.103 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.8029e-05 2023-11-05T08:33:40.103 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.103 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.103 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:40.104 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.431570+0000", 2023-11-05T08:33:40.104 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0015904339999999999 2023-11-05T08:33:40.104 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.104 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.104 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:40.105 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.431609+0000", 2023-11-05T08:33:40.105 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.9267999999999997e-05 2023-11-05T08:33:40.105 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:40.105 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:40.105 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:40.106 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.106 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.106 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es6 66/30 ECSubWrite(tid=91, reqid=osd.2.0:90, at_version=0'0, trim_to=0'0, roll_forward_to=66'69))", 2023-11-05T08:33:40.106 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:19.434454+0000", 2023-11-05T08:33:40.106 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 20.449321571999999, 2023-11-05T08:33:40.107 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001854191, 2023-11-05T08:33:40.107 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:40.107 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:40.107 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:40.107 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.107 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:40.108 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.434454+0000", 2023-11-05T08:33:40.108 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:40.108 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.108 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.108 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:40.109 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.434454+0000", 2023-11-05T08:33:40.109 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:40.109 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.109 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.109 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:40.110 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.434455+0000", 2023-11-05T08:33:40.110 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.3850000000000001e-06 2023-11-05T08:33:40.110 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.110 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.110 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:40.111 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.434471+0000", 2023-11-05T08:33:40.111 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.6319000000000001e-05 2023-11-05T08:33:40.111 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.111 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.111 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:40.112 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.434502+0000", 2023-11-05T08:33:40.112 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.0473000000000001e-05 2023-11-05T08:33:40.112 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.112 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.112 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:40.113 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.434519+0000", 2023-11-05T08:33:40.113 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.6963999999999999e-05 2023-11-05T08:33:40.113 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.113 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.113 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:40.114 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.434694+0000", 2023-11-05T08:33:40.114 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00017542 2023-11-05T08:33:40.114 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.114 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.114 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:40.114 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.434719+0000", 2023-11-05T08:33:40.115 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.4638e-05 2023-11-05T08:33:40.115 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.115 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.115 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:40.115 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.436243+0000", 2023-11-05T08:33:40.116 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0015242999999999999 2023-11-05T08:33:40.116 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.116 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.116 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:40.116 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.436308+0000", 2023-11-05T08:33:40.116 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.4691999999999997e-05 2023-11-05T08:33:40.117 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:40.117 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:40.117 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:40.117 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.117 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.118 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es6 67/30 ECSubWrite(tid=92, reqid=client.4691.0:505, at_version=67'71, trim_to=0'0, roll_forward_to=66'69))", 2023-11-05T08:33:40.118 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:20.386999+0000", 2023-11-05T08:33:40.118 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 19.49677608, 2023-11-05T08:33:40.118 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0018620780000000001, 2023-11-05T08:33:40.118 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:33:40.119 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:33:40.119 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:33:40.119 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.119 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:33:40.119 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:20.386999+0000", 2023-11-05T08:33:40.120 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:40.120 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.120 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.120 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:33:40.120 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:20.386999+0000", 2023-11-05T08:33:40.120 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:33:40.121 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.121 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.121 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:33:40.121 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:20.387000+0000", 2023-11-05T08:33:40.121 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.9999999999999998e-07 2023-11-05T08:33:40.122 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.122 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.122 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:33:40.122 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:20.387036+0000", 2023-11-05T08:33:40.123 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.6740999999999999e-05 2023-11-05T08:33:40.123 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.123 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.123 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:33:40.123 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:20.387064+0000", 2023-11-05T08:33:40.124 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.8042e-05 2023-11-05T08:33:40.124 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.124 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.124 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:33:40.125 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:20.387085+0000", 2023-11-05T08:33:40.125 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.0825000000000001e-05 2023-11-05T08:33:40.125 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.125 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.125 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:33:40.126 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:20.387208+0000", 2023-11-05T08:33:40.126 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00012258399999999999 2023-11-05T08:33:40.126 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.126 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.127 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:33:40.127 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:20.387222+0000", 2023-11-05T08:33:40.127 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.3879e-05 2023-11-05T08:33:40.127 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.127 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.128 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:33:40.128 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:20.388808+0000", 2023-11-05T08:33:40.128 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0015860029999999999 2023-11-05T08:33:40.128 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:33:40.128 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:33:40.129 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:33:40.129 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:20.388861+0000", 2023-11-05T08:33:40.129 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.3603999999999997e-05 2023-11-05T08:33:40.129 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:40.129 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:40.129 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:40.130 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:33:40.130 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:33:40.130 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:33:40.132 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-11-05T08:33:40.133 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:33:40.129+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:33:40.232 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:33:40.228+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:33:40.267 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:33:40.267 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2023-11-05T08:33:40.267 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 0 2023-11-05T08:33:40.267 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:33:40.281 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-11-05T08:33:40.333 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:33:40.329+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:33:40.434 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:33:40.429+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:33:40.450 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:33:40.450 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2023-11-05T08:33:40.450 INFO:teuthology.orchestra.run.smithi053.stdout: "complaint_time": 30, 2023-11-05T08:33:40.450 INFO:teuthology.orchestra.run.smithi053.stdout: "num_blocked_ops": 0 2023-11-05T08:33:40.451 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:33:40.464 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-11-05T08:33:40.535 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:33:40.529+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:33:40.586 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:33:40.586 INFO:teuthology.orchestra.run.smithi053.stdout: "size": 20, 2023-11-05T08:33:40.587 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 600, 2023-11-05T08:33:40.587 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2023-11-05T08:33:40.587 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.587 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es2 41/30 ECSubWrite(tid=40, reqid=osd.2.0:40, at_version=41'27, trim_to=0'0, roll_forward_to=40'25))", 2023-11-05T08:33:40.587 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.127197+0000", 2023-11-05T08:33:40.588 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 46.456091764, 2023-11-05T08:33:40.588 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.019700031, 2023-11-05T08:33:40.588 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:40.588 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:40.588 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:40.589 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.589 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:40.589 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127197+0000", 2023-11-05T08:33:40.589 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.589 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.590 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.590 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:40.590 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127197+0000", 2023-11-05T08:33:40.590 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.590 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.590 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.591 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:40.591 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127198+0000", 2023-11-05T08:33:40.591 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.0999999999999998e-07 2023-11-05T08:33:40.591 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.591 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.592 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:40.592 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127255+0000", 2023-11-05T08:33:40.592 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6762000000000003e-05 2023-11-05T08:33:40.592 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.592 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.593 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:40.593 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127291+0000", 2023-11-05T08:33:40.593 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5948999999999998e-05 2023-11-05T08:33:40.593 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.593 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.594 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:40.594 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127314+0000", 2023-11-05T08:33:40.594 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3686e-05 2023-11-05T08:33:40.594 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.594 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.595 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:40.595 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127534+0000", 2023-11-05T08:33:40.595 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00021922700000000001 2023-11-05T08:33:40.595 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.595 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.596 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:40.596 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127562+0000", 2023-11-05T08:33:40.596 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8099e-05 2023-11-05T08:33:40.596 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.596 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.597 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:40.597 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.146742+0000", 2023-11-05T08:33:40.597 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.019180135000000001 2023-11-05T08:33:40.597 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.597 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.598 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:40.598 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.146897+0000", 2023-11-05T08:33:40.598 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000155463 2023-11-05T08:33:40.598 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.598 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:40.599 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.599 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.599 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.599 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as2 41/33 ECSubWrite(tid=35, reqid=osd.7.0:35, at_version=41'12, trim_to=0'0, roll_forward_to=40'10))", 2023-11-05T08:33:40.599 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.130596+0000", 2023-11-05T08:33:40.600 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 46.452692540999998, 2023-11-05T08:33:40.600 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022182642999999998, 2023-11-05T08:33:40.600 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:40.600 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:40.600 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:40.601 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.601 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:40.601 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130596+0000", 2023-11-05T08:33:40.601 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.601 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.601 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.602 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:40.602 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130596+0000", 2023-11-05T08:33:40.602 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.602 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.602 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.603 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:40.603 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130597+0000", 2023-11-05T08:33:40.603 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.7600000000000006e-07 2023-11-05T08:33:40.603 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.603 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.604 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:40.604 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130660+0000", 2023-11-05T08:33:40.604 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3015000000000006e-05 2023-11-05T08:33:40.604 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.604 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.605 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:40.605 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130725+0000", 2023-11-05T08:33:40.605 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4988999999999997e-05 2023-11-05T08:33:40.605 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.605 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.606 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:40.606 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130750+0000", 2023-11-05T08:33:40.606 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4802000000000001e-05 2023-11-05T08:33:40.606 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.607 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.607 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:40.607 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.131037+0000", 2023-11-05T08:33:40.607 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00028702900000000001 2023-11-05T08:33:40.608 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.608 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.608 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:40.608 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.131064+0000", 2023-11-05T08:33:40.609 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7022e-05 2023-11-05T08:33:40.609 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.609 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.609 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:40.609 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.152706+0000", 2023-11-05T08:33:40.610 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.021641734999999999 2023-11-05T08:33:40.610 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.610 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.610 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:40.610 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.152779+0000", 2023-11-05T08:33:40.610 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.3075000000000001e-05 2023-11-05T08:33:40.611 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.611 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:40.611 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.611 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.611 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.612 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as2 41/33 ECSubWrite(tid=36, reqid=osd.7.0:36, at_version=41'14, trim_to=0'0, roll_forward_to=40'10))", 2023-11-05T08:33:40.612 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.130806+0000", 2023-11-05T08:33:40.612 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 46.452482660999998, 2023-11-05T08:33:40.612 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.035467152000000002, 2023-11-05T08:33:40.612 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:40.613 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:40.613 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:40.613 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.613 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:40.613 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130806+0000", 2023-11-05T08:33:40.613 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.614 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.614 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.614 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:40.614 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130806+0000", 2023-11-05T08:33:40.614 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.615 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.615 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.615 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:40.615 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130807+0000", 2023-11-05T08:33:40.615 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1000000000000001e-06 2023-11-05T08:33:40.616 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.616 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.616 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:40.616 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130852+0000", 2023-11-05T08:33:40.616 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4428999999999998e-05 2023-11-05T08:33:40.616 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.617 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.617 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:40.617 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130876+0000", 2023-11-05T08:33:40.617 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3975999999999999e-05 2023-11-05T08:33:40.617 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.618 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.618 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:40.618 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130896+0000", 2023-11-05T08:33:40.618 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9905999999999999e-05 2023-11-05T08:33:40.618 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.619 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.619 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:40.619 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.143384+0000", 2023-11-05T08:33:40.619 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012488727999999999 2023-11-05T08:33:40.619 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.620 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.620 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:40.620 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.143412+0000", 2023-11-05T08:33:40.620 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7988999999999999e-05 2023-11-05T08:33:40.620 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.620 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.621 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:40.621 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.166186+0000", 2023-11-05T08:33:40.621 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022773787 2023-11-05T08:33:40.621 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.621 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.622 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:40.622 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.166273+0000", 2023-11-05T08:33:40.622 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.7237000000000005e-05 2023-11-05T08:33:40.622 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.622 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:40.623 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.623 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.623 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.623 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es2 41/30 ECSubWrite(tid=41, reqid=osd.2.0:40, at_version=0'0, trim_to=0'0, roll_forward_to=41'27))", 2023-11-05T08:33:40.623 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.148654+0000", 2023-11-05T08:33:40.623 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 46.434635016999998, 2023-11-05T08:33:40.624 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017394370999999999, 2023-11-05T08:33:40.624 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:40.624 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:40.624 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:40.624 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.625 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:40.625 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148654+0000", 2023-11-05T08:33:40.625 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.625 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.625 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.626 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:40.626 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148654+0000", 2023-11-05T08:33:40.626 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.626 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.627 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.627 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:40.627 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148656+0000", 2023-11-05T08:33:40.627 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.677e-06 2023-11-05T08:33:40.627 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.628 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.628 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:40.628 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148683+0000", 2023-11-05T08:33:40.628 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7246e-05 2023-11-05T08:33:40.628 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.628 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.629 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:40.629 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148715+0000", 2023-11-05T08:33:40.629 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2518999999999998e-05 2023-11-05T08:33:40.629 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.630 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.630 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:40.630 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148742+0000", 2023-11-05T08:33:40.630 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6876000000000002e-05 2023-11-05T08:33:40.630 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.631 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.631 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:40.631 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148890+0000", 2023-11-05T08:33:40.631 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014771599999999999 2023-11-05T08:33:40.631 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.632 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.632 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:40.632 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148915+0000", 2023-11-05T08:33:40.632 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5099000000000001e-05 2023-11-05T08:33:40.632 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.633 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.633 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:40.633 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.165914+0000", 2023-11-05T08:33:40.633 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.016999018000000001 2023-11-05T08:33:40.633 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.634 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.634 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:40.634 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.166048+0000", 2023-11-05T08:33:40.634 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013422 2023-11-05T08:33:40.634 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.634 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:40.635 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.635 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.635 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.635 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.2s5 41/40 ECSubWrite(tid=187, reqid=client.4691.0:292, at_version=41'9, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:40.635 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.162642+0000", 2023-11-05T08:33:40.636 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 46.420646617999999, 2023-11-05T08:33:40.636 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010859549, 2023-11-05T08:33:40.636 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:40.636 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:40.636 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:40.637 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.637 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:40.637 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.162642+0000", 2023-11-05T08:33:40.637 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.637 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.637 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.638 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:40.638 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.162642+0000", 2023-11-05T08:33:40.638 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.638 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.638 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.639 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:40.639 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.162644+0000", 2023-11-05T08:33:40.639 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.823e-06 2023-11-05T08:33:40.639 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.639 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.640 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:40.640 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.162700+0000", 2023-11-05T08:33:40.640 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5863999999999999e-05 2023-11-05T08:33:40.640 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.640 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.641 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:40.641 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.162731+0000", 2023-11-05T08:33:40.641 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0731000000000001e-05 2023-11-05T08:33:40.641 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.641 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.642 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:40.642 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.162754+0000", 2023-11-05T08:33:40.642 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3234000000000002e-05 2023-11-05T08:33:40.642 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.642 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.643 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:40.643 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.162988+0000", 2023-11-05T08:33:40.643 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00023393399999999999 2023-11-05T08:33:40.643 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.643 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.643 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:40.644 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.163016+0000", 2023-11-05T08:33:40.644 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8309e-05 2023-11-05T08:33:40.644 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.644 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.644 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:40.645 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.173420+0000", 2023-11-05T08:33:40.645 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010403802 2023-11-05T08:33:40.645 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.645 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.645 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:40.646 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.173502+0000", 2023-11-05T08:33:40.646 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.1852000000000002e-05 2023-11-05T08:33:40.646 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.646 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:40.646 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.647 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.647 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.647 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as2 41/33 ECSubWrite(tid=37, reqid=osd.7.0:36, at_version=0'0, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:33:40.647 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.167511+0000", 2023-11-05T08:33:40.647 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 46.415778179999997, 2023-11-05T08:33:40.648 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011732691, 2023-11-05T08:33:40.648 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:40.648 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:40.648 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:40.648 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.649 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:40.649 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167511+0000", 2023-11-05T08:33:40.649 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.649 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.649 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.650 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:40.650 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167511+0000", 2023-11-05T08:33:40.650 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.650 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.650 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.651 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:40.651 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167513+0000", 2023-11-05T08:33:40.651 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.0829999999999998e-06 2023-11-05T08:33:40.651 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.651 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.652 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:40.652 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167540+0000", 2023-11-05T08:33:40.652 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7095999999999999e-05 2023-11-05T08:33:40.652 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.652 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.653 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:40.653 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167572+0000", 2023-11-05T08:33:40.653 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2462000000000002e-05 2023-11-05T08:33:40.653 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.653 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.654 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:40.654 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167594+0000", 2023-11-05T08:33:40.654 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1316999999999999e-05 2023-11-05T08:33:40.654 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.654 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.655 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:40.655 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167791+0000", 2023-11-05T08:33:40.655 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00019748199999999999 2023-11-05T08:33:40.655 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.655 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.655 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:40.656 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167815+0000", 2023-11-05T08:33:40.656 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3536999999999998e-05 2023-11-05T08:33:40.656 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.656 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.656 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:40.657 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.179154+0000", 2023-11-05T08:33:40.657 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011339373999999999 2023-11-05T08:33:40.657 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.657 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.657 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:40.658 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.179244+0000", 2023-11-05T08:33:40.658 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9339999999999995e-05 2023-11-05T08:33:40.658 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.658 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:40.658 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.659 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.659 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.659 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.2s5 41/40 ECSubWrite(tid=188, reqid=osd.13.0:188, at_version=41'11, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:40.659 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.171651+0000", 2023-11-05T08:33:40.659 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 46.411637593000002, 2023-11-05T08:33:40.660 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0076644390000000003, 2023-11-05T08:33:40.660 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:40.660 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:40.660 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:40.661 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.661 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:40.661 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171651+0000", 2023-11-05T08:33:40.661 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.661 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.661 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.662 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:40.662 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171651+0000", 2023-11-05T08:33:40.662 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.662 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.662 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.663 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:40.663 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171653+0000", 2023-11-05T08:33:40.663 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5039999999999999e-06 2023-11-05T08:33:40.663 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.663 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.664 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:40.664 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171700+0000", 2023-11-05T08:33:40.664 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.6989e-05 2023-11-05T08:33:40.664 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.665 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.665 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:40.665 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171731+0000", 2023-11-05T08:33:40.665 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1310999999999999e-05 2023-11-05T08:33:40.665 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.666 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.666 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:40.666 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171750+0000", 2023-11-05T08:33:40.666 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9188e-05 2023-11-05T08:33:40.666 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.667 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.667 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:40.667 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171900+0000", 2023-11-05T08:33:40.667 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00015010699999999999 2023-11-05T08:33:40.667 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.667 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.668 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:40.668 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171920+0000", 2023-11-05T08:33:40.668 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9626000000000001e-05 2023-11-05T08:33:40.668 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.668 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.669 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:40.669 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.179250+0000", 2023-11-05T08:33:40.669 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0073298900000000004 2023-11-05T08:33:40.669 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.669 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.670 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:40.670 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.179316+0000", 2023-11-05T08:33:40.670 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5823999999999999e-05 2023-11-05T08:33:40.670 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.670 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:40.671 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.671 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.671 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.671 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as2 42/33 ECSubWrite(tid=38, reqid=osd.7.0:38, at_version=42'16, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:33:40.671 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:55.121755+0000", 2023-11-05T08:33:40.672 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 45.461533592999999, 2023-11-05T08:33:40.672 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011609566, 2023-11-05T08:33:40.672 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:40.672 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:40.672 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:40.673 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.673 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:40.673 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.121755+0000", 2023-11-05T08:33:40.673 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.673 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.674 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.674 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:40.674 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.121755+0000", 2023-11-05T08:33:40.674 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.674 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.675 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.675 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:40.675 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.121756+0000", 2023-11-05T08:33:40.675 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.0299999999999998e-07 2023-11-05T08:33:40.675 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.675 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.676 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:40.676 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.121810+0000", 2023-11-05T08:33:40.676 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3918000000000001e-05 2023-11-05T08:33:40.676 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.676 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.677 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:40.677 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.121861+0000", 2023-11-05T08:33:40.677 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.0846000000000001e-05 2023-11-05T08:33:40.677 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.677 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.678 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:40.678 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.121889+0000", 2023-11-05T08:33:40.678 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8359000000000001e-05 2023-11-05T08:33:40.678 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.678 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.679 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:40.679 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.122158+0000", 2023-11-05T08:33:40.679 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00026898800000000001 2023-11-05T08:33:40.679 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.679 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.680 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:40.680 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.122192+0000", 2023-11-05T08:33:40.680 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3259000000000001e-05 2023-11-05T08:33:40.680 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.680 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.681 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:40.681 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.133269+0000", 2023-11-05T08:33:40.681 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011076979000000001 2023-11-05T08:33:40.681 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.681 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.681 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:40.682 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.133365+0000", 2023-11-05T08:33:40.682 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6414000000000002e-05 2023-11-05T08:33:40.682 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.682 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:40.682 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.683 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.683 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.683 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as2 42/33 ECSubWrite(tid=39, reqid=osd.7.0:39, at_version=42'18, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:33:40.683 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:55.124648+0000", 2023-11-05T08:33:40.683 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 45.458640733000003, 2023-11-05T08:33:40.684 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.025426339999999999, 2023-11-05T08:33:40.684 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:40.684 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:40.684 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:40.684 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.685 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:40.685 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.124648+0000", 2023-11-05T08:33:40.685 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.685 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.685 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.685 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:40.686 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.124648+0000", 2023-11-05T08:33:40.686 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.686 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.686 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.687 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:40.687 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.124650+0000", 2023-11-05T08:33:40.687 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.463e-06 2023-11-05T08:33:40.687 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.687 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.688 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:40.688 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.124710+0000", 2023-11-05T08:33:40.688 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0001999999999999e-05 2023-11-05T08:33:40.688 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.689 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.689 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:40.689 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.124739+0000", 2023-11-05T08:33:40.689 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9278999999999999e-05 2023-11-05T08:33:40.689 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.690 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.690 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:40.690 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.124763+0000", 2023-11-05T08:33:40.690 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4036000000000002e-05 2023-11-05T08:33:40.691 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.691 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.691 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:40.691 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.125122+0000", 2023-11-05T08:33:40.691 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00035926900000000001 2023-11-05T08:33:40.692 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.692 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.692 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:40.692 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.125149+0000", 2023-11-05T08:33:40.692 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7192999999999999e-05 2023-11-05T08:33:40.693 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.693 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.693 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:40.693 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.149969+0000", 2023-11-05T08:33:40.693 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.024819074 2023-11-05T08:33:40.694 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.694 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.694 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:40.694 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.150075+0000", 2023-11-05T08:33:40.694 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000106024 2023-11-05T08:33:40.695 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.695 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:40.695 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.695 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.695 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.696 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es2 42/30 ECSubWrite(tid=43, reqid=osd.2.0:43, at_version=42'29, trim_to=0'0, roll_forward_to=41'27))", 2023-11-05T08:33:40.696 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:55.134544+0000", 2023-11-05T08:33:40.696 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 45.448744812999998, 2023-11-05T08:33:40.696 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.02091324, 2023-11-05T08:33:40.696 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:40.697 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:40.697 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:40.697 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.697 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:40.697 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134544+0000", 2023-11-05T08:33:40.698 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.698 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.698 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.698 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:40.698 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134544+0000", 2023-11-05T08:33:40.699 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.699 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.699 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.699 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:40.699 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134545+0000", 2023-11-05T08:33:40.700 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3499999999999996e-07 2023-11-05T08:33:40.700 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.700 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.700 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:40.700 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134593+0000", 2023-11-05T08:33:40.700 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.8538999999999997e-05 2023-11-05T08:33:40.701 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.701 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.701 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:40.701 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134621+0000", 2023-11-05T08:33:40.702 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7939000000000001e-05 2023-11-05T08:33:40.702 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.702 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.702 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:40.702 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134641+0000", 2023-11-05T08:33:40.703 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9352999999999999e-05 2023-11-05T08:33:40.703 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.703 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.703 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:40.703 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134788+0000", 2023-11-05T08:33:40.704 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000147806 2023-11-05T08:33:40.704 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.704 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.704 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:40.704 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134850+0000", 2023-11-05T08:33:40.704 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1614999999999999e-05 2023-11-05T08:33:40.705 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.705 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.705 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:40.705 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.155400+0000", 2023-11-05T08:33:40.705 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.020550023000000001 2023-11-05T08:33:40.706 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.706 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.706 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:40.706 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.155457+0000", 2023-11-05T08:33:40.706 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7330000000000002e-05 2023-11-05T08:33:40.707 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.707 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:40.707 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.707 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.707 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.708 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.2s5 42/40 ECSubWrite(tid=192, reqid=osd.13.0:192, at_version=42'13, trim_to=0'0, roll_forward_to=41'11))", 2023-11-05T08:33:40.708 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:55.150316+0000", 2023-11-05T08:33:40.708 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 45.432973316999998, 2023-11-05T08:33:40.708 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0088926939999999996, 2023-11-05T08:33:40.709 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:40.709 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:40.709 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:40.709 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.709 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:40.710 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.150316+0000", 2023-11-05T08:33:40.710 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.710 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.710 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.710 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:40.711 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.150316+0000", 2023-11-05T08:33:40.711 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.711 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.711 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.711 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:40.712 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.150318+0000", 2023-11-05T08:33:40.712 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.0140000000000001e-06 2023-11-05T08:33:40.712 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.712 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.712 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:40.713 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.150377+0000", 2023-11-05T08:33:40.713 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9231000000000003e-05 2023-11-05T08:33:40.713 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.713 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.713 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:40.714 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.150411+0000", 2023-11-05T08:33:40.714 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4279000000000002e-05 2023-11-05T08:33:40.714 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.714 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.714 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:40.715 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.150435+0000", 2023-11-05T08:33:40.715 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4165999999999999e-05 2023-11-05T08:33:40.715 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.715 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.715 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:40.716 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.152989+0000", 2023-11-05T08:33:40.716 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0025534889999999999 2023-11-05T08:33:40.716 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.716 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.716 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:40.717 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.153035+0000", 2023-11-05T08:33:40.717 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.6078000000000002e-05 2023-11-05T08:33:40.717 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.717 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.717 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:40.718 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.159134+0000", 2023-11-05T08:33:40.718 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0060986069999999998 2023-11-05T08:33:40.718 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.718 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.718 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:40.719 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.159208+0000", 2023-11-05T08:33:40.719 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.483e-05 2023-11-05T08:33:40.719 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.719 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:40.719 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.720 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.720 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.720 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as2 42/33 ECSubWrite(tid=40, reqid=osd.7.0:39, at_version=0'0, trim_to=0'0, roll_forward_to=42'18))", 2023-11-05T08:33:40.720 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:55.151263+0000", 2023-11-05T08:33:40.720 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 45.432025883999998, 2023-11-05T08:33:40.721 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0080059020000000005, 2023-11-05T08:33:40.721 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:40.721 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:40.721 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:40.721 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.722 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:40.722 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.151263+0000", 2023-11-05T08:33:40.722 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.722 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.722 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.723 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:40.723 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.151263+0000", 2023-11-05T08:33:40.723 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.723 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.723 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.724 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:40.724 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.151265+0000", 2023-11-05T08:33:40.724 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9939999999999999e-06 2023-11-05T08:33:40.724 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.724 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.725 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:40.725 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.151295+0000", 2023-11-05T08:33:40.725 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0355e-05 2023-11-05T08:33:40.725 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.725 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.726 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:40.726 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.151328+0000", 2023-11-05T08:33:40.726 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2246000000000003e-05 2023-11-05T08:33:40.726 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.726 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.726 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:40.727 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.151359+0000", 2023-11-05T08:33:40.727 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1458999999999998e-05 2023-11-05T08:33:40.727 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.727 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.728 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:40.728 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.153854+0000", 2023-11-05T08:33:40.728 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002495115 2023-11-05T08:33:40.728 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.728 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.729 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:40.729 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.153879+0000", 2023-11-05T08:33:40.729 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5065000000000001e-05 2023-11-05T08:33:40.729 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.729 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.729 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:40.730 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.159215+0000", 2023-11-05T08:33:40.730 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0053359690000000003 2023-11-05T08:33:40.730 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.730 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.731 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:40.731 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.159269+0000", 2023-11-05T08:33:40.731 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3699000000000002e-05 2023-11-05T08:33:40.731 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.731 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:40.732 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.732 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.732 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.732 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as2 52/45 ECSubWrite(tid=45, reqid=client.4691.0:349, at_version=52'21, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:40.732 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:05.262984+0000", 2023-11-05T08:33:40.733 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 35.320305241, 2023-11-05T08:33:40.733 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0076861730000000001, 2023-11-05T08:33:40.733 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:40.733 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:40.733 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:40.734 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.734 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:40.734 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.262984+0000", 2023-11-05T08:33:40.734 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.734 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.735 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.735 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:40.735 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.262984+0000", 2023-11-05T08:33:40.735 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.735 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.736 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.736 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:40.736 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.262985+0000", 2023-11-05T08:33:40.736 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.8400000000000002e-07 2023-11-05T08:33:40.736 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.737 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.737 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:40.737 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.263110+0000", 2023-11-05T08:33:40.737 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012501999999999999 2023-11-05T08:33:40.737 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.738 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.738 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:40.738 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.263149+0000", 2023-11-05T08:33:40.738 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8899000000000001e-05 2023-11-05T08:33:40.738 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.739 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.739 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:40.739 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.263173+0000", 2023-11-05T08:33:40.739 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4243000000000002e-05 2023-11-05T08:33:40.739 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.740 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.740 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:40.740 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.263816+0000", 2023-11-05T08:33:40.740 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00064262900000000003 2023-11-05T08:33:40.740 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.741 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.741 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:40.741 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.263830+0000", 2023-11-05T08:33:40.741 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4676e-05 2023-11-05T08:33:40.741 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.741 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.742 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:40.742 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.270569+0000", 2023-11-05T08:33:40.742 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0067386939999999999 2023-11-05T08:33:40.742 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.743 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.743 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:40.743 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.270670+0000", 2023-11-05T08:33:40.743 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000101028 2023-11-05T08:33:40.743 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.744 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:40.744 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.744 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.744 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.744 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.bs2 66/37 ECSubWrite(tid=108, reqid=client.4691.0:496, at_version=66'23, trim_to=0'0, roll_forward_to=65'21))", 2023-11-05T08:33:40.745 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.398619+0000", 2023-11-05T08:33:40.745 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 21.184670338, 2023-11-05T08:33:40.745 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.016555743000000001, 2023-11-05T08:33:40.745 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:40.745 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:40.746 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:40.746 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.746 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:40.746 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.398619+0000", 2023-11-05T08:33:40.747 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.747 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.747 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.747 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:40.747 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.398619+0000", 2023-11-05T08:33:40.748 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.748 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.748 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.748 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:40.748 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.398629+0000", 2023-11-05T08:33:40.749 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.9529999999999994e-06 2023-11-05T08:33:40.749 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.749 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.749 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:40.750 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.399216+0000", 2023-11-05T08:33:40.750 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00058774000000000001 2023-11-05T08:33:40.750 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.750 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.750 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:40.751 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.399255+0000", 2023-11-05T08:33:40.751 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8902000000000001e-05 2023-11-05T08:33:40.751 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.751 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.751 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:40.751 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.399280+0000", 2023-11-05T08:33:40.752 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4827000000000001e-05 2023-11-05T08:33:40.752 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.752 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.752 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:40.753 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.399530+0000", 2023-11-05T08:33:40.753 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00024994500000000001 2023-11-05T08:33:40.753 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.753 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.753 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:40.754 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.399558+0000", 2023-11-05T08:33:40.754 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8370000000000001e-05 2023-11-05T08:33:40.754 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.754 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.755 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:40.755 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.415065+0000", 2023-11-05T08:33:40.755 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.015506565 2023-11-05T08:33:40.755 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.755 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.755 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:40.756 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.415174+0000", 2023-11-05T08:33:40.756 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000109441 2023-11-05T08:33:40.756 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.756 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:40.757 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.757 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.757 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.757 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.bs2 66/37 ECSubWrite(tid=109, reqid=client.4691.0:497, at_version=66'24, trim_to=0'0, roll_forward_to=65'21))", 2023-11-05T08:33:40.757 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.403242+0000", 2023-11-05T08:33:40.758 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 21.18004659, 2023-11-05T08:33:40.758 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.018077743, 2023-11-05T08:33:40.758 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:40.758 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:40.758 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:40.759 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.759 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:40.759 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.403242+0000", 2023-11-05T08:33:40.759 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.759 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.760 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.760 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:40.760 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.403242+0000", 2023-11-05T08:33:40.760 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.760 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.761 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.761 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:40.761 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.403245+0000", 2023-11-05T08:33:40.761 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6000000000000001e-06 2023-11-05T08:33:40.761 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.762 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.762 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:40.762 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.403863+0000", 2023-11-05T08:33:40.762 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000617955 2023-11-05T08:33:40.762 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.763 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.763 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:40.763 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.403904+0000", 2023-11-05T08:33:40.763 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1542999999999998e-05 2023-11-05T08:33:40.763 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.763 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.764 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:40.764 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.403931+0000", 2023-11-05T08:33:40.764 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6016e-05 2023-11-05T08:33:40.764 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.764 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.765 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:40.765 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.404129+0000", 2023-11-05T08:33:40.765 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000198397 2023-11-05T08:33:40.765 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.765 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.766 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:40.766 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.404161+0000", 2023-11-05T08:33:40.766 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2039000000000003e-05 2023-11-05T08:33:40.766 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.766 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.767 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:40.767 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.421200+0000", 2023-11-05T08:33:40.767 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017038604999999998 2023-11-05T08:33:40.767 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.767 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.768 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:40.768 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.421320+0000", 2023-11-05T08:33:40.768 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000120588 2023-11-05T08:33:40.768 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.768 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:40.769 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.769 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.769 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.769 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.bs2 66/37 ECSubWrite(tid=110, reqid=client.4691.0:498, at_version=66'25, trim_to=0'0, roll_forward_to=65'21))", 2023-11-05T08:33:40.769 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.408787+0000", 2023-11-05T08:33:40.769 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 21.174501582000001, 2023-11-05T08:33:40.770 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012658930000000001, 2023-11-05T08:33:40.770 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:40.770 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:40.770 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:40.770 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.771 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:40.771 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.408787+0000", 2023-11-05T08:33:40.771 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.771 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.771 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.772 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:40.772 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.408787+0000", 2023-11-05T08:33:40.772 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.772 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.772 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.772 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:40.773 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.408789+0000", 2023-11-05T08:33:40.773 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.0169999999999999e-06 2023-11-05T08:33:40.773 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.773 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.773 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:40.773 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.408927+0000", 2023-11-05T08:33:40.774 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000137811 2023-11-05T08:33:40.774 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.774 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.774 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:40.774 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.408961+0000", 2023-11-05T08:33:40.775 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4173e-05 2023-11-05T08:33:40.775 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.775 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.775 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:40.775 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.408985+0000", 2023-11-05T08:33:40.776 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3961999999999999e-05 2023-11-05T08:33:40.776 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.776 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.776 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:40.776 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.410719+0000", 2023-11-05T08:33:40.777 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0017334729999999999 2023-11-05T08:33:40.777 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.777 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.777 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:40.777 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.410744+0000", 2023-11-05T08:33:40.777 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4956999999999998e-05 2023-11-05T08:33:40.778 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.778 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.778 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:40.778 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.421379+0000", 2023-11-05T08:33:40.779 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010635627999999999 2023-11-05T08:33:40.779 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.779 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.779 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:40.779 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.421446+0000", 2023-11-05T08:33:40.779 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.6909000000000006e-05 2023-11-05T08:33:40.780 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.780 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:40.780 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.780 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.781 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.781 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.bs2 66/37 ECSubWrite(tid=111, reqid=client.4691.0:499, at_version=66'26, trim_to=0'0, roll_forward_to=65'21))", 2023-11-05T08:33:40.781 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.409025+0000", 2023-11-05T08:33:40.781 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 21.174264422, 2023-11-05T08:33:40.781 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012479762, 2023-11-05T08:33:40.782 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:40.782 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:40.782 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:40.782 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.782 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:40.783 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.409025+0000", 2023-11-05T08:33:40.783 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.783 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.783 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.783 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:40.784 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.409025+0000", 2023-11-05T08:33:40.784 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.784 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.784 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.784 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:40.785 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.409026+0000", 2023-11-05T08:33:40.785 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1599999999999999e-06 2023-11-05T08:33:40.785 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.785 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.785 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:40.786 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.409067+0000", 2023-11-05T08:33:40.786 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1189000000000002e-05 2023-11-05T08:33:40.786 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.786 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.786 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:40.787 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.409087+0000", 2023-11-05T08:33:40.787 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9905999999999999e-05 2023-11-05T08:33:40.787 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.787 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.787 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:40.788 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.409098+0000", 2023-11-05T08:33:40.788 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1643e-05 2023-11-05T08:33:40.788 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.788 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.788 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:40.789 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.412893+0000", 2023-11-05T08:33:40.789 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0037942829999999999 2023-11-05T08:33:40.789 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.789 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.789 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:40.790 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.412919+0000", 2023-11-05T08:33:40.790 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6083000000000001e-05 2023-11-05T08:33:40.790 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.790 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.790 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:40.791 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.421452+0000", 2023-11-05T08:33:40.791 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0085328890000000001 2023-11-05T08:33:40.791 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.791 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.791 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:40.792 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.421504+0000", 2023-11-05T08:33:40.792 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2608999999999997e-05 2023-11-05T08:33:40.792 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.792 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:40.792 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.792 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.793 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.793 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es2 66/30 ECSubWrite(tid=87, reqid=osd.2.0:87, at_version=66'65, trim_to=0'0, roll_forward_to=65'63))", 2023-11-05T08:33:40.793 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.411058+0000", 2023-11-05T08:33:40.793 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 21.172230495000001, 2023-11-05T08:33:40.793 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010528946000000001, 2023-11-05T08:33:40.794 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:40.794 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:40.794 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:40.794 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.794 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:40.795 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.411058+0000", 2023-11-05T08:33:40.795 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.795 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.795 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.795 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:40.796 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.411058+0000", 2023-11-05T08:33:40.796 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.796 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.796 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.796 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:40.796 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.411059+0000", 2023-11-05T08:33:40.797 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5300000000000004e-07 2023-11-05T08:33:40.797 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.797 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.797 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:40.798 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.411112+0000", 2023-11-05T08:33:40.798 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2676000000000002e-05 2023-11-05T08:33:40.798 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.798 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.798 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:40.798 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.411142+0000", 2023-11-05T08:33:40.799 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9796000000000001e-05 2023-11-05T08:33:40.799 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.799 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.799 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:40.799 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.411164+0000", 2023-11-05T08:33:40.800 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.2249999999999999e-05 2023-11-05T08:33:40.800 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.800 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.800 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:40.800 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.411322+0000", 2023-11-05T08:33:40.801 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00015797700000000001 2023-11-05T08:33:40.801 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.801 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.801 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:40.801 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.411353+0000", 2023-11-05T08:33:40.801 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0682999999999997e-05 2023-11-05T08:33:40.802 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.802 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.802 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:40.802 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.421482+0000", 2023-11-05T08:33:40.803 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.01012939 2023-11-05T08:33:40.803 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.803 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.803 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:40.803 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.421587+0000", 2023-11-05T08:33:40.804 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010552099999999999 2023-11-05T08:33:40.804 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.804 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:40.804 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.805 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.805 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.805 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es2 66/30 ECSubWrite(tid=88, reqid=osd.2.0:88, at_version=66'67, trim_to=0'0, roll_forward_to=65'63))", 2023-11-05T08:33:40.805 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.414905+0000", 2023-11-05T08:33:40.805 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 21.168383717000001, 2023-11-05T08:33:40.806 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0083023899999999998, 2023-11-05T08:33:40.806 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:40.806 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:40.806 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:40.806 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.807 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:40.807 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.414905+0000", 2023-11-05T08:33:40.807 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.807 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.807 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.808 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:40.808 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.414905+0000", 2023-11-05T08:33:40.808 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.808 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.808 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.809 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:40.809 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.414907+0000", 2023-11-05T08:33:40.809 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.81e-06 2023-11-05T08:33:40.809 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.809 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.810 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:40.810 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.414960+0000", 2023-11-05T08:33:40.810 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3386e-05 2023-11-05T08:33:40.810 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.810 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.811 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:40.811 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.414992+0000", 2023-11-05T08:33:40.811 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1115999999999998e-05 2023-11-05T08:33:40.811 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.811 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.812 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:40.812 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.415012+0000", 2023-11-05T08:33:40.812 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.0843e-05 2023-11-05T08:33:40.812 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.812 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.813 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:40.813 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.415228+0000", 2023-11-05T08:33:40.813 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00021531600000000001 2023-11-05T08:33:40.813 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.813 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.814 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:40.814 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.415259+0000", 2023-11-05T08:33:40.814 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1399000000000002e-05 2023-11-05T08:33:40.814 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.814 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.815 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:40.815 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.423080+0000", 2023-11-05T08:33:40.815 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0078204220000000005 2023-11-05T08:33:40.815 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.815 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.816 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:40.816 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.423208+0000", 2023-11-05T08:33:40.816 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012809799999999999 2023-11-05T08:33:40.816 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.816 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:40.817 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.817 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.817 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.817 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.19s6 86/45 ECSubWrite(tid=51, reqid=client.4691.0:635, at_version=86'33, trim_to=0'0, roll_forward_to=65'31))", 2023-11-05T08:33:40.817 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:39.619629+0000", 2023-11-05T08:33:40.818 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.96366001199999995, 2023-11-05T08:33:40.818 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0081971470000000001, 2023-11-05T08:33:40.818 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:40.818 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:40.819 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:40.819 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.819 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:40.819 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.619629+0000", 2023-11-05T08:33:40.819 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.819 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.820 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.820 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:40.820 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.619629+0000", 2023-11-05T08:33:40.820 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:40.820 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.821 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.821 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:40.821 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.619629+0000", 2023-11-05T08:33:40.821 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4299999999999998e-07 2023-11-05T08:33:40.821 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.822 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.822 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:40.822 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.619925+0000", 2023-11-05T08:33:40.822 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00029548399999999999 2023-11-05T08:33:40.823 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.823 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.823 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:40.823 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.619966+0000", 2023-11-05T08:33:40.823 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1142e-05 2023-11-05T08:33:40.823 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.824 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.824 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:40.824 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.619998+0000", 2023-11-05T08:33:40.824 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1576999999999999e-05 2023-11-05T08:33:40.825 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.825 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.825 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:40.825 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.620107+0000", 2023-11-05T08:33:40.825 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000109158 2023-11-05T08:33:40.825 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.826 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.826 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:40.826 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.620122+0000", 2023-11-05T08:33:40.826 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5594999999999999e-05 2023-11-05T08:33:40.826 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.827 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.827 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:40.827 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.627726+0000", 2023-11-05T08:33:40.827 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.007603233 2023-11-05T08:33:40.827 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:40.828 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:40.828 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:40.828 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.627826+0000", 2023-11-05T08:33:40.828 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000100515 2023-11-05T08:33:40.828 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.829 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:40.829 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.829 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:40.829 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:40.829 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:33:40.830 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-11-05T08:33:40.832 INFO:tasks.rados.rados.0.smithi090.stdout:274: done (2 left) 2023-11-05T08:33:40.833 INFO:tasks.rados.rados.0.smithi090.stdout:275: setattr oid 48 current snap is 27 2023-11-05T08:33:40.833 INFO:tasks.rados.rados.0.smithi090.stdout:273: finishing copy_from racing read to smithi09039252-36 2023-11-05T08:33:40.833 INFO:tasks.rados.rados.0.smithi090.stdout:273: finishing copy_from to smithi09039252-36 2023-11-05T08:33:40.833 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 36 v 19 (ObjNum 21 snap 0 seq_num 21) dirty exists 2023-11-05T08:33:40.834 INFO:tasks.rados.rados.0.smithi090.stdout:273: done (2 left) 2023-11-05T08:33:40.834 INFO:tasks.rados.rados.0.smithi090.stdout:276: snap_create 2023-11-05T08:33:40.834 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 48 v 49 (ObjNum 47 snap 0 seq_num 47) dirty exists 2023-11-05T08:33:40.834 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:33:40.831+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:33:40.835 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-11-05T08:33:40.911 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [3] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 7, 11, 15] 2023-11-05T08:33:40.911 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2023-11-05T08:33:40.912 INFO:tasks.ceph.osd.3:Restarting daemon 2023-11-05T08:33:40.912 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-11-05T08:33:40.915 INFO:tasks.ceph.osd.3:Started 2023-11-05T08:33:40.915 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-11-05T08:33:40.932 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:33:40.928+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:33:40.971 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:33:40.971 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2023-11-05T08:33:40.971 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 0 2023-11-05T08:33:40.972 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:33:40.985 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-11-05T08:33:41.032 INFO:teuthology.orchestra.run.smithi090.stderr:no valid command found; 10 closest matches: 2023-11-05T08:33:41.033 INFO:teuthology.orchestra.run.smithi090.stderr:0 2023-11-05T08:33:41.033 INFO:teuthology.orchestra.run.smithi090.stderr:1 2023-11-05T08:33:41.033 INFO:teuthology.orchestra.run.smithi090.stderr:2 2023-11-05T08:33:41.033 INFO:teuthology.orchestra.run.smithi090.stderr:abort 2023-11-05T08:33:41.034 INFO:teuthology.orchestra.run.smithi090.stderr:assert 2023-11-05T08:33:41.034 INFO:teuthology.orchestra.run.smithi090.stderr:config diff 2023-11-05T08:33:41.034 INFO:teuthology.orchestra.run.smithi090.stderr:config diff get 2023-11-05T08:33:41.034 INFO:teuthology.orchestra.run.smithi090.stderr:config get 2023-11-05T08:33:41.034 INFO:teuthology.orchestra.run.smithi090.stderr:config help [] 2023-11-05T08:33:41.035 INFO:teuthology.orchestra.run.smithi090.stderr:config set ... 2023-11-05T08:33:41.035 INFO:teuthology.orchestra.run.smithi090.stderr:admin_socket: invalid command 2023-11-05T08:33:41.035 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-05T08:33:41.036 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2023-11-05T08:33:41.070 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:33:41.068+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:33:41.074 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-05T08:33:41.134 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:33:41.128+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:33:41.146 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:33:41.146 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2023-11-05T08:33:41.147 INFO:teuthology.orchestra.run.smithi053.stdout: "complaint_time": 30, 2023-11-05T08:33:41.147 INFO:teuthology.orchestra.run.smithi053.stdout: "num_blocked_ops": 0 2023-11-05T08:33:41.147 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:33:41.160 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-11-05T08:33:41.199 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-11-05T08:33:41.219 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1: {} 2023-11-05T08:33:41.219 INFO:teuthology.orchestra.run.smithi002.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-11-05T08:33:41.239 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2: {} 2023-11-05T08:33:41.239 INFO:teuthology.orchestra.run.smithi002.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-11-05T08:33:41.240 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-11-05T08:33:41.250 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:41.248+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:41.257 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4: {} 2023-11-05T08:33:41.257 INFO:teuthology.orchestra.run.smithi002.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-11-05T08:33:41.276 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5: {} 2023-11-05T08:33:41.276 INFO:teuthology.orchestra.run.smithi002.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-11-05T08:33:41.298 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6: {} 2023-11-05T08:33:41.298 INFO:teuthology.orchestra.run.smithi002.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-11-05T08:33:41.320 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7: {} 2023-11-05T08:33:41.320 INFO:teuthology.orchestra.run.smithi002.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-11-05T08:33:41.334 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:41.334 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:41.336 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:33:41.333+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:33:41.339 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8: {} 2023-11-05T08:33:41.339 INFO:teuthology.orchestra.run.smithi002.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-11-05T08:33:41.349 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2023-11-05T08:33:41.359 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9: {} 2023-11-05T08:33:41.359 INFO:teuthology.orchestra.run.smithi002.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-11-05T08:33:41.380 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10: {} 2023-11-05T08:33:41.381 INFO:teuthology.orchestra.run.smithi002.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-11-05T08:33:41.402 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11: {} 2023-11-05T08:33:41.403 INFO:teuthology.orchestra.run.smithi002.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-11-05T08:33:41.424 INFO:teuthology.orchestra.run.smithi002.stdout:osd.12: {} 2023-11-05T08:33:41.424 INFO:teuthology.orchestra.run.smithi002.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-11-05T08:33:41.437 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:33:41.432+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:33:41.445 INFO:teuthology.orchestra.run.smithi002.stdout:osd.13: {} 2023-11-05T08:33:41.446 INFO:teuthology.orchestra.run.smithi002.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-11-05T08:33:41.469 INFO:teuthology.orchestra.run.smithi002.stdout:osd.14: {} 2023-11-05T08:33:41.470 INFO:teuthology.orchestra.run.smithi002.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-11-05T08:33:41.494 INFO:teuthology.orchestra.run.smithi002.stdout:osd.15: {} 2023-11-05T08:33:41.494 INFO:teuthology.orchestra.run.smithi002.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-11-05T08:33:41.498 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-05T08:33:41.499 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-05T08:33:41.518 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:41.518 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:41.532 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2023-11-05T08:33:41.538 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:33:41.533+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:33:41.545 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-11-05T08:33:41.543+0000 7f72b5ca2640 -1 Falling back to public interface 2023-11-05T08:33:41.613 INFO:teuthology.orchestra.run.smithi002.stderr:noscrub is set 2023-11-05T08:33:41.622 INFO:tasks.rados.rados.0.smithi090.stdout:275: done (2 left) 2023-11-05T08:33:41.623 INFO:tasks.rados.rados.0.smithi090.stdout:276: done (1 left) 2023-11-05T08:33:41.623 INFO:tasks.rados.rados.0.smithi090.stdout:277: read oid 10 snap 27 2023-11-05T08:33:41.623 INFO:tasks.rados.rados.0.smithi090.stdout:277: expect (ObjNum 9 snap 0 seq_num 9) 2023-11-05T08:33:41.624 INFO:tasks.rados.rados.0.smithi090.stdout:278: snap_remove snap 24 2023-11-05T08:33:41.638 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:33:41.633+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:33:41.701 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:41.702 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:41.715 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2023-11-05T08:33:41.766 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:33:41.764+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:33:41.840 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:33:41.837+0000 7fdc5cd58640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:33:41.840 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:41.840 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:41.856 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2023-11-05T08:33:41.940 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:33:41.937+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:33:42.027 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:42.027 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:42.041 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:33:42.037+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:33:42.042 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2023-11-05T08:33:42.142 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:33:42.138+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:33:42.171 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:42.172 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:42.186 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2023-11-05T08:33:42.242 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:42.240+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:42.315 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:42.315 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:42.330 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-11-05T08:33:42.380 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:33:42.377+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:33:42.445 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:42.442+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:42.459 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:42.460 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:42.473 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-11-05T08:33:42.546 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:42.540+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:42.631 INFO:tasks.rados.rados.0.smithi090.stdout:278: done (2 left) 2023-11-05T08:33:42.632 INFO:tasks.rados.rados.0.smithi090.stdout:279: snap_create 2023-11-05T08:33:42.632 INFO:tasks.rados.rados.0.smithi090.stdout:277: expect (ObjNum 9 snap 0 seq_num 9) 2023-11-05T08:33:42.637 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:42.637 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:42.647 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:33:42.641+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:33:42.651 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-11-05T08:33:42.748 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:33:42.745+0000 7fdc5cd58640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:33:42.823 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:42.823 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:42.837 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-11-05T08:33:42.849 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:33:42.846+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:33:42.950 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:33:42.945+0000 7fdc5cd58640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:33:42.956 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:42.956 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:42.970 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-11-05T08:33:43.051 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:33:43.045+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:33:43.087 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:43.087 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:43.101 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-11-05T08:33:43.152 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:33:43.149+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:33:43.224 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:43.224 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:43.239 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2023-11-05T08:33:43.252 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:33:43.253+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:33:43.353 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:33:43.350+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:33:43.399 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:43.399 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:43.413 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2023-11-05T08:33:43.454 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:43.450+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:43.500 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-05T08:33:43.528 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:43.529 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:43.543 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2023-11-05T08:33:43.555 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:33:43.550+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:33:43.642 INFO:tasks.rados.rados.0.smithi090.stdout:277: done (2 left) 2023-11-05T08:33:43.642 INFO:tasks.rados.rados.0.smithi090.stdout:279: done (1 left) 2023-11-05T08:33:43.642 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 30 current snap is 29 2023-11-05T08:33:43.643 INFO:tasks.rados.rados.0.smithi090.stdout:280: seq_num 109 ranges {3457024=671744,4128768=753664,4882432=409600} 2023-11-05T08:33:43.643 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-11-05T08:33:43.647 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-11-05T08:33:43.655 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:43.650+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:43.660 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:43.660 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:43.670 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1: {} 2023-11-05T08:33:43.670 INFO:teuthology.orchestra.run.smithi002.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-11-05T08:33:43.674 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2023-11-05T08:33:43.690 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2: {} 2023-11-05T08:33:43.691 INFO:teuthology.orchestra.run.smithi002.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-11-05T08:33:43.691 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-11-05T08:33:43.701 INFO:tasks.rados.rados.0.smithi090.stdout:280: writing smithi09039252-30 from 3457024 to 4128768 tid 1 2023-11-05T08:33:43.708 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4: {} 2023-11-05T08:33:43.708 INFO:teuthology.orchestra.run.smithi002.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-11-05T08:33:43.708 INFO:tasks.rados.rados.0.smithi090.stdout:280: writing smithi09039252-30 from 4128768 to 4882432 tid 2 2023-11-05T08:33:43.713 INFO:tasks.rados.rados.0.smithi090.stdout:280: writing smithi09039252-30 from 4882432 to 5292032 tid 3 2023-11-05T08:33:43.713 INFO:tasks.rados.rados.0.smithi090.stdout:281: read oid 32 snap -1 2023-11-05T08:33:43.713 INFO:tasks.rados.rados.0.smithi090.stdout:281: expect (ObjNum 31 snap 0 seq_num 31) 2023-11-05T08:33:43.714 INFO:tasks.rados.rados.0.smithi090.stdout:282: rollback oid 27 current snap is 29 2023-11-05T08:33:43.714 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 27 to 4 2023-11-05T08:33:43.714 INFO:tasks.rados.rados.0.smithi090.stdout:283: snap_remove snap 28 2023-11-05T08:33:43.732 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5: {} 2023-11-05T08:33:43.733 INFO:teuthology.orchestra.run.smithi002.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-11-05T08:33:43.751 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6: {} 2023-11-05T08:33:43.751 INFO:teuthology.orchestra.run.smithi002.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-11-05T08:33:43.756 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:43.752+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:43.773 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7: {} 2023-11-05T08:33:43.773 INFO:teuthology.orchestra.run.smithi002.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-11-05T08:33:43.789 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8: {} 2023-11-05T08:33:43.789 INFO:teuthology.orchestra.run.smithi002.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-11-05T08:33:43.792 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:43.792 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:43.806 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2023-11-05T08:33:43.809 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9: {} 2023-11-05T08:33:43.809 INFO:teuthology.orchestra.run.smithi002.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-11-05T08:33:43.826 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10: {} 2023-11-05T08:33:43.827 INFO:teuthology.orchestra.run.smithi002.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-11-05T08:33:43.846 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11: {} 2023-11-05T08:33:43.846 INFO:teuthology.orchestra.run.smithi002.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-11-05T08:33:43.857 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:33:43.852+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:33:43.871 INFO:teuthology.orchestra.run.smithi002.stdout:osd.12: {} 2023-11-05T08:33:43.871 INFO:teuthology.orchestra.run.smithi002.stderr:osd.12: osd_enable_op_tracker = 'true' 2023-11-05T08:33:43.891 INFO:teuthology.orchestra.run.smithi002.stdout:osd.13: {} 2023-11-05T08:33:43.891 INFO:teuthology.orchestra.run.smithi002.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-11-05T08:33:43.911 INFO:teuthology.orchestra.run.smithi002.stdout:osd.14: {} 2023-11-05T08:33:43.911 INFO:teuthology.orchestra.run.smithi002.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-11-05T08:33:43.926 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:33:43.926 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2023-11-05T08:33:43.927 INFO:teuthology.orchestra.run.smithi069.stdout: "complaint_time": 30, 2023-11-05T08:33:43.927 INFO:teuthology.orchestra.run.smithi069.stdout: "num_blocked_ops": 0 2023-11-05T08:33:43.927 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:33:43.931 INFO:teuthology.orchestra.run.smithi002.stdout:osd.15: {} 2023-11-05T08:33:43.932 INFO:teuthology.orchestra.run.smithi002.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-11-05T08:33:43.938 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-05T08:33:43.938 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-05T08:33:43.940 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2023-11-05T08:33:43.984 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:33:43.981+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:33:44.059 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:33:44.056+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:33:44.066 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:33:44.066 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 20, 2023-11-05T08:33:44.066 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 600, 2023-11-05T08:33:44.066 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [ 2023-11-05T08:33:44.067 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.067 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.ds2 36/30 ECSubWrite(tid=23, reqid=osd.7.0:23, at_version=36'14, trim_to=0'0, roll_forward_to=34'8))", 2023-11-05T08:33:44.067 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.078426+0000", 2023-11-05T08:33:44.067 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 54.983990417999998, 2023-11-05T08:33:44.067 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0071789940000000002, 2023-11-05T08:33:44.068 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:44.068 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:44.068 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:44.068 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.068 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:44.069 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078426+0000", 2023-11-05T08:33:44.069 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.069 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.069 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.069 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:44.070 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078426+0000", 2023-11-05T08:33:44.070 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.070 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.070 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.070 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:44.071 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078427+0000", 2023-11-05T08:33:44.071 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.093e-06 2023-11-05T08:33:44.071 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.071 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.071 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:44.072 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078484+0000", 2023-11-05T08:33:44.072 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.7240000000000001e-05 2023-11-05T08:33:44.072 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.072 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.072 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:44.073 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078508+0000", 2023-11-05T08:33:44.073 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4032e-05 2023-11-05T08:33:44.073 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.073 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.073 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.073 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078527+0000", 2023-11-05T08:33:44.074 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8754999999999999e-05 2023-11-05T08:33:44.074 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.074 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.074 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:44.074 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079037+0000", 2023-11-05T08:33:44.075 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000509722 2023-11-05T08:33:44.075 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.075 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.075 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:44.075 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079059+0000", 2023-11-05T08:33:44.076 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2467999999999999e-05 2023-11-05T08:33:44.076 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.076 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.076 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:44.076 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.085512+0000", 2023-11-05T08:33:44.076 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0064529059999999996 2023-11-05T08:33:44.077 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.077 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.077 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:44.077 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.085605+0000", 2023-11-05T08:33:44.077 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.2777999999999998e-05 2023-11-05T08:33:44.078 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.078 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:44.078 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.078 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.078 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.079 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.4s5 36/30 ECSubWrite(tid=10, reqid=osd.8.0:10, at_version=36'8, trim_to=0'0, roll_forward_to=34'6))", 2023-11-05T08:33:44.079 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.078716+0000", 2023-11-05T08:33:44.079 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 54.983700632999998, 2023-11-05T08:33:44.079 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.006931782, 2023-11-05T08:33:44.079 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:44.079 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:44.080 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:44.080 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.080 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:44.080 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078716+0000", 2023-11-05T08:33:44.080 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.081 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.081 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.081 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:44.081 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078716+0000", 2023-11-05T08:33:44.081 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.081 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.082 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.082 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:44.082 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078716+0000", 2023-11-05T08:33:44.082 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.0399999999999995e-07 2023-11-05T08:33:44.082 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.083 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.083 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:44.083 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078773+0000", 2023-11-05T08:33:44.083 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.6149000000000001e-05 2023-11-05T08:33:44.083 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.083 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.084 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:44.084 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078804+0000", 2023-11-05T08:33:44.084 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.1677999999999997e-05 2023-11-05T08:33:44.084 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.084 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.085 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.085 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.078827+0000", 2023-11-05T08:33:44.085 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2469000000000001e-05 2023-11-05T08:33:44.085 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.085 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.085 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:44.086 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079007+0000", 2023-11-05T08:33:44.086 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000180242 2023-11-05T08:33:44.086 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.086 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.086 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:44.087 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079032+0000", 2023-11-05T08:33:44.087 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4915000000000001e-05 2023-11-05T08:33:44.087 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.087 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.087 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:44.087 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.085566+0000", 2023-11-05T08:33:44.088 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0065336140000000001 2023-11-05T08:33:44.088 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.088 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.088 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:44.088 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.085648+0000", 2023-11-05T08:33:44.089 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.2010999999999998e-05 2023-11-05T08:33:44.089 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.089 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:44.089 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.089 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.089 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.090 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.8s4 36/30 ECSubWrite(tid=23, reqid=osd.5.0:23, at_version=36'15, trim_to=0'0, roll_forward_to=34'13))", 2023-11-05T08:33:44.090 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.079985+0000", 2023-11-05T08:33:44.090 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 54.982431618, 2023-11-05T08:33:44.090 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.007626023, 2023-11-05T08:33:44.090 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:44.091 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:44.091 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:44.091 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.091 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:44.091 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079985+0000", 2023-11-05T08:33:44.092 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.092 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.092 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.092 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:44.092 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079985+0000", 2023-11-05T08:33:44.092 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.093 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.093 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.093 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:44.093 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.079985+0000", 2023-11-05T08:33:44.093 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.6300000000000005e-07 2023-11-05T08:33:44.094 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.094 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.094 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:44.094 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.080040+0000", 2023-11-05T08:33:44.094 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.4796999999999997e-05 2023-11-05T08:33:44.095 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.095 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.095 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:44.095 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.080073+0000", 2023-11-05T08:33:44.095 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.3034999999999998e-05 2023-11-05T08:33:44.095 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.096 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.096 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.096 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.080095+0000", 2023-11-05T08:33:44.096 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1841999999999999e-05 2023-11-05T08:33:44.096 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.097 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.097 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:44.097 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.080255+0000", 2023-11-05T08:33:44.097 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00016012700000000001 2023-11-05T08:33:44.097 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.098 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.098 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:44.098 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.080280+0000", 2023-11-05T08:33:44.098 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4953999999999999e-05 2023-11-05T08:33:44.098 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.098 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.099 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:44.099 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.087517+0000", 2023-11-05T08:33:44.099 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0072370799999999999 2023-11-05T08:33:44.099 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.099 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.100 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:44.100 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.087611+0000", 2023-11-05T08:33:44.100 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.3524999999999996e-05 2023-11-05T08:33:44.100 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.100 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:44.101 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.101 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.101 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.101 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.ds2 36/30 ECSubWrite(tid=24, reqid=osd.7.0:23, at_version=0'0, trim_to=0'0, roll_forward_to=36'14))", 2023-11-05T08:33:44.101 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.089710+0000", 2023-11-05T08:33:44.101 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 54.972706287999998, 2023-11-05T08:33:44.102 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0067357069999999996, 2023-11-05T08:33:44.102 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:44.102 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:44.102 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:44.102 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.103 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:44.103 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089710+0000", 2023-11-05T08:33:44.103 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.103 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.103 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.104 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:44.104 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089710+0000", 2023-11-05T08:33:44.104 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.104 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.104 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.105 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:44.105 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089711+0000", 2023-11-05T08:33:44.105 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1429999999999999e-06 2023-11-05T08:33:44.105 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.105 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.105 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:44.106 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089733+0000", 2023-11-05T08:33:44.106 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1559000000000001e-05 2023-11-05T08:33:44.106 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.106 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.106 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:44.107 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089757+0000", 2023-11-05T08:33:44.107 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4515000000000002e-05 2023-11-05T08:33:44.107 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.107 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.107 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.107 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089780+0000", 2023-11-05T08:33:44.108 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2789000000000002e-05 2023-11-05T08:33:44.108 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.108 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.108 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:44.108 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.090049+0000", 2023-11-05T08:33:44.109 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000269363 2023-11-05T08:33:44.109 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.109 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.109 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:44.109 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.090074+0000", 2023-11-05T08:33:44.110 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4071999999999999e-05 2023-11-05T08:33:44.110 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.110 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.110 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:44.110 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.096355+0000", 2023-11-05T08:33:44.110 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0062814489999999997 2023-11-05T08:33:44.111 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.111 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.111 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:44.111 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.096446+0000", 2023-11-05T08:33:44.111 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.0816999999999995e-05 2023-11-05T08:33:44.112 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.112 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:44.112 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.112 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.112 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.112 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.8s4 36/30 ECSubWrite(tid=24, reqid=osd.5.0:23, at_version=0'0, trim_to=0'0, roll_forward_to=36'15))", 2023-11-05T08:33:44.113 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.089889+0000", 2023-11-05T08:33:44.113 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 54.972527878999998, 2023-11-05T08:33:44.113 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0065355869999999998, 2023-11-05T08:33:44.113 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:44.113 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:44.114 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:44.114 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.114 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:44.114 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089889+0000", 2023-11-05T08:33:44.114 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.115 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.115 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.115 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:44.115 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089889+0000", 2023-11-05T08:33:44.115 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.115 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.116 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.116 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:44.116 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089890+0000", 2023-11-05T08:33:44.116 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1599999999999999e-06 2023-11-05T08:33:44.116 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.117 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.117 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:44.117 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089908+0000", 2023-11-05T08:33:44.117 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8448999999999999e-05 2023-11-05T08:33:44.117 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.117 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.118 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:44.118 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089929+0000", 2023-11-05T08:33:44.118 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0534999999999999e-05 2023-11-05T08:33:44.118 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.118 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.119 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.119 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089943+0000", 2023-11-05T08:33:44.119 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4469e-05 2023-11-05T08:33:44.119 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.119 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.120 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:44.120 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.090098+0000", 2023-11-05T08:33:44.120 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00015513699999999999 2023-11-05T08:33:44.120 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.120 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.120 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:44.121 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.090114+0000", 2023-11-05T08:33:44.121 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5962999999999999e-05 2023-11-05T08:33:44.121 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.121 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.121 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:44.121 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.096334+0000", 2023-11-05T08:33:44.122 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0062193320000000002 2023-11-05T08:33:44.122 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.122 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.122 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:44.122 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.096424+0000", 2023-11-05T08:33:44.123 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.0542000000000001e-05 2023-11-05T08:33:44.123 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.123 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:44.123 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.123 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.124 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.124 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.as4 41/33 ECSubWrite(tid=35, reqid=osd.7.0:35, at_version=41'12, trim_to=0'0, roll_forward_to=40'10))", 2023-11-05T08:33:44.124 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.128988+0000", 2023-11-05T08:33:44.124 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 49.933428276999997, 2023-11-05T08:33:44.124 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0097032620000000007, 2023-11-05T08:33:44.124 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:44.125 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:44.125 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:44.125 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.125 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:44.125 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.128988+0000", 2023-11-05T08:33:44.126 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.126 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.126 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.126 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:44.126 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.128988+0000", 2023-11-05T08:33:44.127 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.127 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.127 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.127 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:44.127 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.128989+0000", 2023-11-05T08:33:44.127 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.3600000000000003e-07 2023-11-05T08:33:44.128 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.128 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.128 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:44.128 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.129043+0000", 2023-11-05T08:33:44.128 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.4163999999999998e-05 2023-11-05T08:33:44.128 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.129 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.129 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:44.129 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.129080+0000", 2023-11-05T08:33:44.129 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.7144999999999998e-05 2023-11-05T08:33:44.129 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.130 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.130 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.130 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.129104+0000", 2023-11-05T08:33:44.130 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3538999999999999e-05 2023-11-05T08:33:44.130 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.131 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.131 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:44.131 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.129248+0000", 2023-11-05T08:33:44.131 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00014404500000000001 2023-11-05T08:33:44.131 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.131 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.132 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:44.132 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.129273+0000", 2023-11-05T08:33:44.132 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5814999999999999e-05 2023-11-05T08:33:44.132 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.132 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.132 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:44.133 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.138576+0000", 2023-11-05T08:33:44.133 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0093020840000000004 2023-11-05T08:33:44.133 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.133 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.133 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:44.134 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.138691+0000", 2023-11-05T08:33:44.134 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000115834 2023-11-05T08:33:44.134 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.134 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:44.134 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.135 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.135 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.135 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.as4 41/33 ECSubWrite(tid=36, reqid=osd.7.0:36, at_version=41'14, trim_to=0'0, roll_forward_to=40'10))", 2023-11-05T08:33:44.135 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.131320+0000", 2023-11-05T08:33:44.135 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 49.931096396000001, 2023-11-05T08:33:44.136 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0086819870000000004, 2023-11-05T08:33:44.136 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:44.136 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:44.136 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:44.136 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.136 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:44.137 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131320+0000", 2023-11-05T08:33:44.137 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.137 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.137 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.137 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:44.138 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131320+0000", 2023-11-05T08:33:44.138 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.138 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.138 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.138 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:44.139 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131322+0000", 2023-11-05T08:33:44.139 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2500000000000001e-06 2023-11-05T08:33:44.139 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.139 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.139 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:44.139 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131375+0000", 2023-11-05T08:33:44.140 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.3041000000000003e-05 2023-11-05T08:33:44.140 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.140 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.140 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:44.140 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131422+0000", 2023-11-05T08:33:44.141 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.6526999999999997e-05 2023-11-05T08:33:44.141 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.141 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.141 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.141 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131445+0000", 2023-11-05T08:33:44.141 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3672e-05 2023-11-05T08:33:44.142 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.142 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.142 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:44.142 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131641+0000", 2023-11-05T08:33:44.142 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00019531299999999999 2023-11-05T08:33:44.143 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.143 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.143 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:44.143 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131666+0000", 2023-11-05T08:33:44.143 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5122000000000001e-05 2023-11-05T08:33:44.143 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.144 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.144 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:44.144 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.139945+0000", 2023-11-05T08:33:44.144 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0082790889999999999 2023-11-05T08:33:44.144 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.145 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.145 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:44.145 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.140002+0000", 2023-11-05T08:33:44.145 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.6972999999999998e-05 2023-11-05T08:33:44.145 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.146 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:44.146 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.146 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.146 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.146 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpReadReply(3.19s0 41/40 ECSubReadReply(tid=67, attrs_read=0))", 2023-11-05T08:33:44.146 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.176501+0000", 2023-11-05T08:33:44.147 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 49.885915879000002, 2023-11-05T08:33:44.147 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.006463318, 2023-11-05T08:33:44.147 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:44.147 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:44.147 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:44.148 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.148 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:44.148 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.176501+0000", 2023-11-05T08:33:44.148 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.148 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.148 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.149 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:44.149 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.176501+0000", 2023-11-05T08:33:44.149 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.149 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.149 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.150 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:44.150 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.176501+0000", 2023-11-05T08:33:44.150 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.5199999999999995e-07 2023-11-05T08:33:44.150 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.150 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.151 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:44.151 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.176802+0000", 2023-11-05T08:33:44.151 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00030095700000000003 2023-11-05T08:33:44.151 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.151 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.152 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:44.152 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.176822+0000", 2023-11-05T08:33:44.152 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9179000000000001e-05 2023-11-05T08:33:44.152 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.152 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.152 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.153 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.176834+0000", 2023-11-05T08:33:44.153 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2152e-05 2023-11-05T08:33:44.153 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.153 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.153 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:44.154 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.181163+0000", 2023-11-05T08:33:44.154 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0043295790000000001 2023-11-05T08:33:44.154 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.154 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.154 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:44.154 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.182964+0000", 2023-11-05T08:33:44.155 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0018004989999999999 2023-11-05T08:33:44.155 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.155 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:44.155 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.155 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.156 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.156 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:293 3.9s0 3:96f693b2:::smithi09039252-13:head [read 0~1064960,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2023-11-05T08:33:44.156 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.184012+0000", 2023-11-05T08:33:44.156 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 49.878404555000003, 2023-11-05T08:33:44.156 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0082043189999999998, 2023-11-05T08:33:44.157 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:44.157 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2023-11-05T08:33:44.157 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:33:44.157 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:33:44.157 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:44.157 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 293 2023-11-05T08:33:44.158 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.158 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:44.158 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.158 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:44.158 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184012+0000", 2023-11-05T08:33:44.159 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.159 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.159 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.159 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:44.159 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184012+0000", 2023-11-05T08:33:44.159 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.160 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.160 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.160 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:44.160 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184014+0000", 2023-11-05T08:33:44.160 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4930000000000001e-06 2023-11-05T08:33:44.161 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.161 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.161 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:44.161 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184024+0000", 2023-11-05T08:33:44.161 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0114e-05 2023-11-05T08:33:44.161 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.162 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.162 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:44.162 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184047+0000", 2023-11-05T08:33:44.162 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2463000000000002e-05 2023-11-05T08:33:44.162 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.162 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.163 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.163 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184063+0000", 2023-11-05T08:33:44.163 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6212e-05 2023-11-05T08:33:44.163 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.163 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.164 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:44.164 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184172+0000", 2023-11-05T08:33:44.164 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000108501 2023-11-05T08:33:44.164 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.164 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.164 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:33:44.165 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184445+0000", 2023-11-05T08:33:44.165 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00027307699999999999 2023-11-05T08:33:44.165 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.165 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.165 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:44.165 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.192216+0000", 2023-11-05T08:33:44.166 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0077714589999999997 2023-11-05T08:33:44.166 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.166 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:44.166 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.166 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.167 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.167 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:294 3.9s0 3:96f693b2:::smithi09039252-13:head [read 0~1064960,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2023-11-05T08:33:44.167 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.184100+0000", 2023-11-05T08:33:44.167 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 49.878316239999997, 2023-11-05T08:33:44.167 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0099915090000000008, 2023-11-05T08:33:44.168 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:44.168 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2023-11-05T08:33:44.168 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:33:44.168 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:33:44.168 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:44.169 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 294 2023-11-05T08:33:44.169 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.169 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:44.169 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.169 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:44.169 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184100+0000", 2023-11-05T08:33:44.170 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.170 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.170 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.170 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:44.170 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184100+0000", 2023-11-05T08:33:44.171 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.171 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.171 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.171 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:44.171 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184101+0000", 2023-11-05T08:33:44.171 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2100000000000001e-06 2023-11-05T08:33:44.172 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.172 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.172 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:44.172 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184109+0000", 2023-11-05T08:33:44.172 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.5630000000000003e-06 2023-11-05T08:33:44.173 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.173 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.173 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:44.173 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184126+0000", 2023-11-05T08:33:44.173 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6674e-05 2023-11-05T08:33:44.173 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.174 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.174 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.174 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184138+0000", 2023-11-05T08:33:44.174 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2306e-05 2023-11-05T08:33:44.174 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.175 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.175 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:44.175 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.189352+0000", 2023-11-05T08:33:44.175 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0052140629999999997 2023-11-05T08:33:44.175 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.176 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.176 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:33:44.176 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.189596+0000", 2023-11-05T08:33:44.176 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00024365899999999999 2023-11-05T08:33:44.176 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.177 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.177 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:44.177 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.194092+0000", 2023-11-05T08:33:44.177 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0044960340000000003 2023-11-05T08:33:44.177 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.177 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:44.178 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.178 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.178 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.178 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:295 3.9s0 3:96f693b2:::smithi09039252-13:head [read 0~1064960,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2023-11-05T08:33:44.178 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.184155+0000", 2023-11-05T08:33:44.179 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 49.878261434999999, 2023-11-05T08:33:44.179 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.011309207999999999, 2023-11-05T08:33:44.179 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:44.179 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2023-11-05T08:33:44.179 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:33:44.179 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:33:44.180 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:44.180 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 295 2023-11-05T08:33:44.180 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.180 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:44.180 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.181 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:44.181 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184155+0000", 2023-11-05T08:33:44.181 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.181 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.181 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.182 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:44.182 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184155+0000", 2023-11-05T08:33:44.182 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.182 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.182 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.182 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:44.183 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184156+0000", 2023-11-05T08:33:44.183 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0139999999999999e-06 2023-11-05T08:33:44.183 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.183 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.183 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:44.184 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184162+0000", 2023-11-05T08:33:44.184 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.5330000000000002e-06 2023-11-05T08:33:44.184 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.184 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.184 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:44.184 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184176+0000", 2023-11-05T08:33:44.185 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.486e-05 2023-11-05T08:33:44.185 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.185 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.185 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.185 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184188+0000", 2023-11-05T08:33:44.186 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1246e-05 2023-11-05T08:33:44.186 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.186 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.186 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:44.186 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.190012+0000", 2023-11-05T08:33:44.186 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0058240920000000003 2023-11-05T08:33:44.187 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.187 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.187 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:33:44.187 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.190143+0000", 2023-11-05T08:33:44.187 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000130984 2023-11-05T08:33:44.187 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.188 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.188 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:44.188 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.195464+0000", 2023-11-05T08:33:44.188 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0053214789999999996 2023-11-05T08:33:44.188 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.189 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:44.189 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.189 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.189 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.189 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.14s5 41/40 ECSubWrite(tid=17, reqid=client.4691.0:278, at_version=41'14, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:44.190 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.185798+0000", 2023-11-05T08:33:44.190 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 49.876618157999999, 2023-11-05T08:33:44.190 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0063439719999999998, 2023-11-05T08:33:44.190 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:44.190 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:44.190 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:44.191 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.191 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:44.191 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.185798+0000", 2023-11-05T08:33:44.191 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.191 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.192 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.192 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:44.192 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.185798+0000", 2023-11-05T08:33:44.192 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.192 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.192 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.193 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:44.193 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.185800+0000", 2023-11-05T08:33:44.193 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.689e-06 2023-11-05T08:33:44.193 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.193 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.194 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:44.194 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.186569+0000", 2023-11-05T08:33:44.194 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00076927199999999999 2023-11-05T08:33:44.194 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.194 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.194 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:44.195 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.186598+0000", 2023-11-05T08:33:44.195 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8776999999999999e-05 2023-11-05T08:33:44.195 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.195 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.195 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.196 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.186615+0000", 2023-11-05T08:33:44.196 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6759e-05 2023-11-05T08:33:44.196 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.196 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.196 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:44.196 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.186763+0000", 2023-11-05T08:33:44.197 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000148646 2023-11-05T08:33:44.197 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.197 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.197 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:44.197 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.186784+0000", 2023-11-05T08:33:44.198 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0709000000000001e-05 2023-11-05T08:33:44.198 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.198 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.198 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:44.198 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.192059+0000", 2023-11-05T08:33:44.199 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0052751400000000002 2023-11-05T08:33:44.199 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.199 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.199 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:44.199 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.192142+0000", 2023-11-05T08:33:44.199 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.2979999999999995e-05 2023-11-05T08:33:44.200 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.200 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:44.200 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.200 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.200 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.201 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.as4 42/33 ECSubWrite(tid=38, reqid=osd.7.0:38, at_version=42'16, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:33:44.201 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:55.127593+0000", 2023-11-05T08:33:44.201 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 48.934823217000002, 2023-11-05T08:33:44.201 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.014145875, 2023-11-05T08:33:44.201 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:44.201 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:44.202 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:44.202 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.202 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:44.202 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127593+0000", 2023-11-05T08:33:44.202 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.203 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.203 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.203 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:44.203 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127593+0000", 2023-11-05T08:33:44.203 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.203 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.204 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.204 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:44.204 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127594+0000", 2023-11-05T08:33:44.204 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.1399999999999996e-07 2023-11-05T08:33:44.204 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.205 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.205 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:44.205 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127658+0000", 2023-11-05T08:33:44.205 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.3590000000000006e-05 2023-11-05T08:33:44.205 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.205 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.206 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:44.206 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127690+0000", 2023-11-05T08:33:44.206 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2447999999999998e-05 2023-11-05T08:33:44.206 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.206 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.207 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.207 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127716+0000", 2023-11-05T08:33:44.207 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5931999999999998e-05 2023-11-05T08:33:44.207 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.207 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.208 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:44.208 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127955+0000", 2023-11-05T08:33:44.208 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00023882100000000001 2023-11-05T08:33:44.208 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.208 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.209 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:44.209 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127983+0000", 2023-11-05T08:33:44.209 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7928000000000001e-05 2023-11-05T08:33:44.209 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.209 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.209 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:44.210 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.141638+0000", 2023-11-05T08:33:44.210 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.013655244 2023-11-05T08:33:44.210 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.210 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.210 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:44.211 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.141739+0000", 2023-11-05T08:33:44.211 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000101098 2023-11-05T08:33:44.211 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.211 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:44.211 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.212 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.212 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.212 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.as4 42/33 ECSubWrite(tid=39, reqid=osd.7.0:39, at_version=42'18, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:33:44.212 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:55.127804+0000", 2023-11-05T08:33:44.212 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 48.934612393999998, 2023-11-05T08:33:44.212 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.017037818, 2023-11-05T08:33:44.213 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:44.213 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:44.213 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:44.213 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.213 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:44.214 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127804+0000", 2023-11-05T08:33:44.214 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.214 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.214 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.214 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:44.214 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127804+0000", 2023-11-05T08:33:44.215 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.215 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.215 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.215 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:44.215 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127805+0000", 2023-11-05T08:33:44.216 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.026e-06 2023-11-05T08:33:44.216 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.216 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.216 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:44.216 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127848+0000", 2023-11-05T08:33:44.217 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.3195000000000002e-05 2023-11-05T08:33:44.217 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.217 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.217 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:44.217 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.133513+0000", 2023-11-05T08:33:44.217 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.005664741 2023-11-05T08:33:44.218 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.218 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.218 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.218 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.133539+0000", 2023-11-05T08:33:44.218 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6389000000000001e-05 2023-11-05T08:33:44.219 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.219 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.219 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:44.219 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.135275+0000", 2023-11-05T08:33:44.219 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001735349 2023-11-05T08:33:44.219 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.220 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.220 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:44.220 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.135312+0000", 2023-11-05T08:33:44.220 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.6965000000000003e-05 2023-11-05T08:33:44.220 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.221 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.221 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:44.221 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.144746+0000", 2023-11-05T08:33:44.221 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0094338319999999996 2023-11-05T08:33:44.221 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.221 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.222 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:44.222 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.144842+0000", 2023-11-05T08:33:44.222 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.6321000000000002e-05 2023-11-05T08:33:44.222 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.222 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:44.223 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.223 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.223 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.223 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:341 3.9s0 3:96f693b2:::smithi09039252-13:head [create,setxattr ]ScjNOsReWmcrsXtQckiNfVsraic]liXUgaZfrPb (12280) in=40b,setxattr RP^fql]PXPbbV]`\\UXhZUcc\\s\\eVVh]On^oUhNgZ (593) in=40b,setxattr ejajVrghj]sk_[VhasYoUSNhlO_PNopbXh\\iYad\\ (1963) in=40b,setxattr ocopbYhpUShgXjpVTtXhgrZ^fhTrWUZnka_sOlQ] (1506) in=40b,setxattr drSfWQp[TZZaai[S^\\ecpepRjetPTZ[Qn[slVhsn (1437) in=40b,setxattr qfWkbgV_hlqXWl`QOe]pOPUPZspbtU__NVPUUltj (877) in=40b,setxattr ]Vrng_\\anafncgktlQh_okb\\i]chR]P]jeoaR\\h] (1858) in=40b,setxattr ]W_abhUq]ocdkOt[ZkpjgPX^drZi]OcUWNRonRa\\ (455) in=40b,setxattr WiZeSWPfsqgYXk`d\\gOeSecbpakdsPTXZfZhn_k^ (875) in=40b,setxattr kqn^akbbamUTRXRhQS^i`lQYomhV_P`j\\ahTj]dk (1894) in=40b,setxattr TUcrapOXk_aZjTSdOT`rYcU_YYstnVgNTojk`lXZ (805) in=40b,setxattr a`RfekoeadQgiYossXre`rV^qNeUNiNf\\lSXhohq (15648) in=40b,setxattr VigYdf[rkVcbX`_aXcZrljeilZZV\\O[gpOhof]\\P (1473) in=40b,setxattr Qrfl^hc[_sp[p`bcN[Ne\\tg^]rgrOlRqfQ_VWNt\\ (1261) in=40b,setxattr iQVqR\\bssbQRQf^a`UrVROfgpPZlnstjPUaUnWWt (665) in=40b,setxattr aO^mjdTNk[qYOgYsgTigbbP`]mhe`WmaNUO[^ele (979) in=40b,setxattr cfQeZUernkePqgPcSrokkcnPqZ^Osc^^ng[ilga_ (16840) in=40b,setxattr fdbVa_]fdVq_`kcpaWlQSgY^VgakSqVhj`qqdj[m (1858) in=40b,setxattr tXieTi_TgP_dVipjmQcU_QVRqNdboYhRWgNrpVhm (1858) in=40b,setxattr ]YTcidkg\\][ZtajYPqmOji_UPVPZ`j^]tnNRefcd (2876) in=40b] snapc 10=[7] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:44.223 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:05.255132+0000", 2023-11-05T08:33:44.224 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 38.807283970999997, 2023-11-05T08:33:44.224 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.01486664, 2023-11-05T08:33:44.224 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:44.224 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:44.224 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:33:44.225 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:33:44.225 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:44.225 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 341 2023-11-05T08:33:44.225 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.225 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:44.225 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.226 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:44.226 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255132+0000", 2023-11-05T08:33:44.226 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.226 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.226 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.226 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:44.227 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255132+0000", 2023-11-05T08:33:44.227 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.227 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.227 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.227 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:44.228 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255136+0000", 2023-11-05T08:33:44.228 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.5159999999999999e-06 2023-11-05T08:33:44.228 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.228 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.228 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:44.228 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255423+0000", 2023-11-05T08:33:44.229 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00028735799999999998 2023-11-05T08:33:44.229 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.229 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.229 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:44.229 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255462+0000", 2023-11-05T08:33:44.230 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.8512e-05 2023-11-05T08:33:44.230 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.230 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.230 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.230 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255499+0000", 2023-11-05T08:33:44.231 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.7531999999999999e-05 2023-11-05T08:33:44.231 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.231 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.231 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:44.231 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255651+0000", 2023-11-05T08:33:44.231 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00015121700000000001 2023-11-05T08:33:44.232 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.232 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.232 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:33:44.232 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.256051+0000", 2023-11-05T08:33:44.232 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00040081699999999998 2023-11-05T08:33:44.233 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.233 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.233 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:44.233 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.257968+0000", 2023-11-05T08:33:44.233 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001917052 2023-11-05T08:33:44.234 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.234 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.234 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:44.234 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.265314+0000", 2023-11-05T08:33:44.234 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0073454469999999997 2023-11-05T08:33:44.234 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.235 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.235 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:33:44.235 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.269385+0000", 2023-11-05T08:33:44.235 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0040709839999999997 2023-11-05T08:33:44.235 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.236 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.236 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:44.236 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.269999+0000", 2023-11-05T08:33:44.236 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00061420500000000005 2023-11-05T08:33:44.236 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.236 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:44.237 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.237 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.237 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.237 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:344 3.9s0 3:936041dc:::smithi09039252-3:head [copy-from ver 4] snapc 10=[7] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:44.237 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:05.255538+0000", 2023-11-05T08:33:44.237 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 38.806877960999998, 2023-11-05T08:33:44.238 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.032906886000000003, 2023-11-05T08:33:44.238 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:44.238 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:44.238 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:33:44.238 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:33:44.239 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:44.239 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 344 2023-11-05T08:33:44.239 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.239 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:44.239 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.239 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:44.240 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255538+0000", 2023-11-05T08:33:44.240 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.240 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.240 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.240 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:44.241 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255538+0000", 2023-11-05T08:33:44.241 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.241 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.241 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.241 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:44.242 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255540+0000", 2023-11-05T08:33:44.242 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5060000000000001e-06 2023-11-05T08:33:44.242 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.242 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.242 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:44.243 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255550+0000", 2023-11-05T08:33:44.243 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0063e-05 2023-11-05T08:33:44.243 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.243 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.243 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:44.243 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255589+0000", 2023-11-05T08:33:44.244 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.9245000000000001e-05 2023-11-05T08:33:44.244 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.244 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.244 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.244 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255606+0000", 2023-11-05T08:33:44.245 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6307000000000002e-05 2023-11-05T08:33:44.245 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.245 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.245 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:44.245 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.259641+0000", 2023-11-05T08:33:44.245 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.004035421 2023-11-05T08:33:44.246 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.246 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.246 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:33:44.246 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.260112+0000", 2023-11-05T08:33:44.246 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00047097799999999999 2023-11-05T08:33:44.247 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.247 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.247 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:44.247 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.280466+0000", 2023-11-05T08:33:44.247 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.020353850999999999 2023-11-05T08:33:44.247 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.248 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.248 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:44.248 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.286378+0000", 2023-11-05T08:33:44.248 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0059124490000000002 2023-11-05T08:33:44.248 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.249 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.249 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:33:44.249 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.288092+0000", 2023-11-05T08:33:44.249 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0017133300000000001 2023-11-05T08:33:44.249 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.249 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.250 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:44.250 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.288445+0000", 2023-11-05T08:33:44.250 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000353736 2023-11-05T08:33:44.250 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.250 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:44.251 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.251 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.251 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.251 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:345 3.9s0 3:936041dc:::smithi09039252-3:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:44.251 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:05.255630+0000", 2023-11-05T08:33:44.251 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 38.806786500999998, 2023-11-05T08:33:44.252 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.029547922000000001, 2023-11-05T08:33:44.252 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:44.252 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2023-11-05T08:33:44.252 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:33:44.252 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:33:44.253 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:44.253 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 345 2023-11-05T08:33:44.253 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.253 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:44.253 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.254 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:44.254 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255630+0000", 2023-11-05T08:33:44.254 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.254 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.254 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.255 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:44.255 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255630+0000", 2023-11-05T08:33:44.255 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.255 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.255 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.255 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:44.256 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255631+0000", 2023-11-05T08:33:44.256 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.519e-06 2023-11-05T08:33:44.256 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.256 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.256 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:44.256 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255640+0000", 2023-11-05T08:33:44.257 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.9360000000000005e-06 2023-11-05T08:33:44.257 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.257 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.257 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:44.257 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255659+0000", 2023-11-05T08:33:44.258 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9089999999999998e-05 2023-11-05T08:33:44.258 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.258 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.258 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.258 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255675+0000", 2023-11-05T08:33:44.259 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5133000000000001e-05 2023-11-05T08:33:44.259 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.259 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.259 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:44.259 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.260602+0000", 2023-11-05T08:33:44.259 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0049270449999999997 2023-11-05T08:33:44.260 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.260 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.260 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for blocked object", 2023-11-05T08:33:44.260 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.260717+0000", 2023-11-05T08:33:44.260 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000115182 2023-11-05T08:33:44.261 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.261 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.261 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:44.261 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.284014+0000", 2023-11-05T08:33:44.261 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.023297577999999999 2023-11-05T08:33:44.261 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.262 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.262 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:33:44.262 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.284370+0000", 2023-11-05T08:33:44.262 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000355306 2023-11-05T08:33:44.262 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.263 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.263 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:44.263 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.285178+0000", 2023-11-05T08:33:44.263 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00080813300000000005 2023-11-05T08:33:44.263 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.263 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:44.264 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.264 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.264 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.264 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.as4 52/45 ECSubWrite(tid=45, reqid=client.4691.0:349, at_version=52'21, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:44.264 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:05.264746+0000", 2023-11-05T08:33:44.264 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 38.797670439999997, 2023-11-05T08:33:44.265 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0072026099999999999, 2023-11-05T08:33:44.265 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:44.265 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:44.265 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:44.265 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.266 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:44.266 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.264746+0000", 2023-11-05T08:33:44.266 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.266 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.266 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.266 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:44.267 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.264746+0000", 2023-11-05T08:33:44.267 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.267 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.267 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.267 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:44.268 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.264748+0000", 2023-11-05T08:33:44.268 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.677e-06 2023-11-05T08:33:44.268 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.268 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.268 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:44.268 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.264893+0000", 2023-11-05T08:33:44.269 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000145067 2023-11-05T08:33:44.269 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.269 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.269 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:44.269 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.264921+0000", 2023-11-05T08:33:44.270 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8328999999999999e-05 2023-11-05T08:33:44.270 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.270 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.270 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.270 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.264942+0000", 2023-11-05T08:33:44.271 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1046000000000001e-05 2023-11-05T08:33:44.271 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.271 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.271 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:44.271 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.265123+0000", 2023-11-05T08:33:44.271 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00018091299999999999 2023-11-05T08:33:44.272 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.272 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.272 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:44.272 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.265150+0000", 2023-11-05T08:33:44.272 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6645e-05 2023-11-05T08:33:44.273 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.273 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.273 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:44.273 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.271806+0000", 2023-11-05T08:33:44.273 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0066566919999999996 2023-11-05T08:33:44.273 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.274 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.274 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:44.274 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.271949+0000", 2023-11-05T08:33:44.274 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00014224100000000001 2023-11-05T08:33:44.274 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.275 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:44.275 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.275 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.275 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.275 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.as4 61/45 ECSubWrite(tid=57, reqid=client.4691.0:405, at_version=61'23, trim_to=0'0, roll_forward_to=52'21))", 2023-11-05T08:33:44.275 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:14.366205+0000", 2023-11-05T08:33:44.276 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 29.69621102, 2023-11-05T08:33:44.276 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0087065779999999995, 2023-11-05T08:33:44.276 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:44.276 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:44.276 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:44.277 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.277 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:44.277 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.366205+0000", 2023-11-05T08:33:44.277 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.277 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.278 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.278 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:44.278 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.366205+0000", 2023-11-05T08:33:44.278 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.278 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.279 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.279 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:44.279 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.366206+0000", 2023-11-05T08:33:44.279 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.3000000000000001e-07 2023-11-05T08:33:44.279 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.279 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.280 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:44.280 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.366951+0000", 2023-11-05T08:33:44.280 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00074537699999999998 2023-11-05T08:33:44.280 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.280 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.281 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:44.281 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.366978+0000", 2023-11-05T08:33:44.281 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6721999999999999e-05 2023-11-05T08:33:44.281 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.281 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.281 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.282 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.366994+0000", 2023-11-05T08:33:44.282 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6189999999999999e-05 2023-11-05T08:33:44.282 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.282 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.282 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:44.283 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.367186+0000", 2023-11-05T08:33:44.283 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000191387 2023-11-05T08:33:44.283 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.283 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.283 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:44.283 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.367213+0000", 2023-11-05T08:33:44.284 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7169e-05 2023-11-05T08:33:44.284 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.284 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.284 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:44.284 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.374835+0000", 2023-11-05T08:33:44.285 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0076217589999999997 2023-11-05T08:33:44.285 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.285 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.285 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:44.285 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.374912+0000", 2023-11-05T08:33:44.285 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.7443999999999999e-05 2023-11-05T08:33:44.286 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.286 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:44.286 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.286 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.286 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.286 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.fs2 77/37 ECSubWrite(tid=130, reqid=client.4691.0:579, at_version=77'44, trim_to=0'0, roll_forward_to=70'42))", 2023-11-05T08:33:44.287 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:30.540041+0000", 2023-11-05T08:33:44.287 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 13.522375805999999, 2023-11-05T08:33:44.287 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0083478800000000002, 2023-11-05T08:33:44.287 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:33:44.287 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:33:44.288 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:33:44.288 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.288 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:33:44.288 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.540041+0000", 2023-11-05T08:33:44.288 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.288 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.289 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.289 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:33:44.289 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.540041+0000", 2023-11-05T08:33:44.289 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:33:44.289 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.290 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.290 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:33:44.290 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.540042+0000", 2023-11-05T08:33:44.290 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0130000000000001e-06 2023-11-05T08:33:44.290 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.290 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.291 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:33:44.291 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.540574+0000", 2023-11-05T08:33:44.291 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00053246000000000001 2023-11-05T08:33:44.291 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.291 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.292 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:33:44.292 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.540600+0000", 2023-11-05T08:33:44.292 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6007999999999999e-05 2023-11-05T08:33:44.292 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.292 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.293 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.293 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.540623+0000", 2023-11-05T08:33:44.293 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3416000000000001e-05 2023-11-05T08:33:44.293 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.293 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.293 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:33:44.294 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.540798+0000", 2023-11-05T08:33:44.294 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000174897 2023-11-05T08:33:44.294 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.294 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.294 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:33:44.295 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.540830+0000", 2023-11-05T08:33:44.295 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.1943999999999998e-05 2023-11-05T08:33:44.295 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.295 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.295 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:33:44.295 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.548328+0000", 2023-11-05T08:33:44.296 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0074971990000000004 2023-11-05T08:33:44.296 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:33:44.296 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:33:44.296 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:33:44.296 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:30.548388+0000", 2023-11-05T08:33:44.296 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.0943000000000002e-05 2023-11-05T08:33:44.297 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.297 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:44.297 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.297 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:33:44.297 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:33:44.298 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:33:44.299 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-11-05T08:33:44.300 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:33:44.297+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:33:44.400 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:44.400+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:44.431 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:33:44.432 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-11-05T08:33:44.432 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-11-05T08:33:44.432 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:33:44.447 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-11-05T08:33:44.502 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:33:44.496+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:33:44.568 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:33:44.569 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-11-05T08:33:44.569 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-11-05T08:33:44.569 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-11-05T08:33:44.569 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:33:44.582 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-11-05T08:33:44.602 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:33:44.601+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:33:44.647 INFO:teuthology.orchestra.run.smithi002.stderr:nodeep-scrub is set 2023-11-05T08:33:44.654 INFO:tasks.rados.rados.0.smithi090.stdout:283: done (4 left) 2023-11-05T08:33:44.654 INFO:tasks.rados.rados.0.smithi090.stdout:284: snap_remove snap 27 2023-11-05T08:33:44.703 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:33:44.700+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:33:44.709 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:33:44.709 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-11-05T08:33:44.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-11-05T08:33:44.710 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-11-05T08:33:44.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.710 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:233 3.ds0 3:bf7e600d:::smithi09039252-23:head [append 0~0] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2023-11-05T08:33:44.710 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.059123+0000", 2023-11-05T08:33:44.711 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 55.646191354999999, 2023-11-05T08:33:44.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019416857999999999, 2023-11-05T08:33:44.711 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:44.711 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:44.711 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:44.711 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:44.712 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:44.712 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 233 2023-11-05T08:33:44.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.712 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:44.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:44.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059123+0000", 2023-11-05T08:33:44.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:44.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059123+0000", 2023-11-05T08:33:44.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:44.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059126+0000", 2023-11-05T08:33:44.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8370000000000001e-06 2023-11-05T08:33:44.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:44.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059140+0000", 2023-11-05T08:33:44.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3669000000000001e-05 2023-11-05T08:33:44.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:44.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059176+0000", 2023-11-05T08:33:44.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6548000000000003e-05 2023-11-05T08:33:44.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059198+0000", 2023-11-05T08:33:44.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1489e-05 2023-11-05T08:33:44.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:44.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059401+0000", 2023-11-05T08:33:44.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020313300000000001 2023-11-05T08:33:44.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-11-05T08:33:44.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059422+0000", 2023-11-05T08:33:44.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1212e-05 2023-11-05T08:33:44.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:44.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.063981+0000", 2023-11-05T08:33:44.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004558604 2023-11-05T08:33:44.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:44.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.064128+0000", 2023-11-05T08:33:44.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014734899999999999 2023-11-05T08:33:44.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:44.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.066042+0000", 2023-11-05T08:33:44.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019138079999999999 2023-11-05T08:33:44.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:44.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.077186+0000", 2023-11-05T08:33:44.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011144536 2023-11-05T08:33:44.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:44.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.078357+0000", 2023-11-05T08:33:44.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011709089999999999 2023-11-05T08:33:44.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:44.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.078540+0000", 2023-11-05T08:33:44.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000182764 2023-11-05T08:33:44.725 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.725 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:44.725 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.726 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:234 3.ds0 3:bf7e600d:::smithi09039252-23:head [rollback 3] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2023-11-05T08:33:44.726 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.059247+0000", 2023-11-05T08:33:44.726 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 55.646067758000001, 2023-11-05T08:33:44.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019739059, 2023-11-05T08:33:44.727 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:44.727 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:44.727 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:44.727 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:44.727 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:44.728 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 234 2023-11-05T08:33:44.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.728 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:44.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:44.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059247+0000", 2023-11-05T08:33:44.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:44.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059247+0000", 2023-11-05T08:33:44.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:44.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059248+0000", 2023-11-05T08:33:44.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.077e-06 2023-11-05T08:33:44.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:44.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059256+0000", 2023-11-05T08:33:44.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1359999999999997e-06 2023-11-05T08:33:44.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:44.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059283+0000", 2023-11-05T08:33:44.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6855e-05 2023-11-05T08:33:44.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059298+0000", 2023-11-05T08:33:44.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5356e-05 2023-11-05T08:33:44.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:44.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059582+0000", 2023-11-05T08:33:44.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028341299999999999 2023-11-05T08:33:44.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:33:44.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059592+0000", 2023-11-05T08:33:44.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0103e-05 2023-11-05T08:33:44.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:44.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.067711+0000", 2023-11-05T08:33:44.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0081197660000000005 2023-11-05T08:33:44.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:44.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.067965+0000", 2023-11-05T08:33:44.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025397000000000001 2023-11-05T08:33:44.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:44.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.069296+0000", 2023-11-05T08:33:44.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013308910000000001 2023-11-05T08:33:44.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:44.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.077189+0000", 2023-11-05T08:33:44.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0078931079999999994 2023-11-05T08:33:44.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:44.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.078829+0000", 2023-11-05T08:33:44.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016394459999999999 2023-11-05T08:33:44.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:44.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.078986+0000", 2023-11-05T08:33:44.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015693799999999999 2023-11-05T08:33:44.741 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.741 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:44.741 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.742 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:235 3.ds0 3:bf7e600d:::smithi09039252-23:head [append 0~0] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2023-11-05T08:33:44.742 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.059370+0000", 2023-11-05T08:33:44.742 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 55.645944544000002, 2023-11-05T08:33:44.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025190444999999999, 2023-11-05T08:33:44.743 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:44.743 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:44.743 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:44.743 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:44.743 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:44.743 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 235 2023-11-05T08:33:44.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.744 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:44.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:44.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059370+0000", 2023-11-05T08:33:44.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:44.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059370+0000", 2023-11-05T08:33:44.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:44.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059371+0000", 2023-11-05T08:33:44.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2899999999999999e-06 2023-11-05T08:33:44.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:44.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059378+0000", 2023-11-05T08:33:44.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0729999999999999e-06 2023-11-05T08:33:44.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:44.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059408+0000", 2023-11-05T08:33:44.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9904999999999999e-05 2023-11-05T08:33:44.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059418+0000", 2023-11-05T08:33:44.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7659999999999993e-06 2023-11-05T08:33:44.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:44.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059648+0000", 2023-11-05T08:33:44.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022963200000000001 2023-11-05T08:33:44.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:33:44.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059657+0000", 2023-11-05T08:33:44.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9700000000000005e-06 2023-11-05T08:33:44.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:44.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.070350+0000", 2023-11-05T08:33:44.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010693342 2023-11-05T08:33:44.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:44.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.070601+0000", 2023-11-05T08:33:44.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025082000000000001 2023-11-05T08:33:44.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:44.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.071898+0000", 2023-11-05T08:33:44.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012976190000000001 2023-11-05T08:33:44.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:44.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.077192+0000", 2023-11-05T08:33:44.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052931840000000003 2023-11-05T08:33:44.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:44.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.084398+0000", 2023-11-05T08:33:44.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0072062619999999997 2023-11-05T08:33:44.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:44.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.084560+0000", 2023-11-05T08:33:44.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016258200000000001 2023-11-05T08:33:44.757 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.757 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:44.757 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.757 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:286 3.as1 3:5396e455:::smithi09039252-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:33:44.758 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:53.170019+0000", 2023-11-05T08:33:44.758 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 51.535295118000001, 2023-11-05T08:33:44.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014799489000000001, 2023-11-05T08:33:44.758 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:44.758 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:44.759 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:44.759 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:44.759 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:44.759 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 286 2023-11-05T08:33:44.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.759 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:44.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:44.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.170019+0000", 2023-11-05T08:33:44.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:44.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.170019+0000", 2023-11-05T08:33:44.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:44.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.170021+0000", 2023-11-05T08:33:44.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.877e-06 2023-11-05T08:33:44.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:44.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.170029+0000", 2023-11-05T08:33:44.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6960000000000005e-06 2023-11-05T08:33:44.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:44.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.170056+0000", 2023-11-05T08:33:44.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6928999999999999e-05 2023-11-05T08:33:44.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.170072+0000", 2023-11-05T08:33:44.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6515999999999999e-05 2023-11-05T08:33:44.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:44.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.180292+0000", 2023-11-05T08:33:44.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010219293000000001 2023-11-05T08:33:44.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-11-05T08:33:44.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.180558+0000", 2023-11-05T08:33:44.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026664499999999998 2023-11-05T08:33:44.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:44.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.182159+0000", 2023-11-05T08:33:44.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016003650000000001 2023-11-05T08:33:44.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:44.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.182378+0000", 2023-11-05T08:33:44.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021884699999999999 2023-11-05T08:33:44.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.769 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:44.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:53.184819+0000", 2023-11-05T08:33:44.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024413210000000002 2023-11-05T08:33:44.770 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.770 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:44.770 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.771 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:349 3.as1 3:56f12b5c:::smithi09039252-15:head [copy-from ver 2] snapc 10=[7] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:44.771 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:05.254950+0000", 2023-11-05T08:33:44.771 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.450364546000003, 2023-11-05T08:33:44.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018320570000000001, 2023-11-05T08:33:44.771 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:44.771 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:44.772 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:44.772 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:44.772 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:44.772 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 349 2023-11-05T08:33:44.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.772 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:44.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:44.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.254950+0000", 2023-11-05T08:33:44.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:44.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.254950+0000", 2023-11-05T08:33:44.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:44.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.254953+0000", 2023-11-05T08:33:44.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6699999999999998e-06 2023-11-05T08:33:44.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:44.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.254966+0000", 2023-11-05T08:33:44.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.344e-05 2023-11-05T08:33:44.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:44.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255101+0000", 2023-11-05T08:33:44.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013478200000000001 2023-11-05T08:33:44.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255123+0000", 2023-11-05T08:33:44.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2541999999999999e-05 2023-11-05T08:33:44.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:44.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255409+0000", 2023-11-05T08:33:44.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028556799999999999 2023-11-05T08:33:44.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:44.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255826+0000", 2023-11-05T08:33:44.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041679299999999999 2023-11-05T08:33:44.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:44.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.262849+0000", 2023-11-05T08:33:44.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0070237030000000001 2023-11-05T08:33:44.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:44.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.272188+0000", 2023-11-05T08:33:44.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0093385970000000006 2023-11-05T08:33:44.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:44.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.273116+0000", 2023-11-05T08:33:44.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00092777199999999999 2023-11-05T08:33:44.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:44.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.273271+0000", 2023-11-05T08:33:44.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015470300000000001 2023-11-05T08:33:44.783 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.784 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:44.784 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.784 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:357 3.as1 3:5396e455:::smithi09039252-43:head [read 0~1851392,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:44.785 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:05.279538+0000", 2023-11-05T08:33:44.785 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.425776573, 2023-11-05T08:33:44.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015991607000000001, 2023-11-05T08:33:44.785 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:44.785 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:44.785 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:44.786 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:44.786 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:44.786 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 357 2023-11-05T08:33:44.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.786 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:44.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:44.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.279538+0000", 2023-11-05T08:33:44.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:44.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.279538+0000", 2023-11-05T08:33:44.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:44.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.279538+0000", 2023-11-05T08:33:44.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0100000000000005e-07 2023-11-05T08:33:44.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:44.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.279542+0000", 2023-11-05T08:33:44.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8519999999999997e-06 2023-11-05T08:33:44.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:44.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.279551+0000", 2023-11-05T08:33:44.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2409999999999995e-06 2023-11-05T08:33:44.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.279557+0000", 2023-11-05T08:33:44.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4219999999999997e-06 2023-11-05T08:33:44.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:44.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.281448+0000", 2023-11-05T08:33:44.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001891104 2023-11-05T08:33:44.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:44.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.281739+0000", 2023-11-05T08:33:44.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029091399999999997 2023-11-05T08:33:44.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:44.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.295530+0000", 2023-11-05T08:33:44.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013790573 2023-11-05T08:33:44.795 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.795 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:44.795 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.796 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:363 3.ds0 3:bf537852:::smithi09039252-10:head [read 0~163840,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2023-11-05T08:33:44.796 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:08.268702+0000", 2023-11-05T08:33:44.796 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 36.436612124, 2023-11-05T08:33:44.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012581584, 2023-11-05T08:33:44.796 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:44.796 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:44.797 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:44.797 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:44.797 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:44.797 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 363 2023-11-05T08:33:44.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.798 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:44.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:44.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.268702+0000", 2023-11-05T08:33:44.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:44.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.268702+0000", 2023-11-05T08:33:44.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:44.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.268705+0000", 2023-11-05T08:33:44.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2139999999999999e-06 2023-11-05T08:33:44.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:44.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.268715+0000", 2023-11-05T08:33:44.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0066e-05 2023-11-05T08:33:44.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:44.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.268737+0000", 2023-11-05T08:33:44.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2073e-05 2023-11-05T08:33:44.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.268753+0000", 2023-11-05T08:33:44.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6503000000000001e-05 2023-11-05T08:33:44.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:44.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.269068+0000", 2023-11-05T08:33:44.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031519600000000001 2023-11-05T08:33:44.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:33:44.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.269097+0000", 2023-11-05T08:33:44.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8670000000000002e-05 2023-11-05T08:33:44.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:44.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.276952+0000", 2023-11-05T08:33:44.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0078548060000000006 2023-11-05T08:33:44.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:44.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.277120+0000", 2023-11-05T08:33:44.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016855400000000001 2023-11-05T08:33:44.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:44.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.281284+0000", 2023-11-05T08:33:44.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0041635020000000003 2023-11-05T08:33:44.808 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.808 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:44.808 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.809 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:364 3.ds0 3:bf537852:::smithi09039252-10:head [read 0~163840,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2023-11-05T08:33:44.809 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:08.268782+0000", 2023-11-05T08:33:44.809 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 36.436532868999997, 2023-11-05T08:33:44.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013418439000000001, 2023-11-05T08:33:44.809 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:44.809 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:44.810 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:44.810 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:44.810 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:44.810 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 364 2023-11-05T08:33:44.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.811 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:44.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:44.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.268782+0000", 2023-11-05T08:33:44.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:44.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.268782+0000", 2023-11-05T08:33:44.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:44.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.268783+0000", 2023-11-05T08:33:44.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2729999999999999e-06 2023-11-05T08:33:44.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:44.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.268791+0000", 2023-11-05T08:33:44.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9000000000000006e-06 2023-11-05T08:33:44.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:44.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.268810+0000", 2023-11-05T08:33:44.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9556e-05 2023-11-05T08:33:44.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.268825+0000", 2023-11-05T08:33:44.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4392999999999999e-05 2023-11-05T08:33:44.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:44.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.269166+0000", 2023-11-05T08:33:44.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034102200000000002 2023-11-05T08:33:44.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:33:44.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.269178+0000", 2023-11-05T08:33:44.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2289999999999999e-05 2023-11-05T08:33:44.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:44.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.279976+0000", 2023-11-05T08:33:44.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010797543 2023-11-05T08:33:44.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:44.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.280117+0000", 2023-11-05T08:33:44.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000141755 2023-11-05T08:33:44.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:44.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:08.282200+0000", 2023-11-05T08:33:44.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002082707 2023-11-05T08:33:44.820 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.821 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:44.821 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.821 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:379 3.ds0 3:bf537852:::smithi09039252-10:head [read 0~163840,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2023-11-05T08:33:44.821 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:10.278322+0000", 2023-11-05T08:33:44.822 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.426992695999999, 2023-11-05T08:33:44.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01338255, 2023-11-05T08:33:44.822 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:44.822 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:44.822 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:44.823 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:44.823 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:44.823 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 379 2023-11-05T08:33:44.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.823 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:44.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:44.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:10.278322+0000", 2023-11-05T08:33:44.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:44.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:10.278322+0000", 2023-11-05T08:33:44.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:44.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:10.278322+0000", 2023-11-05T08:33:44.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3000000000000001e-07 2023-11-05T08:33:44.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:44.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:10.278326+0000", 2023-11-05T08:33:44.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4139999999999998e-06 2023-11-05T08:33:44.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:44.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:10.278335+0000", 2023-11-05T08:33:44.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8030000000000008e-06 2023-11-05T08:33:44.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:10.278341+0000", 2023-11-05T08:33:44.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6559999999999997e-06 2023-11-05T08:33:44.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:44.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:10.278660+0000", 2023-11-05T08:33:44.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031871500000000001 2023-11-05T08:33:44.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:33:44.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:10.278687+0000", 2023-11-05T08:33:44.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6829e-05 2023-11-05T08:33:44.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:44.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:10.285854+0000", 2023-11-05T08:33:44.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0071673539999999999 2023-11-05T08:33:44.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:44.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:10.286007+0000", 2023-11-05T08:33:44.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000152583 2023-11-05T08:33:44.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:44.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:10.291704+0000", 2023-11-05T08:33:44.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0056977659999999999 2023-11-05T08:33:44.834 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.834 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:44.834 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.835 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:405 3.as1 3:5396e455:::smithi09039252-43:head [append 0~622592] snapc 1b=[1b,1a,17,7] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2023-11-05T08:33:44.835 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:14.355514+0000", 2023-11-05T08:33:44.835 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.349800093999999, 2023-11-05T08:33:44.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022581047, 2023-11-05T08:33:44.835 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:44.836 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:44.836 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:44.836 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:44.836 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:44.836 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 405 2023-11-05T08:33:44.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.837 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:44.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:44.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.355514+0000", 2023-11-05T08:33:44.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:44.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.355514+0000", 2023-11-05T08:33:44.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:44.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.355517+0000", 2023-11-05T08:33:44.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7930000000000002e-06 2023-11-05T08:33:44.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:44.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.356220+0000", 2023-11-05T08:33:44.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00070250200000000003 2023-11-05T08:33:44.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:44.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.356253+0000", 2023-11-05T08:33:44.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3719e-05 2023-11-05T08:33:44.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.356268+0000", 2023-11-05T08:33:44.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5033e-05 2023-11-05T08:33:44.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:44.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.356444+0000", 2023-11-05T08:33:44.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000175549 2023-11-05T08:33:44.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-11-05T08:33:44.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.356471+0000", 2023-11-05T08:33:44.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6755999999999999e-05 2023-11-05T08:33:44.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:44.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.361462+0000", 2023-11-05T08:33:44.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049916350000000003 2023-11-05T08:33:44.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:44.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.361764+0000", 2023-11-05T08:33:44.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030161199999999998 2023-11-05T08:33:44.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:44.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.365071+0000", 2023-11-05T08:33:44.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033064940000000001 2023-11-05T08:33:44.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:44.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.377076+0000", 2023-11-05T08:33:44.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012005969999999999 2023-11-05T08:33:44.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:44.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.377756+0000", 2023-11-05T08:33:44.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067910900000000001 2023-11-05T08:33:44.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:44.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.378095+0000", 2023-11-05T08:33:44.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033987500000000001 2023-11-05T08:33:44.850 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.850 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:44.850 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.851 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:406 3.as1 3:5396e455:::smithi09039252-43:head [append 0~475136] snapc 1b=[1b,1a,17,7] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2023-11-05T08:33:44.851 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:14.363708+0000", 2023-11-05T08:33:44.851 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.341606474999999, 2023-11-05T08:33:44.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018912735999999999, 2023-11-05T08:33:44.851 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:44.852 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:44.852 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:44.852 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:44.852 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:44.852 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 406 2023-11-05T08:33:44.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.853 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:44.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:44.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.363708+0000", 2023-11-05T08:33:44.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:44.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.363708+0000", 2023-11-05T08:33:44.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:44.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.363710+0000", 2023-11-05T08:33:44.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3700000000000002e-06 2023-11-05T08:33:44.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:44.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.364282+0000", 2023-11-05T08:33:44.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057111800000000002 2023-11-05T08:33:44.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:44.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.364315+0000", 2023-11-05T08:33:44.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3868999999999997e-05 2023-11-05T08:33:44.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.364331+0000", 2023-11-05T08:33:44.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5130000000000001e-05 2023-11-05T08:33:44.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:44.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.369348+0000", 2023-11-05T08:33:44.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050178250000000001 2023-11-05T08:33:44.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:44.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.369681+0000", 2023-11-05T08:33:44.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033315800000000001 2023-11-05T08:33:44.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:44.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.371729+0000", 2023-11-05T08:33:44.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020475340000000002 2023-11-05T08:33:44.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:44.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.377079+0000", 2023-11-05T08:33:44.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053501529999999999 2023-11-05T08:33:44.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:44.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.382367+0000", 2023-11-05T08:33:44.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052874100000000002 2023-11-05T08:33:44.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:44.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.382621+0000", 2023-11-05T08:33:44.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000254169 2023-11-05T08:33:44.864 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.864 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:44.864 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.865 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:407 3.as1 3:5396e455:::smithi09039252-43:head [append 0~294912] snapc 1b=[1b,1a,17,7] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2023-11-05T08:33:44.865 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:14.369667+0000", 2023-11-05T08:33:44.865 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.335647285, 2023-11-05T08:33:44.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013804643, 2023-11-05T08:33:44.865 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:44.866 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:44.866 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:44.866 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:44.866 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:44.866 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 407 2023-11-05T08:33:44.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.867 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:44.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:44.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.369667+0000", 2023-11-05T08:33:44.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:44.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.369667+0000", 2023-11-05T08:33:44.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:44.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.369670+0000", 2023-11-05T08:33:44.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.044e-06 2023-11-05T08:33:44.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:44.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.369905+0000", 2023-11-05T08:33:44.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023446400000000001 2023-11-05T08:33:44.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:44.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.369940+0000", 2023-11-05T08:33:44.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5159e-05 2023-11-05T08:33:44.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.369963+0000", 2023-11-05T08:33:44.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3575999999999999e-05 2023-11-05T08:33:44.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:44.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.373576+0000", 2023-11-05T08:33:44.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036127709999999999 2023-11-05T08:33:44.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:44.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.373808+0000", 2023-11-05T08:33:44.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023190999999999999 2023-11-05T08:33:44.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:44.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.375278+0000", 2023-11-05T08:33:44.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014693989999999999 2023-11-05T08:33:44.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:44.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.378341+0000", 2023-11-05T08:33:44.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030631880000000001 2023-11-05T08:33:44.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:44.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.383024+0000", 2023-11-05T08:33:44.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0046831959999999997 2023-11-05T08:33:44.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:44.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.383472+0000", 2023-11-05T08:33:44.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044793600000000002 2023-11-05T08:33:44.878 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.878 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:44.878 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.879 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:408 3.as1 3:5396e455:::smithi09039252-43:head [setxattr _header (57) in=7b] snapc 1b=[1b,1a,17,7] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2023-11-05T08:33:44.879 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:14.370011+0000", 2023-11-05T08:33:44.879 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.335303097000001, 2023-11-05T08:33:44.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016422719999999998, 2023-11-05T08:33:44.880 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:44.880 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:44.880 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:44.880 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:44.880 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:44.881 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 408 2023-11-05T08:33:44.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.881 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:44.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:44.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.370011+0000", 2023-11-05T08:33:44.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:44.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.370011+0000", 2023-11-05T08:33:44.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:44.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.370013+0000", 2023-11-05T08:33:44.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7370000000000001e-06 2023-11-05T08:33:44.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:44.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.370024+0000", 2023-11-05T08:33:44.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0976e-05 2023-11-05T08:33:44.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:44.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.370046+0000", 2023-11-05T08:33:44.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2065999999999998e-05 2023-11-05T08:33:44.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.370062+0000", 2023-11-05T08:33:44.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5407e-05 2023-11-05T08:33:44.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:44.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.378961+0000", 2023-11-05T08:33:44.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0088990579999999996 2023-11-05T08:33:44.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:44.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.379295+0000", 2023-11-05T08:33:44.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033471100000000001 2023-11-05T08:33:44.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:44.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.380741+0000", 2023-11-05T08:33:44.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014457129999999999 2023-11-05T08:33:44.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:44.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.383921+0000", 2023-11-05T08:33:44.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031802660000000002 2023-11-05T08:33:44.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:44.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.386122+0000", 2023-11-05T08:33:44.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022002219999999999 2023-11-05T08:33:44.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:44.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.386434+0000", 2023-11-05T08:33:44.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031256399999999998 2023-11-05T08:33:44.892 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.892 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:44.892 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.893 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:409 3.as1 3:5396e455:::smithi09039252-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e61)", 2023-11-05T08:33:44.893 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:14.370085+0000", 2023-11-05T08:33:44.893 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.335229145, 2023-11-05T08:33:44.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019674614999999999, 2023-11-05T08:33:44.893 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:44.893 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:44.894 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:44.894 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:44.894 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:44.894 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 409 2023-11-05T08:33:44.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.895 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:44.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:44.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.370085+0000", 2023-11-05T08:33:44.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:44.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.370085+0000", 2023-11-05T08:33:44.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:44.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.370087+0000", 2023-11-05T08:33:44.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2729999999999999e-06 2023-11-05T08:33:44.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:44.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.370093+0000", 2023-11-05T08:33:44.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0700000000000003e-06 2023-11-05T08:33:44.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:44.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.370113+0000", 2023-11-05T08:33:44.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0276e-05 2023-11-05T08:33:44.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.370132+0000", 2023-11-05T08:33:44.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8626000000000001e-05 2023-11-05T08:33:44.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:44.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.383576+0000", 2023-11-05T08:33:44.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013444717 2023-11-05T08:33:44.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-11-05T08:33:44.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.383807+0000", 2023-11-05T08:33:44.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023070100000000001 2023-11-05T08:33:44.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:44.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.387071+0000", 2023-11-05T08:33:44.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032641710000000002 2023-11-05T08:33:44.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:44.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.387425+0000", 2023-11-05T08:33:44.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035352400000000003 2023-11-05T08:33:44.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:44.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.389760+0000", 2023-11-05T08:33:44.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023352569999999999 2023-11-05T08:33:44.905 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.905 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:44.905 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.906 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:570 3.ds0 3:bf7e600d:::smithi09039252-23:head [append 0~0] snapc 27=[7] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2023-11-05T08:33:44.906 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:29.487235+0000", 2023-11-05T08:33:44.906 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.218079470999999, 2023-11-05T08:33:44.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013670231, 2023-11-05T08:33:44.907 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:44.907 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:44.907 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:44.907 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:44.907 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:44.908 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 570 2023-11-05T08:33:44.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.908 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:44.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:44.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487235+0000", 2023-11-05T08:33:44.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:44.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487235+0000", 2023-11-05T08:33:44.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:44.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487238+0000", 2023-11-05T08:33:44.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.61e-06 2023-11-05T08:33:44.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:44.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487249+0000", 2023-11-05T08:33:44.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1379999999999999e-05 2023-11-05T08:33:44.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:44.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487283+0000", 2023-11-05T08:33:44.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3538000000000003e-05 2023-11-05T08:33:44.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487301+0000", 2023-11-05T08:33:44.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8122999999999999e-05 2023-11-05T08:33:44.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:44.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487463+0000", 2023-11-05T08:33:44.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016224300000000001 2023-11-05T08:33:44.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-11-05T08:33:44.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487480+0000", 2023-11-05T08:33:44.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7085999999999999e-05 2023-11-05T08:33:44.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:44.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.493331+0000", 2023-11-05T08:33:44.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0058510599999999999 2023-11-05T08:33:44.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:44.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.493515+0000", 2023-11-05T08:33:44.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018418899999999999 2023-11-05T08:33:44.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:44.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.494556+0000", 2023-11-05T08:33:44.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0010403529999999999 2023-11-05T08:33:44.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:44.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.499777+0000", 2023-11-05T08:33:44.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.005221102 2023-11-05T08:33:44.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:44.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.500424+0000", 2023-11-05T08:33:44.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064702 2023-11-05T08:33:44.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:44.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.500905+0000", 2023-11-05T08:33:44.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00048152699999999999 2023-11-05T08:33:44.921 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.921 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:44.921 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.922 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:571 3.ds0 3:bf7e600d:::smithi09039252-23:head [rollback 7] snapc 27=[7] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2023-11-05T08:33:44.922 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:29.487342+0000", 2023-11-05T08:33:44.922 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.217972849000001, 2023-11-05T08:33:44.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017607034000000001, 2023-11-05T08:33:44.922 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:44.923 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:44.923 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:44.923 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:44.923 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:44.923 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 571 2023-11-05T08:33:44.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.924 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:44.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:44.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487342+0000", 2023-11-05T08:33:44.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:44.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487342+0000", 2023-11-05T08:33:44.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:44.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487343+0000", 2023-11-05T08:33:44.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1699999999999997e-07 2023-11-05T08:33:44.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.927 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:44.927 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487347+0000", 2023-11-05T08:33:44.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7099999999999998e-06 2023-11-05T08:33:44.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:44.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487362+0000", 2023-11-05T08:33:44.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4639e-05 2023-11-05T08:33:44.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487371+0000", 2023-11-05T08:33:44.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7700000000000007e-06 2023-11-05T08:33:44.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:44.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487637+0000", 2023-11-05T08:33:44.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026612599999999999 2023-11-05T08:33:44.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:33:44.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487655+0000", 2023-11-05T08:33:44.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7691999999999999e-05 2023-11-05T08:33:44.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:44.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.495233+0000", 2023-11-05T08:33:44.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0075784720000000002 2023-11-05T08:33:44.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:44.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.495448+0000", 2023-11-05T08:33:44.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000215274 2023-11-05T08:33:44.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:44.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.496724+0000", 2023-11-05T08:33:44.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012761669999999999 2023-11-05T08:33:44.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:44.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.499780+0000", 2023-11-05T08:33:44.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030555449999999998 2023-11-05T08:33:44.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:44.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.504307+0000", 2023-11-05T08:33:44.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0045266990000000003 2023-11-05T08:33:44.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:44.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.504949+0000", 2023-11-05T08:33:44.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064202300000000003 2023-11-05T08:33:44.937 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.937 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:44.937 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.938 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:572 3.ds0 3:bf7e600d:::smithi09039252-23:head [append 0~0] snapc 27=[7] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2023-11-05T08:33:44.938 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:29.487466+0000", 2023-11-05T08:33:44.938 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.21784881, 2023-11-05T08:33:44.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018798550000000001, 2023-11-05T08:33:44.938 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:44.938 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:44.939 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:44.939 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:44.939 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:44.939 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 572 2023-11-05T08:33:44.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.940 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:44.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:44.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487466+0000", 2023-11-05T08:33:44.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:44.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487466+0000", 2023-11-05T08:33:44.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:44.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487466+0000", 2023-11-05T08:33:44.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9599999999999998e-07 2023-11-05T08:33:44.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:44.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487471+0000", 2023-11-05T08:33:44.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2139999999999998e-06 2023-11-05T08:33:44.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:44.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487483+0000", 2023-11-05T08:33:44.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2048999999999999e-05 2023-11-05T08:33:44.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487496+0000", 2023-11-05T08:33:44.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2819999999999999e-05 2023-11-05T08:33:44.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:44.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487722+0000", 2023-11-05T08:33:44.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022617000000000001 2023-11-05T08:33:44.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:33:44.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487738+0000", 2023-11-05T08:33:44.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6303e-05 2023-11-05T08:33:44.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:44.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.497597+0000", 2023-11-05T08:33:44.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098588319999999997 2023-11-05T08:33:44.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:44.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.497825+0000", 2023-11-05T08:33:44.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022776099999999999 2023-11-05T08:33:44.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:44.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.498970+0000", 2023-11-05T08:33:44.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001145622 2023-11-05T08:33:44.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:44.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.501416+0000", 2023-11-05T08:33:44.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002445268 2023-11-05T08:33:44.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:44.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.505581+0000", 2023-11-05T08:33:44.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0041650699999999999 2023-11-05T08:33:44.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:44.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.506264+0000", 2023-11-05T08:33:44.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00068364499999999998 2023-11-05T08:33:44.953 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.953 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:44.953 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.953 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:614 3.ds0 3:bf7e600d:::smithi09039252-23:head [read 0~1392640,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e81)", 2023-11-05T08:33:44.954 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:34.541363+0000", 2023-11-05T08:33:44.954 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 10.163951186, 2023-11-05T08:33:44.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014792108, 2023-11-05T08:33:44.954 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:44.954 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:44.955 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:44.955 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:44.955 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:44.955 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 614 2023-11-05T08:33:44.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.956 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:44.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:44.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541363+0000", 2023-11-05T08:33:44.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:44.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541363+0000", 2023-11-05T08:33:44.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:44.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541366+0000", 2023-11-05T08:33:44.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2469999999999998e-06 2023-11-05T08:33:44.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:44.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541383+0000", 2023-11-05T08:33:44.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7348999999999999e-05 2023-11-05T08:33:44.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:44.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541416+0000", 2023-11-05T08:33:44.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2672000000000002e-05 2023-11-05T08:33:44.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541439+0000", 2023-11-05T08:33:44.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3465999999999999e-05 2023-11-05T08:33:44.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:44.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541591+0000", 2023-11-05T08:33:44.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015148200000000001 2023-11-05T08:33:44.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-11-05T08:33:44.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541607+0000", 2023-11-05T08:33:44.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6328999999999999e-05 2023-11-05T08:33:44.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:44.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.547709+0000", 2023-11-05T08:33:44.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061023279999999997 2023-11-05T08:33:44.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:44.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.548073+0000", 2023-11-05T08:33:44.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036379400000000002 2023-11-05T08:33:44.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:44.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.556155+0000", 2023-11-05T08:33:44.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0080824409999999992 2023-11-05T08:33:44.966 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.966 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:44.966 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.967 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:615 3.ds0 3:bf7e600d:::smithi09039252-23:head [read 0~1392640,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e81)", 2023-11-05T08:33:44.967 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:34.541498+0000", 2023-11-05T08:33:44.967 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 10.163816904000001, 2023-11-05T08:33:44.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01790396, 2023-11-05T08:33:44.967 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:44.968 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:44.968 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:44.968 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:44.968 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:44.968 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 615 2023-11-05T08:33:44.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.969 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:44.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:44.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541498+0000", 2023-11-05T08:33:44.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:44.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541498+0000", 2023-11-05T08:33:44.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:44.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541499+0000", 2023-11-05T08:33:44.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.623e-06 2023-11-05T08:33:44.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:44.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541510+0000", 2023-11-05T08:33:44.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0366999999999999e-05 2023-11-05T08:33:44.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:44.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541532+0000", 2023-11-05T08:33:44.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2762e-05 2023-11-05T08:33:44.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541549+0000", 2023-11-05T08:33:44.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6946e-05 2023-11-05T08:33:44.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:44.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541716+0000", 2023-11-05T08:33:44.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016715100000000001 2023-11-05T08:33:44.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:33:44.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541732+0000", 2023-11-05T08:33:44.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5506e-05 2023-11-05T08:33:44.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:44.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.549010+0000", 2023-11-05T08:33:44.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0072776270000000001 2023-11-05T08:33:44.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:44.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.549197+0000", 2023-11-05T08:33:44.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000187646 2023-11-05T08:33:44.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:44.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.559402+0000", 2023-11-05T08:33:44.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010204332 2023-11-05T08:33:44.979 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.979 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:44.979 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.980 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:616 3.ds0 3:bf7e600d:::smithi09039252-23:head [read 0~1392640,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e81)", 2023-11-05T08:33:44.980 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:34.541580+0000", 2023-11-05T08:33:44.980 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 10.163734570000001, 2023-11-05T08:33:44.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020950334000000001, 2023-11-05T08:33:44.980 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:44.981 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:44.981 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:44.981 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:44.981 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:44.981 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 616 2023-11-05T08:33:44.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.982 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:44.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:44.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541580+0000", 2023-11-05T08:33:44.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:44.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541580+0000", 2023-11-05T08:33:44.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:44.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:44.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541581+0000", 2023-11-05T08:33:44.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.226e-06 2023-11-05T08:33:44.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:44.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541609+0000", 2023-11-05T08:33:44.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7645999999999999e-05 2023-11-05T08:33:44.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:44.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541629+0000", 2023-11-05T08:33:44.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0186e-05 2023-11-05T08:33:44.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:44.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541643+0000", 2023-11-05T08:33:44.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.428e-05 2023-11-05T08:33:44.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:44.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541783+0000", 2023-11-05T08:33:44.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013925799999999999 2023-11-05T08:33:44.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:33:44.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541796+0000", 2023-11-05T08:33:44.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3426e-05 2023-11-05T08:33:44.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:44.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.556302+0000", 2023-11-05T08:33:44.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014506022 2023-11-05T08:33:44.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:44.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.556475+0000", 2023-11-05T08:33:44.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017302999999999999 2023-11-05T08:33:44.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:44.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:44.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:44.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.562530+0000", 2023-11-05T08:33:44.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0060552599999999998 2023-11-05T08:33:44.992 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.992 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:44.992 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.992 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:44.993 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:44.993 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:33:44.994 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2023-11-05T08:33:44.995 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:33:44.992+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:33:45.097 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:33:45.093+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:33:45.129 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:33:45.130 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-11-05T08:33:45.130 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-11-05T08:33:45.130 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:33:45.144 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2023-11-05T08:33:45.196 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:33:45.193+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:33:45.269 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:33:45.270 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-11-05T08:33:45.270 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-11-05T08:33:45.270 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-11-05T08:33:45.271 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:33:45.284 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2023-11-05T08:33:45.288 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-11-05T08:33:45.286+0000 7f72b5ca2640 -1 osd.3 43 log_to_monitors true 2023-11-05T08:33:45.297 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:45.294+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:45.399 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:33:45.399 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-11-05T08:33:45.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-11-05T08:33:45.400 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-11-05T08:33:45.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.400 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.as6 42/33 ECSubWrite(tid=38, reqid=osd.7.0:38, at_version=42'16, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:33:45.400 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:55.122849+0000", 2023-11-05T08:33:45.400 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.273576654000003, 2023-11-05T08:33:45.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015960385000000001, 2023-11-05T08:33:45.401 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:45.401 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:45.401 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:45.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:45.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.122849+0000", 2023-11-05T08:33:45.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:45.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.122849+0000", 2023-11-05T08:33:45.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:45.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.122850+0000", 2023-11-05T08:33:45.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1699999999999997e-07 2023-11-05T08:33:45.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:45.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.122895+0000", 2023-11-05T08:33:45.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5717999999999998e-05 2023-11-05T08:33:45.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:45.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.122919+0000", 2023-11-05T08:33:45.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3932000000000001e-05 2023-11-05T08:33:45.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:45.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.122941+0000", 2023-11-05T08:33:45.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1708999999999998e-05 2023-11-05T08:33:45.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.407 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:45.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.125137+0000", 2023-11-05T08:33:45.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021962219999999998 2023-11-05T08:33:45.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:45.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.125161+0000", 2023-11-05T08:33:45.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3436000000000001e-05 2023-11-05T08:33:45.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:45.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.138719+0000", 2023-11-05T08:33:45.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013558115000000001 2023-11-05T08:33:45.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:45.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.138809+0000", 2023-11-05T08:33:45.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0536000000000002e-05 2023-11-05T08:33:45.411 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.411 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:45.411 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.411 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.as6 42/33 ECSubWrite(tid=39, reqid=osd.7.0:39, at_version=42'18, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:33:45.412 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:55.125666+0000", 2023-11-05T08:33:45.412 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.270759495, 2023-11-05T08:33:45.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013940509, 2023-11-05T08:33:45.412 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:45.412 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:45.412 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:45.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:45.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.125666+0000", 2023-11-05T08:33:45.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:45.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.125666+0000", 2023-11-05T08:33:45.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:45.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.125667+0000", 2023-11-05T08:33:45.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.47e-07 2023-11-05T08:33:45.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:45.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.125713+0000", 2023-11-05T08:33:45.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5748000000000003e-05 2023-11-05T08:33:45.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:45.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.125737+0000", 2023-11-05T08:33:45.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4369e-05 2023-11-05T08:33:45.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:45.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.125758+0000", 2023-11-05T08:33:45.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0608999999999999e-05 2023-11-05T08:33:45.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:45.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.126227+0000", 2023-11-05T08:33:45.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000468944 2023-11-05T08:33:45.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:45.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.126251+0000", 2023-11-05T08:33:45.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4788999999999999e-05 2023-11-05T08:33:45.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:45.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.139523+0000", 2023-11-05T08:33:45.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013271577 2023-11-05T08:33:45.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:45.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.139607+0000", 2023-11-05T08:33:45.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3526e-05 2023-11-05T08:33:45.421 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.422 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:45.422 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.422 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:346 3.fs0 3:f2348bf8:::smithi09039252-39:head [read 0~638976,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:45.423 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:05.254913+0000", 2023-11-05T08:33:45.423 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.141512278, 2023-11-05T08:33:45.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019954645999999999, 2023-11-05T08:33:45.423 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:45.423 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:45.424 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:45.424 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:45.424 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:45.424 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 346 2023-11-05T08:33:45.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.424 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:45.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:45.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.254913+0000", 2023-11-05T08:33:45.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:45.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.254913+0000", 2023-11-05T08:33:45.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:45.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.254917+0000", 2023-11-05T08:33:45.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1330000000000001e-06 2023-11-05T08:33:45.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:45.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.254934+0000", 2023-11-05T08:33:45.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6897000000000001e-05 2023-11-05T08:33:45.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:45.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.254965+0000", 2023-11-05T08:33:45.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0769000000000003e-05 2023-11-05T08:33:45.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:45.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.254981+0000", 2023-11-05T08:33:45.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6015000000000002e-05 2023-11-05T08:33:45.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:45.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255140+0000", 2023-11-05T08:33:45.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015884499999999999 2023-11-05T08:33:45.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-11-05T08:33:45.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255161+0000", 2023-11-05T08:33:45.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1033e-05 2023-11-05T08:33:45.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:45.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.261568+0000", 2023-11-05T08:33:45.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0064071639999999999 2023-11-05T08:33:45.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:45.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.261728+0000", 2023-11-05T08:33:45.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015980500000000001 2023-11-05T08:33:45.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:45.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.274868+0000", 2023-11-05T08:33:45.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013139985 2023-11-05T08:33:45.435 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.435 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:45.435 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.436 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:347 3.fs0 3:f2348bf8:::smithi09039252-39:head [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:45.436 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:05.255019+0000", 2023-11-05T08:33:45.436 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.141406060999998, 2023-11-05T08:33:45.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021017214999999999, 2023-11-05T08:33:45.436 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:45.437 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:45.437 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:45.437 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:45.437 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:45.437 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 347 2023-11-05T08:33:45.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.438 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:45.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:45.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255019+0000", 2023-11-05T08:33:45.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:45.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255019+0000", 2023-11-05T08:33:45.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:45.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255021+0000", 2023-11-05T08:33:45.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1260000000000001e-06 2023-11-05T08:33:45.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:45.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255026+0000", 2023-11-05T08:33:45.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8640000000000001e-06 2023-11-05T08:33:45.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:45.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255039+0000", 2023-11-05T08:33:45.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2238999999999999e-05 2023-11-05T08:33:45.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:45.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255047+0000", 2023-11-05T08:33:45.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7159999999999992e-06 2023-11-05T08:33:45.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:45.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255354+0000", 2023-11-05T08:33:45.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030627999999999997 2023-11-05T08:33:45.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:33:45.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255382+0000", 2023-11-05T08:33:45.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8439e-05 2023-11-05T08:33:45.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:45.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.266020+0000", 2023-11-05T08:33:45.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010637807000000001 2023-11-05T08:33:45.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:45.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.266295+0000", 2023-11-05T08:33:45.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027543400000000002 2023-11-05T08:33:45.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:45.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.276037+0000", 2023-11-05T08:33:45.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0097413099999999996 2023-11-05T08:33:45.451 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.451 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:45.451 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.451 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:348 3.fs0 3:f2348bf8:::smithi09039252-39:head [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:45.452 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:05.255103+0000", 2023-11-05T08:33:45.452 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.141322103999997, 2023-11-05T08:33:45.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022537980999999999, 2023-11-05T08:33:45.452 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:45.452 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:45.453 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:45.453 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:45.453 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:45.453 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 348 2023-11-05T08:33:45.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.454 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:45.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:45.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255103+0000", 2023-11-05T08:33:45.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:45.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255103+0000", 2023-11-05T08:33:45.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:45.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255105+0000", 2023-11-05T08:33:45.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.24e-06 2023-11-05T08:33:45.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:45.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255136+0000", 2023-11-05T08:33:45.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0979000000000003e-05 2023-11-05T08:33:45.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:45.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255164+0000", 2023-11-05T08:33:45.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8662000000000001e-05 2023-11-05T08:33:45.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:45.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255181+0000", 2023-11-05T08:33:45.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6935999999999998e-05 2023-11-05T08:33:45.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:45.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255509+0000", 2023-11-05T08:33:45.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032754300000000002 2023-11-05T08:33:45.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:33:45.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255536+0000", 2023-11-05T08:33:45.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6889e-05 2023-11-05T08:33:45.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:45.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.267115+0000", 2023-11-05T08:33:45.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011579104 2023-11-05T08:33:45.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.462 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:45.462 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.267222+0000", 2023-11-05T08:33:45.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106864 2023-11-05T08:33:45.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:45.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.277641+0000", 2023-11-05T08:33:45.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010419764 2023-11-05T08:33:45.464 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.464 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:45.464 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.465 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:369 3.16s0 3:6df625fb:::smithi09039252-31:head [append 0~524288] snapc 15=[7] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2023-11-05T08:33:45.465 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:09.281473+0000", 2023-11-05T08:33:45.465 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 36.114952377000002, 2023-11-05T08:33:45.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020569061999999999, 2023-11-05T08:33:45.465 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:45.465 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:45.466 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:45.466 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:45.466 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:45.466 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 369 2023-11-05T08:33:45.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.467 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:45.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:45.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.281473+0000", 2023-11-05T08:33:45.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:45.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.281473+0000", 2023-11-05T08:33:45.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:45.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.281477+0000", 2023-11-05T08:33:45.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.98e-06 2023-11-05T08:33:45.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:45.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.282265+0000", 2023-11-05T08:33:45.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00078817999999999998 2023-11-05T08:33:45.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:45.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.282308+0000", 2023-11-05T08:33:45.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2775999999999998e-05 2023-11-05T08:33:45.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:45.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.282331+0000", 2023-11-05T08:33:45.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3332999999999998e-05 2023-11-05T08:33:45.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:45.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.282525+0000", 2023-11-05T08:33:45.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000193671 2023-11-05T08:33:45.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-11-05T08:33:45.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.282555+0000", 2023-11-05T08:33:45.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9669e-05 2023-11-05T08:33:45.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:45.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.287913+0000", 2023-11-05T08:33:45.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053581360000000003 2023-11-05T08:33:45.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:45.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.288217+0000", 2023-11-05T08:33:45.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030429900000000001 2023-11-05T08:33:45.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:45.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.291625+0000", 2023-11-05T08:33:45.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034076779999999999 2023-11-05T08:33:45.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:45.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.301060+0000", 2023-11-05T08:33:45.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0094351580000000008 2023-11-05T08:33:45.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:45.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.301904+0000", 2023-11-05T08:33:45.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00084351399999999996 2023-11-05T08:33:45.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:45.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.302042+0000", 2023-11-05T08:33:45.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013866799999999999 2023-11-05T08:33:45.479 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.480 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:45.480 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.480 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:370 3.16s0 3:6df625fb:::smithi09039252-31:head [append 0~524288] snapc 15=[7] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2023-11-05T08:33:45.481 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:09.290980+0000", 2023-11-05T08:33:45.481 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 36.105445058999997, 2023-11-05T08:33:45.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019531185999999999, 2023-11-05T08:33:45.481 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:45.481 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:45.481 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:45.482 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:45.482 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:45.482 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 370 2023-11-05T08:33:45.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.482 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:45.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:45.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.290980+0000", 2023-11-05T08:33:45.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:45.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.290980+0000", 2023-11-05T08:33:45.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:45.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.290983+0000", 2023-11-05T08:33:45.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9560000000000002e-06 2023-11-05T08:33:45.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:45.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.291841+0000", 2023-11-05T08:33:45.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00085721599999999995 2023-11-05T08:33:45.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:45.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.291879+0000", 2023-11-05T08:33:45.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7882000000000001e-05 2023-11-05T08:33:45.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:45.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.291904+0000", 2023-11-05T08:33:45.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5658999999999999e-05 2023-11-05T08:33:45.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:45.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.294143+0000", 2023-11-05T08:33:45.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022388830000000001 2023-11-05T08:33:45.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:45.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.294499+0000", 2023-11-05T08:33:45.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035557600000000003 2023-11-05T08:33:45.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:45.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.296802+0000", 2023-11-05T08:33:45.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023036509999999999 2023-11-05T08:33:45.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:45.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.302059+0000", 2023-11-05T08:33:45.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052565379999999998 2023-11-05T08:33:45.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:45.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.310264+0000", 2023-11-05T08:33:45.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0082055919999999994 2023-11-05T08:33:45.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:45.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.310512+0000", 2023-11-05T08:33:45.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024723300000000002 2023-11-05T08:33:45.494 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.494 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:45.494 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.494 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:371 3.16s0 3:6df625fb:::smithi09039252-31:head [append 0~245760] snapc 15=[7] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2023-11-05T08:33:45.495 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:09.296540+0000", 2023-11-05T08:33:45.495 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 36.099885149000002, 2023-11-05T08:33:45.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015979499000000001, 2023-11-05T08:33:45.495 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:45.495 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:45.496 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:45.496 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:45.496 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:45.496 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 371 2023-11-05T08:33:45.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.496 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:45.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:45.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.296540+0000", 2023-11-05T08:33:45.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:45.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.296540+0000", 2023-11-05T08:33:45.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:45.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.296543+0000", 2023-11-05T08:33:45.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8329999999999998e-06 2023-11-05T08:33:45.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:45.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.296864+0000", 2023-11-05T08:33:45.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032048100000000001 2023-11-05T08:33:45.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:45.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.296899+0000", 2023-11-05T08:33:45.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5735999999999998e-05 2023-11-05T08:33:45.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:45.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.296920+0000", 2023-11-05T08:33:45.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0268999999999999e-05 2023-11-05T08:33:45.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:45.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.302604+0000", 2023-11-05T08:33:45.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0056841879999999997 2023-11-05T08:33:45.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:45.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.302966+0000", 2023-11-05T08:33:45.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036163200000000002 2023-11-05T08:33:45.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:45.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.304745+0000", 2023-11-05T08:33:45.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001779596 2023-11-05T08:33:45.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:45.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.309738+0000", 2023-11-05T08:33:45.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049929500000000003 2023-11-05T08:33:45.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:45.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.312278+0000", 2023-11-05T08:33:45.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002540231 2023-11-05T08:33:45.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:45.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.312520+0000", 2023-11-05T08:33:45.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024158299999999999 2023-11-05T08:33:45.507 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.507 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:45.508 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.508 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:372 3.16s0 3:6df625fb:::smithi09039252-31:head [setxattr _header (57) in=7b] snapc 15=[7] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2023-11-05T08:33:45.508 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:09.296991+0000", 2023-11-05T08:33:45.509 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 36.099434944999999, 2023-11-05T08:33:45.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016671208, 2023-11-05T08:33:45.509 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:45.509 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:45.509 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:45.509 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:45.510 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:45.510 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 372 2023-11-05T08:33:45.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.510 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:45.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:45.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.296991+0000", 2023-11-05T08:33:45.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:45.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.296991+0000", 2023-11-05T08:33:45.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:45.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.296992+0000", 2023-11-05T08:33:45.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5230000000000001e-06 2023-11-05T08:33:45.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:45.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297001+0000", 2023-11-05T08:33:45.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1570000000000006e-06 2023-11-05T08:33:45.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:45.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297023+0000", 2023-11-05T08:33:45.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1965999999999999e-05 2023-11-05T08:33:45.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:45.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297074+0000", 2023-11-05T08:33:45.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0875000000000003e-05 2023-11-05T08:33:45.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:45.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.306255+0000", 2023-11-05T08:33:45.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0091812700000000001 2023-11-05T08:33:45.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:45.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.306557+0000", 2023-11-05T08:33:45.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030117799999999998 2023-11-05T08:33:45.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:45.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.308901+0000", 2023-11-05T08:33:45.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023442990000000002 2023-11-05T08:33:45.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:45.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.312840+0000", 2023-11-05T08:33:45.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039388670000000004 2023-11-05T08:33:45.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:45.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.313428+0000", 2023-11-05T08:33:45.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058814000000000002 2023-11-05T08:33:45.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:45.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.313662+0000", 2023-11-05T08:33:45.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000233933 2023-11-05T08:33:45.521 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.522 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:45.522 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.522 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:373 3.16s0 3:6df625fb:::smithi09039252-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e56)", 2023-11-05T08:33:45.523 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:09.297100+0000", 2023-11-05T08:33:45.523 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 36.099325301999997, 2023-11-05T08:33:45.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020900037999999999, 2023-11-05T08:33:45.523 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:45.523 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:45.523 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:45.524 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:45.524 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:45.524 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 373 2023-11-05T08:33:45.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.524 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:45.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:45.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297100+0000", 2023-11-05T08:33:45.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:45.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297100+0000", 2023-11-05T08:33:45.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:45.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297102+0000", 2023-11-05T08:33:45.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.454e-06 2023-11-05T08:33:45.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:45.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297108+0000", 2023-11-05T08:33:45.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8260000000000004e-06 2023-11-05T08:33:45.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:45.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297131+0000", 2023-11-05T08:33:45.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2042999999999999e-05 2023-11-05T08:33:45.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:45.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297145+0000", 2023-11-05T08:33:45.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4783000000000001e-05 2023-11-05T08:33:45.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:45.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.314858+0000", 2023-11-05T08:33:45.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017712525 2023-11-05T08:33:45.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:45.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.315292+0000", 2023-11-05T08:33:45.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043406100000000002 2023-11-05T08:33:45.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:45.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.318000+0000", 2023-11-05T08:33:45.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0027083459999999999 2023-11-05T08:33:45.533 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.533 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:45.533 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.534 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:493 3.fs0 3:f9a1f673:::smithi09039252-46:head [copy-from ver 2] snapc 1c=[1a,7] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2023-11-05T08:33:45.534 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:18.605410+0000", 2023-11-05T08:33:45.534 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 26.791015113, 2023-11-05T08:33:45.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015750235000000001, 2023-11-05T08:33:45.534 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:45.534 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:45.535 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:45.535 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:45.535 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:45.535 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 493 2023-11-05T08:33:45.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.536 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:45.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:45.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.605410+0000", 2023-11-05T08:33:45.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:45.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.605410+0000", 2023-11-05T08:33:45.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:45.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.605411+0000", 2023-11-05T08:33:45.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0100000000000005e-07 2023-11-05T08:33:45.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:45.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.605414+0000", 2023-11-05T08:33:45.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5039999999999998e-06 2023-11-05T08:33:45.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:45.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.605423+0000", 2023-11-05T08:33:45.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6759999999999996e-06 2023-11-05T08:33:45.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:45.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.605429+0000", 2023-11-05T08:33:45.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.93e-06 2023-11-05T08:33:45.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:45.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.609936+0000", 2023-11-05T08:33:45.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0045067650000000002 2023-11-05T08:33:45.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:45.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.610130+0000", 2023-11-05T08:33:45.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019458499999999999 2023-11-05T08:33:45.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:45.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.616944+0000", 2023-11-05T08:33:45.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0068134689999999999 2023-11-05T08:33:45.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:45.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.619996+0000", 2023-11-05T08:33:45.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030521609999999998 2023-11-05T08:33:45.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:45.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.621010+0000", 2023-11-05T08:33:45.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001014048 2023-11-05T08:33:45.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:45.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.621161+0000", 2023-11-05T08:33:45.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015059599999999999 2023-11-05T08:33:45.546 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.547 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:45.547 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.547 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:494 3.fs0 3:f9a1f673:::smithi09039252-46:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2023-11-05T08:33:45.548 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:18.605437+0000", 2023-11-05T08:33:45.548 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 26.790988045999999, 2023-11-05T08:33:45.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014062043, 2023-11-05T08:33:45.548 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:45.548 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:45.548 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:45.549 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:45.549 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:45.549 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 494 2023-11-05T08:33:45.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.549 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:45.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:45.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.605437+0000", 2023-11-05T08:33:45.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:45.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.605437+0000", 2023-11-05T08:33:45.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:45.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.605438+0000", 2023-11-05T08:33:45.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2300000000000001e-07 2023-11-05T08:33:45.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:45.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.605441+0000", 2023-11-05T08:33:45.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6180000000000002e-06 2023-11-05T08:33:45.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:45.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.605455+0000", 2023-11-05T08:33:45.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4464000000000001e-05 2023-11-05T08:33:45.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:45.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.605461+0000", 2023-11-05T08:33:45.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6149999999999996e-06 2023-11-05T08:33:45.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:45.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.611206+0000", 2023-11-05T08:33:45.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0057451589999999997 2023-11-05T08:33:45.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-11-05T08:33:45.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.611315+0000", 2023-11-05T08:33:45.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108844 2023-11-05T08:33:45.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:45.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.619080+0000", 2023-11-05T08:33:45.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0077654639999999997 2023-11-05T08:33:45.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:45.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.619201+0000", 2023-11-05T08:33:45.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121274 2023-11-05T08:33:45.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:45.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:18.619500+0000", 2023-11-05T08:33:45.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029808200000000002 2023-11-05T08:33:45.559 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.559 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:45.560 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.560 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:501 3.fs0 3:f91b0618:::smithi09039252-26:7 [read 0~425984,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2023-11-05T08:33:45.560 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:19.393991+0000", 2023-11-05T08:33:45.561 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 26.002434553000001, 2023-11-05T08:33:45.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016310758000000002, 2023-11-05T08:33:45.561 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:45.561 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:45.561 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:45.561 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:45.562 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:45.562 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 501 2023-11-05T08:33:45.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.562 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:45.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:45.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.393991+0000", 2023-11-05T08:33:45.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:45.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.393991+0000", 2023-11-05T08:33:45.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:45.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.393995+0000", 2023-11-05T08:33:45.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6210000000000001e-06 2023-11-05T08:33:45.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:45.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394010+0000", 2023-11-05T08:33:45.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5065000000000001e-05 2023-11-05T08:33:45.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:45.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394045+0000", 2023-11-05T08:33:45.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5095000000000002e-05 2023-11-05T08:33:45.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:45.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394063+0000", 2023-11-05T08:33:45.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8091e-05 2023-11-05T08:33:45.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:45.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394163+0000", 2023-11-05T08:33:45.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100367 2023-11-05T08:33:45.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-11-05T08:33:45.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394195+0000", 2023-11-05T08:33:45.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1349000000000001e-05 2023-11-05T08:33:45.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:45.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.398429+0000", 2023-11-05T08:33:45.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0042346190000000002 2023-11-05T08:33:45.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:45.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.398635+0000", 2023-11-05T08:33:45.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020568600000000001 2023-11-05T08:33:45.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:45.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.410302+0000", 2023-11-05T08:33:45.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011666865 2023-11-05T08:33:45.572 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.572 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:45.573 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.573 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:502 3.fs0 3:f91b0618:::smithi09039252-26:7 [read 0~425984,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2023-11-05T08:33:45.573 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:19.394105+0000", 2023-11-05T08:33:45.573 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 26.002320751999999, 2023-11-05T08:33:45.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017035277000000001, 2023-11-05T08:33:45.574 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:45.574 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:45.574 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:45.574 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:45.574 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:45.575 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 502 2023-11-05T08:33:45.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.575 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:45.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:45.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394105+0000", 2023-11-05T08:33:45.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:45.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394105+0000", 2023-11-05T08:33:45.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:45.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394106+0000", 2023-11-05T08:33:45.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9999999999999996e-07 2023-11-05T08:33:45.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:45.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394111+0000", 2023-11-05T08:33:45.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9620000000000001e-06 2023-11-05T08:33:45.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:45.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394122+0000", 2023-11-05T08:33:45.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1837e-05 2023-11-05T08:33:45.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:45.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394135+0000", 2023-11-05T08:33:45.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3103e-05 2023-11-05T08:33:45.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:45.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394300+0000", 2023-11-05T08:33:45.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016444800000000001 2023-11-05T08:33:45.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:33:45.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394334+0000", 2023-11-05T08:33:45.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4518999999999999e-05 2023-11-05T08:33:45.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:45.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.399381+0000", 2023-11-05T08:33:45.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.005046315 2023-11-05T08:33:45.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:45.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.399560+0000", 2023-11-05T08:33:45.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179457 2023-11-05T08:33:45.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:45.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.411140+0000", 2023-11-05T08:33:45.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011579836 2023-11-05T08:33:45.585 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.585 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:45.585 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.586 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:503 3.fs0 3:f91b0618:::smithi09039252-26:7 [read 0~425984,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2023-11-05T08:33:45.586 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:19.394176+0000", 2023-11-05T08:33:45.586 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 26.002249342999999, 2023-11-05T08:33:45.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018069346999999999, 2023-11-05T08:33:45.587 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:45.587 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:45.587 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:45.587 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:45.587 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:45.588 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 503 2023-11-05T08:33:45.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.588 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:45.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:45.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394176+0000", 2023-11-05T08:33:45.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:45.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394176+0000", 2023-11-05T08:33:45.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:45.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394177+0000", 2023-11-05T08:33:45.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9800000000000003e-07 2023-11-05T08:33:45.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:45.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394183+0000", 2023-11-05T08:33:45.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4489999999999996e-06 2023-11-05T08:33:45.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:45.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394195+0000", 2023-11-05T08:33:45.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1572e-05 2023-11-05T08:33:45.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:45.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394203+0000", 2023-11-05T08:33:45.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0399999999999993e-06 2023-11-05T08:33:45.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:45.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394375+0000", 2023-11-05T08:33:45.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017232799999999999 2023-11-05T08:33:45.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:33:45.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394382+0000", 2023-11-05T08:33:45.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5180000000000001e-06 2023-11-05T08:33:45.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:45.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.401433+0000", 2023-11-05T08:33:45.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0070509659999999997 2023-11-05T08:33:45.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:45.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.401608+0000", 2023-11-05T08:33:45.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017502199999999999 2023-11-05T08:33:45.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:45.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.412246+0000", 2023-11-05T08:33:45.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010637654 2023-11-05T08:33:45.600 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.600 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:45.601 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.601 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:579 3.fs0 3:f2f7b156:::smithi09039252-30:head [append 0~622592] snapc 2b=[2b,7] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2023-11-05T08:33:45.601 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:30.527658+0000", 2023-11-05T08:33:45.601 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 14.868767608000001, 2023-11-05T08:33:45.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.023980108, 2023-11-05T08:33:45.602 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:45.602 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:45.602 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:45.602 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:45.603 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:45.603 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 579 2023-11-05T08:33:45.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.603 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:45.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:45.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.527658+0000", 2023-11-05T08:33:45.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:45.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.527658+0000", 2023-11-05T08:33:45.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:45.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.527662+0000", 2023-11-05T08:33:45.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8530000000000002e-06 2023-11-05T08:33:45.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:45.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.528482+0000", 2023-11-05T08:33:45.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00081984099999999997 2023-11-05T08:33:45.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:45.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.528516+0000", 2023-11-05T08:33:45.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4604999999999998e-05 2023-11-05T08:33:45.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:45.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.528537+0000", 2023-11-05T08:33:45.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0933000000000001e-05 2023-11-05T08:33:45.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:45.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.528729+0000", 2023-11-05T08:33:45.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019200900000000001 2023-11-05T08:33:45.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-11-05T08:33:45.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.528762+0000", 2023-11-05T08:33:45.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3324999999999997e-05 2023-11-05T08:33:45.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:45.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.535307+0000", 2023-11-05T08:33:45.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0065444600000000002 2023-11-05T08:33:45.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:45.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.535736+0000", 2023-11-05T08:33:45.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042863200000000003 2023-11-05T08:33:45.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:45.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.539268+0000", 2023-11-05T08:33:45.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003532931 2023-11-05T08:33:45.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:45.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.547548+0000", 2023-11-05T08:33:45.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0082790180000000008 2023-11-05T08:33:45.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:45.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.551199+0000", 2023-11-05T08:33:45.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036517780000000001 2023-11-05T08:33:45.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:45.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.551638+0000", 2023-11-05T08:33:45.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043872300000000002 2023-11-05T08:33:45.616 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.616 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:45.616 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.617 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:580 3.fs0 3:f2f7b156:::smithi09039252-30:head [append 0~442368] snapc 2b=[2b,7] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2023-11-05T08:33:45.617 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:30.535237+0000", 2023-11-05T08:33:45.617 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 14.861188379, 2023-11-05T08:33:45.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017577461999999999, 2023-11-05T08:33:45.618 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:45.618 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:45.618 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:45.618 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:45.618 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:45.619 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 580 2023-11-05T08:33:45.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.619 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:45.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:45.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.535237+0000", 2023-11-05T08:33:45.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:45.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.535237+0000", 2023-11-05T08:33:45.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:45.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.535241+0000", 2023-11-05T08:33:45.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.963e-06 2023-11-05T08:33:45.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:45.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.535853+0000", 2023-11-05T08:33:45.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00061147299999999996 2023-11-05T08:33:45.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:45.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.535888+0000", 2023-11-05T08:33:45.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5355000000000003e-05 2023-11-05T08:33:45.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:45.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.535904+0000", 2023-11-05T08:33:45.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5699e-05 2023-11-05T08:33:45.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:45.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.543209+0000", 2023-11-05T08:33:45.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0073058610000000003 2023-11-05T08:33:45.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:45.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.543547+0000", 2023-11-05T08:33:45.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033735500000000001 2023-11-05T08:33:45.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:45.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.545547+0000", 2023-11-05T08:33:45.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020002169999999999 2023-11-05T08:33:45.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:45.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.551911+0000", 2023-11-05T08:33:45.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0063640119999999996 2023-11-05T08:33:45.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:45.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.552375+0000", 2023-11-05T08:33:45.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046346499999999998 2023-11-05T08:33:45.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:45.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.552815+0000", 2023-11-05T08:33:45.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044006200000000002 2023-11-05T08:33:45.630 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.630 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:45.630 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.631 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:582 3.fs0 3:f2f7b156:::smithi09039252-30:head [append 0~196608] snapc 2b=[2b,7] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2023-11-05T08:33:45.631 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:30.548185+0000", 2023-11-05T08:33:45.631 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 14.84824068, 2023-11-05T08:33:45.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013807253, 2023-11-05T08:33:45.632 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:45.632 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:45.632 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:45.632 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:45.632 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:45.633 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 582 2023-11-05T08:33:45.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.633 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:45.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:45.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.548185+0000", 2023-11-05T08:33:45.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:45.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.548185+0000", 2023-11-05T08:33:45.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:45.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.548188+0000", 2023-11-05T08:33:45.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.824e-06 2023-11-05T08:33:45.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:45.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.548504+0000", 2023-11-05T08:33:45.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031676300000000002 2023-11-05T08:33:45.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:45.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.548532+0000", 2023-11-05T08:33:45.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7469000000000001e-05 2023-11-05T08:33:45.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:45.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.548551+0000", 2023-11-05T08:33:45.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9496000000000001e-05 2023-11-05T08:33:45.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:45.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.552963+0000", 2023-11-05T08:33:45.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0044113549999999996 2023-11-05T08:33:45.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:45.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.553313+0000", 2023-11-05T08:33:45.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000350076 2023-11-05T08:33:45.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:45.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.555227+0000", 2023-11-05T08:33:45.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019142440000000001 2023-11-05T08:33:45.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:45.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.560443+0000", 2023-11-05T08:33:45.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052160669999999996 2023-11-05T08:33:45.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:45.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.561259+0000", 2023-11-05T08:33:45.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00081575799999999998 2023-11-05T08:33:45.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:45.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.561992+0000", 2023-11-05T08:33:45.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073320099999999995 2023-11-05T08:33:45.644 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.644 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:45.644 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.645 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:583 3.fs0 3:f2f7b156:::smithi09039252-30:head [setxattr _header (57) in=7b] snapc 2b=[2b,7] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2023-11-05T08:33:45.645 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:30.548589+0000", 2023-11-05T08:33:45.645 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 14.847836368999999, 2023-11-05T08:33:45.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015644193000000001, 2023-11-05T08:33:45.645 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:45.646 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:45.646 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:45.646 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:45.646 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:45.646 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 583 2023-11-05T08:33:45.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.647 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:45.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:45.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.548589+0000", 2023-11-05T08:33:45.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:45.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.548589+0000", 2023-11-05T08:33:45.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:45.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.548591+0000", 2023-11-05T08:33:45.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.517e-06 2023-11-05T08:33:45.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:45.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.548599+0000", 2023-11-05T08:33:45.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6759999999999996e-06 2023-11-05T08:33:45.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:45.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.548614+0000", 2023-11-05T08:33:45.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4703e-05 2023-11-05T08:33:45.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:45.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.548623+0000", 2023-11-05T08:33:45.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1260000000000004e-06 2023-11-05T08:33:45.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:45.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.558538+0000", 2023-11-05T08:33:45.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0099148970000000006 2023-11-05T08:33:45.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:45.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.558744+0000", 2023-11-05T08:33:45.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020564199999999999 2023-11-05T08:33:45.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:45.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.559684+0000", 2023-11-05T08:33:45.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000940732 2023-11-05T08:33:45.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:45.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.562463+0000", 2023-11-05T08:33:45.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0027782000000000002 2023-11-05T08:33:45.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:45.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.563447+0000", 2023-11-05T08:33:45.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00098426500000000001 2023-11-05T08:33:45.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:45.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.564233+0000", 2023-11-05T08:33:45.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00078643500000000004 2023-11-05T08:33:45.658 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.658 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:45.658 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.658 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:584 3.fs0 3:f2f7b156:::smithi09039252-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e77)", 2023-11-05T08:33:45.659 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:30.548638+0000", 2023-11-05T08:33:45.659 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 14.847787111000001, 2023-11-05T08:33:45.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020181330000000001, 2023-11-05T08:33:45.659 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:45.659 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:45.660 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:45.660 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:45.660 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:45.660 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 584 2023-11-05T08:33:45.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.660 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:45.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:45.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.548638+0000", 2023-11-05T08:33:45.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:45.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.548638+0000", 2023-11-05T08:33:45.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:45.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:45.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.548639+0000", 2023-11-05T08:33:45.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7000000000000003e-07 2023-11-05T08:33:45.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:45.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.548644+0000", 2023-11-05T08:33:45.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1500000000000001e-06 2023-11-05T08:33:45.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:45.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.548655+0000", 2023-11-05T08:33:45.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1046000000000001e-05 2023-11-05T08:33:45.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:45.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.548663+0000", 2023-11-05T08:33:45.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9799999999999998e-06 2023-11-05T08:33:45.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:45.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.562111+0000", 2023-11-05T08:33:45.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013448402999999999 2023-11-05T08:33:45.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-11-05T08:33:45.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.562442+0000", 2023-11-05T08:33:45.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033102700000000001 2023-11-05T08:33:45.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:45.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.565260+0000", 2023-11-05T08:33:45.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028184479999999999 2023-11-05T08:33:45.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:45.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.565441+0000", 2023-11-05T08:33:45.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018098999999999999 2023-11-05T08:33:45.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:45.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:45.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:45.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.568820+0000", 2023-11-05T08:33:45.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033783160000000001 2023-11-05T08:33:45.671 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.671 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:45.671 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.671 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:45.671 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:45.671 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:33:45.672 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:33:45.393+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:33:45.674 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2023-11-05T08:33:45.674 INFO:tasks.rados.rados.0.smithi090.stdout:284: done (4 left) 2023-11-05T08:33:45.674 INFO:tasks.rados.rados.0.smithi090.stdout:285: delete oid 21 current snap is 29 2023-11-05T08:33:45.675 INFO:tasks.rados.rados.0.smithi090.stdout:280: finishing write tid 1 to smithi09039252-30 2023-11-05T08:33:45.675 INFO:tasks.rados.rados.0.smithi090.stdout:282: finishing rollback tid 0 to smithi09039252-27 2023-11-05T08:33:45.675 INFO:tasks.rados.rados.0.smithi090.stdout:281: expect (ObjNum 31 snap 0 seq_num 31) 2023-11-05T08:33:45.675 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:45.670+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:45.709 INFO:tasks.rados.rados.0.smithi090.stdout:282: finishing rollback tid 1 to smithi09039252-27 2023-11-05T08:33:45.709 INFO:tasks.rados.rados.0.smithi090.stdout:280: finishing write tid 2 to smithi09039252-30 2023-11-05T08:33:45.709 INFO:tasks.rados.rados.0.smithi090.stdout:280: finishing write tid 3 to smithi09039252-30 2023-11-05T08:33:45.710 INFO:tasks.rados.rados.0.smithi090.stdout:282: finishing rollback tid 2 to smithi09039252-27 2023-11-05T08:33:45.710 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 27 v 84 (ObjNum 26 snap 0 seq_num 26) dirty exists 2023-11-05T08:33:45.710 INFO:tasks.rados.rados.0.smithi090.stdout:280: finishing write tid 4 to smithi09039252-30 2023-11-05T08:33:45.710 INFO:tasks.rados.rados.0.smithi090.stdout:281: done (4 left) 2023-11-05T08:33:45.710 INFO:tasks.rados.rados.0.smithi090.stdout:282: done (3 left) 2023-11-05T08:33:45.711 INFO:tasks.rados.rados.0.smithi090.stdout:285: done (2 left) 2023-11-05T08:33:45.711 INFO:tasks.rados.rados.0.smithi090.stdout:286: rmattr oid 6 current snap is 29 2023-11-05T08:33:45.711 INFO:tasks.rados.rados.0.smithi090.stdout:286: done (2 left) 2023-11-05T08:33:45.711 INFO:tasks.rados.rados.0.smithi090.stdout:287: read oid 34 snap 4 2023-11-05T08:33:45.711 INFO:tasks.rados.rados.0.smithi090.stdout:287: expect (ObjNum 33 snap 0 seq_num 33) 2023-11-05T08:33:45.711 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 13 current snap is 29 2023-11-05T08:33:45.712 INFO:tasks.rados.rados.0.smithi090.stdout:280: finishing write tid 5 to smithi09039252-30 2023-11-05T08:33:45.712 INFO:tasks.rados.rados.0.smithi090.stdout:280: oid 30 updating version 0 to 57 2023-11-05T08:33:45.712 INFO:tasks.rados.rados.0.smithi090.stdout:280: oid 30 version 57 is already newer than 56 2023-11-05T08:33:45.712 INFO:tasks.rados.rados.0.smithi090.stdout:280: oid 30 version 57 is already newer than 55 2023-11-05T08:33:45.712 INFO:tasks.rados.rados.0.smithi090.stdout:280: oid 30 version 57 is already newer than 54 2023-11-05T08:33:45.713 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 30 v 57 (ObjNum 109 snap 29 seq_num 109) dirty exists 2023-11-05T08:33:45.713 INFO:tasks.rados.rados.0.smithi090.stdout:280: left oid 30 (ObjNum 109 snap 29 seq_num 109) 2023-11-05T08:33:45.713 INFO:tasks.rados.rados.0.smithi090.stdout:288: seq_num 111 ranges {1064960=704512,1769472=655360,2424832=507904} 2023-11-05T08:33:45.739 INFO:tasks.rados.rados.0.smithi090.stdout:288: writing smithi09039252-13 from 1064960 to 1769472 tid 1 2023-11-05T08:33:45.750 INFO:tasks.rados.rados.0.smithi090.stdout:288: writing smithi09039252-13 from 1769472 to 2424832 tid 2 2023-11-05T08:33:45.756 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-11-05T08:33:45.754+0000 7f72aba83640 -1 osd.3 92 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-05T08:33:45.759 INFO:tasks.rados.rados.0.smithi090.stdout:288: writing smithi09039252-13 from 2424832 to 2932736 tid 3 2023-11-05T08:33:45.760 INFO:tasks.rados.rados.0.smithi090.stdout:280: done (3 left) 2023-11-05T08:33:45.760 INFO:tasks.rados.rados.0.smithi090.stdout:289: read oid 44 snap -1 2023-11-05T08:33:45.760 INFO:tasks.rados.rados.0.smithi090.stdout:289: expect (ObjNum 43 snap 0 seq_num 43) 2023-11-05T08:33:45.760 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 27 current snap is 29 2023-11-05T08:33:45.760 INFO:tasks.rados.rados.0.smithi090.stdout:290: seq_num 112 ranges {442368=720896,1163264=802816,1966080=360448} 2023-11-05T08:33:45.773 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:33:45.770+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:33:45.775 INFO:tasks.rados.rados.0.smithi090.stdout:290: writing smithi09039252-27 from 442368 to 1163264 tid 1 2023-11-05T08:33:45.783 INFO:tasks.rados.rados.0.smithi090.stdout:290: writing smithi09039252-27 from 1163264 to 1966080 tid 2 2023-11-05T08:33:45.786 INFO:tasks.rados.rados.0.smithi090.stdout:290: writing smithi09039252-27 from 1966080 to 2326528 tid 3 2023-11-05T08:33:45.787 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 31 current snap is 29 2023-11-05T08:33:45.787 INFO:tasks.rados.rados.0.smithi090.stdout:291: seq_num 113 ranges {2768896=737280,3506176=557056,4063232=606208} 2023-11-05T08:33:45.812 INFO:tasks.rados.rados.0.smithi090.stdout:291: writing smithi09039252-31 from 2768896 to 3506176 tid 1 2023-11-05T08:33:45.816 INFO:tasks.rados.rados.0.smithi090.stdout:291: writing smithi09039252-31 from 3506176 to 4063232 tid 2 2023-11-05T08:33:45.821 INFO:tasks.rados.rados.0.smithi090.stdout:291: writing smithi09039252-31 from 4063232 to 4669440 tid 3 2023-11-05T08:33:45.821 INFO:tasks.rados.rados.0.smithi090.stdout:292: read oid 6 snap 4 2023-11-05T08:33:45.821 INFO:tasks.rados.rados.0.smithi090.stdout:292: expect (ObjNum 5 snap 0 seq_num 5) 2023-11-05T08:33:45.821 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 29 current snap is 29 2023-11-05T08:33:45.821 INFO:tasks.rados.rados.0.smithi090.stdout:287: expect (ObjNum 33 snap 0 seq_num 33) 2023-11-05T08:33:45.837 INFO:tasks.rados.rados.0.smithi090.stdout:288: finishing write tid 1 to smithi09039252-13 2023-11-05T08:33:45.837 INFO:tasks.rados.rados.0.smithi090.stdout:288: finishing write tid 2 to smithi09039252-13 2023-11-05T08:33:45.838 INFO:tasks.rados.rados.0.smithi090.stdout:288: finishing write tid 3 to smithi09039252-13 2023-11-05T08:33:45.838 INFO:tasks.rados.rados.0.smithi090.stdout:289: expect (ObjNum 43 snap 0 seq_num 43) 2023-11-05T08:33:45.845 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:33:45.846 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-11-05T08:33:45.846 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-11-05T08:33:45.846 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:33:45.856 INFO:tasks.rados.rados.0.smithi090.stdout:288: finishing write tid 4 to smithi09039252-13 2023-11-05T08:33:45.856 INFO:tasks.rados.rados.0.smithi090.stdout:288: finishing write tid 5 to smithi09039252-13 2023-11-05T08:33:45.856 INFO:tasks.rados.rados.0.smithi090.stdout:288: oid 13 updating version 0 to 40 2023-11-05T08:33:45.857 INFO:tasks.rados.rados.0.smithi090.stdout:288: oid 13 version 40 is already newer than 38 2023-11-05T08:33:45.857 INFO:tasks.rados.rados.0.smithi090.stdout:288: oid 13 updating version 40 to 41 2023-11-05T08:33:45.857 INFO:tasks.rados.rados.0.smithi090.stdout:288: oid 13 version 41 is already newer than 39 2023-11-05T08:33:45.857 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 13 v 41 (ObjNum 111 snap 29 seq_num 111) dirty exists 2023-11-05T08:33:45.857 INFO:tasks.rados.rados.0.smithi090.stdout:288: left oid 13 (ObjNum 111 snap 29 seq_num 111) 2023-11-05T08:33:45.857 INFO:tasks.rados.rados.0.smithi090.stdout:290: finishing write tid 1 to smithi09039252-27 2023-11-05T08:33:45.858 INFO:tasks.rados.rados.0.smithi090.stdout:290: finishing write tid 2 to smithi09039252-27 2023-11-05T08:33:45.858 INFO:tasks.rados.rados.0.smithi090.stdout:290: finishing write tid 3 to smithi09039252-27 2023-11-05T08:33:45.858 INFO:tasks.rados.rados.0.smithi090.stdout:290: finishing write tid 4 to smithi09039252-27 2023-11-05T08:33:45.858 INFO:tasks.rados.rados.0.smithi090.stdout:290: finishing write tid 5 to smithi09039252-27 2023-11-05T08:33:45.858 INFO:tasks.rados.rados.0.smithi090.stdout:290: oid 27 updating version 0 to 90 2023-11-05T08:33:45.859 INFO:tasks.rados.rados.0.smithi090.stdout:290: oid 27 updating version 90 to 92 2023-11-05T08:33:45.859 INFO:tasks.rados.rados.0.smithi090.stdout:290: oid 27 version 92 is already newer than 91 2023-11-05T08:33:45.859 INFO:tasks.rados.rados.0.smithi090.stdout:290: oid 27 version 92 is already newer than 89 2023-11-05T08:33:45.859 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 27 v 92 (ObjNum 112 snap 29 seq_num 112) dirty exists 2023-11-05T08:33:45.859 INFO:tasks.rados.rados.0.smithi090.stdout:290: left oid 27 (ObjNum 112 snap 29 seq_num 112) 2023-11-05T08:33:45.860 INFO:tasks.rados.rados.0.smithi090.stdout:291: finishing write tid 1 to smithi09039252-31 2023-11-05T08:33:45.860 INFO:tasks.rados.rados.0.smithi090.stdout:293: seq_num 114 ranges {0=753664,753664=688128,1441792=475136} 2023-11-05T08:33:45.861 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2023-11-05T08:33:45.862 INFO:tasks.rados.rados.0.smithi090.stdout:293: writing smithi09039252-29 from 0 to 753664 tid 1 2023-11-05T08:33:45.868 INFO:tasks.rados.rados.0.smithi090.stdout:293: writing smithi09039252-29 from 753664 to 1441792 tid 2 2023-11-05T08:33:45.872 INFO:tasks.rados.rados.0.smithi090.stdout:293: writing smithi09039252-29 from 1441792 to 1916928 tid 3 2023-11-05T08:33:45.873 INFO:tasks.rados.rados.0.smithi090.stdout:287: done (7 left) 2023-11-05T08:33:45.873 INFO:tasks.rados.rados.0.smithi090.stdout:288: done (6 left) 2023-11-05T08:33:45.873 INFO:tasks.rados.rados.0.smithi090.stdout:289: done (5 left) 2023-11-05T08:33:45.873 INFO:tasks.rados.rados.0.smithi090.stdout:290: done (4 left) 2023-11-05T08:33:45.874 INFO:tasks.rados.rados.0.smithi090.stdout:294: snap_create 2023-11-05T08:33:45.874 INFO:tasks.rados.rados.0.smithi090.stdout:291: finishing write tid 2 to smithi09039252-31 2023-11-05T08:33:45.874 INFO:tasks.rados.rados.0.smithi090.stdout:292: expect (ObjNum 5 snap 0 seq_num 5) 2023-11-05T08:33:45.875 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:33:45.872+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:33:45.877 INFO:tasks.rados.rados.0.smithi090.stdout:291: finishing write tid 3 to smithi09039252-31 2023-11-05T08:33:45.877 INFO:tasks.rados.rados.0.smithi090.stdout:291: finishing write tid 4 to smithi09039252-31 2023-11-05T08:33:45.877 INFO:tasks.rados.rados.0.smithi090.stdout:291: finishing write tid 5 to smithi09039252-31 2023-11-05T08:33:45.877 INFO:tasks.rados.rados.0.smithi090.stdout:291: oid 31 updating version 0 to 44 2023-11-05T08:33:45.878 INFO:tasks.rados.rados.0.smithi090.stdout:291: oid 31 updating version 44 to 45 2023-11-05T08:33:45.878 INFO:tasks.rados.rados.0.smithi090.stdout:291: oid 31 updating version 45 to 46 2023-11-05T08:33:45.878 INFO:tasks.rados.rados.0.smithi090.stdout:291: oid 31 version 46 is already newer than 43 2023-11-05T08:33:45.878 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 31 v 46 (ObjNum 113 snap 29 seq_num 113) dirty exists 2023-11-05T08:33:45.878 INFO:tasks.rados.rados.0.smithi090.stdout:291: left oid 31 (ObjNum 113 snap 29 seq_num 113) 2023-11-05T08:33:45.879 INFO:tasks.rados.rados.0.smithi090.stdout:293: finishing write tid 1 to smithi09039252-29 2023-11-05T08:33:45.882 INFO:tasks.rados.rados.0.smithi090.stdout:293: finishing write tid 2 to smithi09039252-29 2023-11-05T08:33:45.884 INFO:tasks.rados.rados.0.smithi090.stdout:293: finishing write tid 3 to smithi09039252-29 2023-11-05T08:33:45.886 INFO:tasks.rados.rados.0.smithi090.stdout:293: finishing write tid 4 to smithi09039252-29 2023-11-05T08:33:45.890 INFO:tasks.rados.rados.0.smithi090.stdout:293: finishing write tid 5 to smithi09039252-29 2023-11-05T08:33:45.890 INFO:tasks.rados.rados.0.smithi090.stdout:293: oid 29 updating version 0 to 29 2023-11-05T08:33:45.891 INFO:tasks.rados.rados.0.smithi090.stdout:293: oid 29 updating version 29 to 32 2023-11-05T08:33:45.891 INFO:tasks.rados.rados.0.smithi090.stdout:293: oid 29 version 32 is already newer than 31 2023-11-05T08:33:45.891 INFO:tasks.rados.rados.0.smithi090.stdout:293: oid 29 version 32 is already newer than 30 2023-11-05T08:33:45.891 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 29 v 32 (ObjNum 114 snap 29 seq_num 114) dirty exists 2023-11-05T08:33:45.891 INFO:tasks.rados.rados.0.smithi090.stdout:293: left oid 29 (ObjNum 114 snap 29 seq_num 114) 2023-11-05T08:33:45.939 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-05T08:33:45.976 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:33:45.973+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:33:45.977 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:33:45.977 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-11-05T08:33:45.977 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-11-05T08:33:45.978 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-11-05T08:33:45.978 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:33:45.990 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2023-11-05T08:33:46.037 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-11-05T08:33:46.077 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:46.074+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:46.079 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-11-05T08:33:46.101 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1: {} 2023-11-05T08:33:46.101 INFO:teuthology.orchestra.run.smithi002.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-11-05T08:33:46.122 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2: {} 2023-11-05T08:33:46.122 INFO:teuthology.orchestra.run.smithi002.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-11-05T08:33:46.122 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-11-05T08:33:46.140 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4: {} 2023-11-05T08:33:46.141 INFO:teuthology.orchestra.run.smithi002.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-11-05T08:33:46.158 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:33:46.158 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-11-05T08:33:46.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-11-05T08:33:46.159 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-11-05T08:33:46.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.159 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.3s3 36/30 ECSubWrite(tid=183, reqid=osd.13.0:183, at_version=36'12, trim_to=0'0, roll_forward_to=35'10))", 2023-11-05T08:33:46.159 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.072455+0000", 2023-11-05T08:33:46.160 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 57.083898003000002, 2023-11-05T08:33:46.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0063400499999999999, 2023-11-05T08:33:46.160 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:46.160 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:46.160 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:46.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.161 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:46.161 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072455+0000", 2023-11-05T08:33:46.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.162 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:46.162 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072455+0000", 2023-11-05T08:33:46.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:46.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072456+0000", 2023-11-05T08:33:46.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9999999999999998e-07 2023-11-05T08:33:46.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.164 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:46.164 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072512+0000", 2023-11-05T08:33:46.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5667000000000001e-05 2023-11-05T08:33:46.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:46.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072537+0000", 2023-11-05T08:33:46.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5833000000000002e-05 2023-11-05T08:33:46.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:46.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072581+0000", 2023-11-05T08:33:46.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3470999999999998e-05 2023-11-05T08:33:46.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:46.167 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072752+0000", 2023-11-05T08:33:46.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017130100000000001 2023-11-05T08:33:46.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:46.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072775+0000", 2023-11-05T08:33:46.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3379000000000001e-05 2023-11-05T08:33:46.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:46.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.078646+0000", 2023-11-05T08:33:46.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0058709419999999997 2023-11-05T08:33:46.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:46.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.078795+0000", 2023-11-05T08:33:46.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014895699999999999 2023-11-05T08:33:46.170 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.170 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:46.171 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.171 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.8s5 36/30 ECSubWrite(tid=23, reqid=osd.5.0:23, at_version=36'15, trim_to=0'0, roll_forward_to=34'13))", 2023-11-05T08:33:46.171 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.079609+0000", 2023-11-05T08:33:46.171 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 57.076744300999998, 2023-11-05T08:33:46.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049579109999999997, 2023-11-05T08:33:46.172 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:46.172 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:46.172 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:46.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:46.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079609+0000", 2023-11-05T08:33:46.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:46.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079609+0000", 2023-11-05T08:33:46.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:46.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079609+0000", 2023-11-05T08:33:46.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.27e-07 2023-11-05T08:33:46.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:46.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079645+0000", 2023-11-05T08:33:46.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5522000000000003e-05 2023-11-05T08:33:46.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:46.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079672+0000", 2023-11-05T08:33:46.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6648e-05 2023-11-05T08:33:46.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:46.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079693+0000", 2023-11-05T08:33:46.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1206e-05 2023-11-05T08:33:46.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:46.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079849+0000", 2023-11-05T08:33:46.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156112 2023-11-05T08:33:46.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:46.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079873+0000", 2023-11-05T08:33:46.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3589000000000001e-05 2023-11-05T08:33:46.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:46.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.084477+0000", 2023-11-05T08:33:46.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0046040969999999997 2023-11-05T08:33:46.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:46.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.084567+0000", 2023-11-05T08:33:46.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0309999999999994e-05 2023-11-05T08:33:46.182 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.182 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:46.183 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.183 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.8s5 36/30 ECSubWrite(tid=24, reqid=osd.5.0:23, at_version=0'0, trim_to=0'0, roll_forward_to=36'15))", 2023-11-05T08:33:46.183 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.088806+0000", 2023-11-05T08:33:46.184 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 57.067547030999997, 2023-11-05T08:33:46.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055977739999999998, 2023-11-05T08:33:46.184 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:46.185 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:46.185 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:46.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:46.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.088806+0000", 2023-11-05T08:33:46.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:46.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.088806+0000", 2023-11-05T08:33:46.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:46.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.092263+0000", 2023-11-05T08:33:46.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034565210000000002 2023-11-05T08:33:46.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:46.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.092287+0000", 2023-11-05T08:33:46.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4505999999999999e-05 2023-11-05T08:33:46.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:46.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.092314+0000", 2023-11-05T08:33:46.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6335000000000001e-05 2023-11-05T08:33:46.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:46.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.092329+0000", 2023-11-05T08:33:46.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5486e-05 2023-11-05T08:33:46.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:46.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.092519+0000", 2023-11-05T08:33:46.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019026700000000001 2023-11-05T08:33:46.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:46.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.092545+0000", 2023-11-05T08:33:46.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5644999999999999e-05 2023-11-05T08:33:46.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:46.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.094327+0000", 2023-11-05T08:33:46.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001781543 2023-11-05T08:33:46.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:46.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.094404+0000", 2023-11-05T08:33:46.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7471000000000004e-05 2023-11-05T08:33:46.195 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.195 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:46.195 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.196 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.2s4 37/30 ECSubWrite(tid=185, reqid=client.4691.0:240, at_version=37'7, trim_to=0'0, roll_forward_to=33'5))", 2023-11-05T08:33:46.196 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:50.088409+0000", 2023-11-05T08:33:46.196 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 56.067944828999998, 2023-11-05T08:33:46.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036490530000000002, 2023-11-05T08:33:46.197 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:46.197 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:46.197 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:46.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:46.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.088409+0000", 2023-11-05T08:33:46.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:46.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.088409+0000", 2023-11-05T08:33:46.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:46.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.088409+0000", 2023-11-05T08:33:46.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.569999999999999e-07 2023-11-05T08:33:46.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:46.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.088461+0000", 2023-11-05T08:33:46.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1665000000000001e-05 2023-11-05T08:33:46.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:46.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.088486+0000", 2023-11-05T08:33:46.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4757999999999999e-05 2023-11-05T08:33:46.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:46.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.088505+0000", 2023-11-05T08:33:46.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8885999999999998e-05 2023-11-05T08:33:46.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:46.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.088651+0000", 2023-11-05T08:33:46.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014625900000000001 2023-11-05T08:33:46.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:46.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.088678+0000", 2023-11-05T08:33:46.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7229e-05 2023-11-05T08:33:46.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:46.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.091969+0000", 2023-11-05T08:33:46.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032904900000000001 2023-11-05T08:33:46.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:46.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:50.092058+0000", 2023-11-05T08:33:46.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8808999999999996e-05 2023-11-05T08:33:46.207 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.207 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:46.207 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.208 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.2s4 41/40 ECSubWrite(tid=187, reqid=client.4691.0:292, at_version=41'9, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:46.208 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:54.162954+0000", 2023-11-05T08:33:46.208 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 51.993398907, 2023-11-05T08:33:46.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0045082990000000003, 2023-11-05T08:33:46.209 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:46.209 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:46.209 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:46.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:46.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.162954+0000", 2023-11-05T08:33:46.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:46.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.162954+0000", 2023-11-05T08:33:46.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:46.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.162956+0000", 2023-11-05T08:33:46.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.86e-06 2023-11-05T08:33:46.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:46.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.163012+0000", 2023-11-05T08:33:46.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5744e-05 2023-11-05T08:33:46.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:46.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.163048+0000", 2023-11-05T08:33:46.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6346000000000001e-05 2023-11-05T08:33:46.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:46.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.163075+0000", 2023-11-05T08:33:46.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6622e-05 2023-11-05T08:33:46.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:46.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.163236+0000", 2023-11-05T08:33:46.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016064599999999999 2023-11-05T08:33:46.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:46.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.163262+0000", 2023-11-05T08:33:46.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6356e-05 2023-11-05T08:33:46.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:46.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.167302+0000", 2023-11-05T08:33:46.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040403000000000001 2023-11-05T08:33:46.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:46.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.167463+0000", 2023-11-05T08:33:46.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016042500000000001 2023-11-05T08:33:46.220 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.220 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:46.220 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.221 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.2s4 41/40 ECSubWrite(tid=188, reqid=osd.13.0:188, at_version=41'11, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:46.221 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:54.171988+0000", 2023-11-05T08:33:46.222 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 51.984365377000003, 2023-11-05T08:33:46.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037888499999999999, 2023-11-05T08:33:46.222 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:46.222 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:46.222 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:46.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:46.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.171988+0000", 2023-11-05T08:33:46.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:46.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.171988+0000", 2023-11-05T08:33:46.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:46.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.171988+0000", 2023-11-05T08:33:46.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1400000000000001e-07 2023-11-05T08:33:46.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:46.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.172004+0000", 2023-11-05T08:33:46.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6308e-05 2023-11-05T08:33:46.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:46.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.172011+0000", 2023-11-05T08:33:46.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9430000000000003e-06 2023-11-05T08:33:46.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:46.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.172018+0000", 2023-11-05T08:33:46.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6250000000000001e-06 2023-11-05T08:33:46.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:46.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.172082+0000", 2023-11-05T08:33:46.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4311000000000001e-05 2023-11-05T08:33:46.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:46.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.172093+0000", 2023-11-05T08:33:46.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1045e-05 2023-11-05T08:33:46.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:46.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.175705+0000", 2023-11-05T08:33:46.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036118819999999999 2023-11-05T08:33:46.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:46.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.175777+0000", 2023-11-05T08:33:46.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1521999999999997e-05 2023-11-05T08:33:46.236 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.236 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:46.237 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.237 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.13s3 41/40 ECSubWrite(tid=190, reqid=osd.13.0:190, at_version=41'14, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:46.238 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:54.189971+0000", 2023-11-05T08:33:46.238 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 51.966382174000003, 2023-11-05T08:33:46.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003736746, 2023-11-05T08:33:46.239 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:46.239 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:46.240 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:46.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:46.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.189971+0000", 2023-11-05T08:33:46.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:46.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.189971+0000", 2023-11-05T08:33:46.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:46.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.189972+0000", 2023-11-05T08:33:46.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4900000000000005e-07 2023-11-05T08:33:46.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:46.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.190023+0000", 2023-11-05T08:33:46.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1162999999999998e-05 2023-11-05T08:33:46.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:46.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.190044+0000", 2023-11-05T08:33:46.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0761999999999999e-05 2023-11-05T08:33:46.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:46.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.190062+0000", 2023-11-05T08:33:46.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7977000000000001e-05 2023-11-05T08:33:46.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:46.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.190138+0000", 2023-11-05T08:33:46.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5907000000000004e-05 2023-11-05T08:33:46.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:46.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.190148+0000", 2023-11-05T08:33:46.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0344e-05 2023-11-05T08:33:46.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:46.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.193643+0000", 2023-11-05T08:33:46.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003494718 2023-11-05T08:33:46.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:46.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.193708+0000", 2023-11-05T08:33:46.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5025999999999997e-05 2023-11-05T08:33:46.249 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.249 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:46.250 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.251 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.2s4 42/40 ECSubWrite(tid=192, reqid=osd.13.0:192, at_version=42'13, trim_to=0'0, roll_forward_to=41'11))", 2023-11-05T08:33:46.251 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:55.150758+0000", 2023-11-05T08:33:46.251 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 51.005595818000003, 2023-11-05T08:33:46.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032901889999999998, 2023-11-05T08:33:46.251 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:46.252 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:46.252 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:46.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:46.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.150758+0000", 2023-11-05T08:33:46.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:46.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.150758+0000", 2023-11-05T08:33:46.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:46.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.150758+0000", 2023-11-05T08:33:46.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9999999999999998e-07 2023-11-05T08:33:46.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:46.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.150792+0000", 2023-11-05T08:33:46.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4369999999999998e-05 2023-11-05T08:33:46.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:46.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.150820+0000", 2023-11-05T08:33:46.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7285999999999999e-05 2023-11-05T08:33:46.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:46.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.150833+0000", 2023-11-05T08:33:46.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3603e-05 2023-11-05T08:33:46.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:46.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.150981+0000", 2023-11-05T08:33:46.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000148028 2023-11-05T08:33:46.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:46.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.151008+0000", 2023-11-05T08:33:46.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7169e-05 2023-11-05T08:33:46.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:46.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.153963+0000", 2023-11-05T08:33:46.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029548700000000001 2023-11-05T08:33:46.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:46.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.154048+0000", 2023-11-05T08:33:46.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4363000000000005e-05 2023-11-05T08:33:46.264 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.264 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:46.264 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.265 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.16s1 46/45 ECSubWrite(tid=45, reqid=osd.11.0:45, at_version=46'11, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:46.265 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:59.219281+0000", 2023-11-05T08:33:46.265 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 46.937071891999999, 2023-11-05T08:33:46.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.005540456, 2023-11-05T08:33:46.266 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:46.266 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:46.266 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:46.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:46.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219281+0000", 2023-11-05T08:33:46.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:46.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219281+0000", 2023-11-05T08:33:46.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:46.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219283+0000", 2023-11-05T08:33:46.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.637e-06 2023-11-05T08:33:46.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:46.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219344+0000", 2023-11-05T08:33:46.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0831000000000001e-05 2023-11-05T08:33:46.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:46.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219376+0000", 2023-11-05T08:33:46.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1945e-05 2023-11-05T08:33:46.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:46.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219417+0000", 2023-11-05T08:33:46.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1335000000000003e-05 2023-11-05T08:33:46.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:46.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219601+0000", 2023-11-05T08:33:46.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018345699999999999 2023-11-05T08:33:46.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:46.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219627+0000", 2023-11-05T08:33:46.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6732000000000001e-05 2023-11-05T08:33:46.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:46.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.224749+0000", 2023-11-05T08:33:46.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0051213129999999997 2023-11-05T08:33:46.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:46.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.224822+0000", 2023-11-05T08:33:46.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3206000000000003e-05 2023-11-05T08:33:46.276 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.276 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:46.276 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.276 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.11s2 46/45 ECSubWrite(tid=0, reqid=osd.10.0:0, at_version=46'17, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:46.277 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:59.222006+0000", 2023-11-05T08:33:46.277 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 46.934347199999998, 2023-11-05T08:33:46.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053541609999999996, 2023-11-05T08:33:46.277 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:46.278 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:46.278 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:46.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:46.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222006+0000", 2023-11-05T08:33:46.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:46.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222006+0000", 2023-11-05T08:33:46.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:46.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222008+0000", 2023-11-05T08:33:46.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7430000000000001e-06 2023-11-05T08:33:46.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:46.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222062+0000", 2023-11-05T08:33:46.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3891000000000003e-05 2023-11-05T08:33:46.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:46.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222095+0000", 2023-11-05T08:33:46.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3148999999999997e-05 2023-11-05T08:33:46.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:46.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222118+0000", 2023-11-05T08:33:46.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3258999999999999e-05 2023-11-05T08:33:46.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:46.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222279+0000", 2023-11-05T08:33:46.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016113 2023-11-05T08:33:46.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:46.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222303+0000", 2023-11-05T08:33:46.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4088999999999999e-05 2023-11-05T08:33:46.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:46.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.227275+0000", 2023-11-05T08:33:46.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049715599999999999 2023-11-05T08:33:46.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:46.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.227360+0000", 2023-11-05T08:33:46.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5340000000000006e-05 2023-11-05T08:33:46.287 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.288 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:46.288 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.288 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.11s2 46/45 ECSubWrite(tid=1, reqid=osd.10.0:0, at_version=0'0, trim_to=0'0, roll_forward_to=46'17))", 2023-11-05T08:33:46.289 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:59.232093+0000", 2023-11-05T08:33:46.289 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 46.924260724, 2023-11-05T08:33:46.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037473570000000002, 2023-11-05T08:33:46.290 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:46.290 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:46.290 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:46.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:46.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232093+0000", 2023-11-05T08:33:46.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:46.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232093+0000", 2023-11-05T08:33:46.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:46.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232094+0000", 2023-11-05T08:33:46.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0380000000000001e-06 2023-11-05T08:33:46.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:46.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232122+0000", 2023-11-05T08:33:46.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8585000000000002e-05 2023-11-05T08:33:46.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:46.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232142+0000", 2023-11-05T08:33:46.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9357000000000001e-05 2023-11-05T08:33:46.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:46.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232157+0000", 2023-11-05T08:33:46.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5141e-05 2023-11-05T08:33:46.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:46.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232272+0000", 2023-11-05T08:33:46.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114945 2023-11-05T08:33:46.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:46.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.232285+0000", 2023-11-05T08:33:46.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2917e-05 2023-11-05T08:33:46.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:46.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.235777+0000", 2023-11-05T08:33:46.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003492069 2023-11-05T08:33:46.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:46.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.235840+0000", 2023-11-05T08:33:46.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3305000000000005e-05 2023-11-05T08:33:46.302 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.303 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:46.303 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.303 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.16s1 46/45 ECSubWrite(tid=47, reqid=osd.11.0:47, at_version=46'13, trim_to=0'0, roll_forward_to=46'11))", 2023-11-05T08:33:46.304 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:59.233948+0000", 2023-11-05T08:33:46.304 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 46.922404970999999, 2023-11-05T08:33:46.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048084130000000001, 2023-11-05T08:33:46.304 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:46.304 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:46.305 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:46.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:46.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.233948+0000", 2023-11-05T08:33:46.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:46.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.233948+0000", 2023-11-05T08:33:46.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:46.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.233950+0000", 2023-11-05T08:33:46.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5969999999999999e-06 2023-11-05T08:33:46.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:46.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.234005+0000", 2023-11-05T08:33:46.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5414999999999997e-05 2023-11-05T08:33:46.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:46.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.234040+0000", 2023-11-05T08:33:46.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.455e-05 2023-11-05T08:33:46.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:46.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.234061+0000", 2023-11-05T08:33:46.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1036e-05 2023-11-05T08:33:46.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:46.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.234178+0000", 2023-11-05T08:33:46.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000116822 2023-11-05T08:33:46.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:46.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.234195+0000", 2023-11-05T08:33:46.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6801999999999999e-05 2023-11-05T08:33:46.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:46.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.238676+0000", 2023-11-05T08:33:46.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004481044 2023-11-05T08:33:46.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:46.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.238757+0000", 2023-11-05T08:33:46.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1147000000000001e-05 2023-11-05T08:33:46.314 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.314 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:46.314 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.315 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.16s1 46/45 ECSubWrite(tid=48, reqid=osd.11.0:47, at_version=0'0, trim_to=0'0, roll_forward_to=46'13))", 2023-11-05T08:33:46.315 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:59.240171+0000", 2023-11-05T08:33:46.315 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 46.916182364000001, 2023-11-05T08:33:46.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0045632529999999998, 2023-11-05T08:33:46.316 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:46.316 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:46.317 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:46.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:46.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240171+0000", 2023-11-05T08:33:46.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:46.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240171+0000", 2023-11-05T08:33:46.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:46.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240172+0000", 2023-11-05T08:33:46.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.282e-06 2023-11-05T08:33:46.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:46.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240193+0000", 2023-11-05T08:33:46.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0601000000000001e-05 2023-11-05T08:33:46.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:46.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240214+0000", 2023-11-05T08:33:46.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1574999999999999e-05 2023-11-05T08:33:46.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:46.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240230+0000", 2023-11-05T08:33:46.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5656999999999999e-05 2023-11-05T08:33:46.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:46.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240361+0000", 2023-11-05T08:33:46.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000130488 2023-11-05T08:33:46.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:46.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240372+0000", 2023-11-05T08:33:46.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1158999999999999e-05 2023-11-05T08:33:46.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:46.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.244663+0000", 2023-11-05T08:33:46.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004291563 2023-11-05T08:33:46.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:46.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.244734+0000", 2023-11-05T08:33:46.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0927999999999996e-05 2023-11-05T08:33:46.329 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.329 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:46.330 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.330 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.16s1 51/45 ECSubWrite(tid=58, reqid=osd.11.0:58, at_version=51'19, trim_to=0'0, roll_forward_to=49'17))", 2023-11-05T08:33:46.330 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:04.226221+0000", 2023-11-05T08:33:46.331 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 41.930132592, 2023-11-05T08:33:46.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.005713414, 2023-11-05T08:33:46.331 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:46.331 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:46.331 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:46.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:46.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226221+0000", 2023-11-05T08:33:46.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:46.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226221+0000", 2023-11-05T08:33:46.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:46.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226221+0000", 2023-11-05T08:33:46.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.53e-07 2023-11-05T08:33:46.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:46.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226253+0000", 2023-11-05T08:33:46.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1671999999999998e-05 2023-11-05T08:33:46.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:46.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226269+0000", 2023-11-05T08:33:46.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6150999999999998e-05 2023-11-05T08:33:46.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:46.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226282+0000", 2023-11-05T08:33:46.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3465000000000001e-05 2023-11-05T08:33:46.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:46.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226474+0000", 2023-11-05T08:33:46.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019140499999999999 2023-11-05T08:33:46.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:46.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226501+0000", 2023-11-05T08:33:46.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7523000000000001e-05 2023-11-05T08:33:46.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:46.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.231902+0000", 2023-11-05T08:33:46.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0054011019999999996 2023-11-05T08:33:46.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:46.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.231934+0000", 2023-11-05T08:33:46.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1742999999999997e-05 2023-11-05T08:33:46.343 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.343 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:46.343 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.344 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.9s4 52/45 ECSubWrite(tid=75, reqid=client.4691.0:341, at_version=52'24, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:46.345 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:05.257804+0000", 2023-11-05T08:33:46.345 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.898549404000001, 2023-11-05T08:33:46.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010353123000000001, 2023-11-05T08:33:46.345 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:46.345 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:46.346 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:46.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:46.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.257804+0000", 2023-11-05T08:33:46.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:46.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.257804+0000", 2023-11-05T08:33:46.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:46.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.257805+0000", 2023-11-05T08:33:46.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3399999999999999e-07 2023-11-05T08:33:46.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:46.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.258142+0000", 2023-11-05T08:33:46.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033764100000000002 2023-11-05T08:33:46.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:46.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.258157+0000", 2023-11-05T08:33:46.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4625e-05 2023-11-05T08:33:46.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.353 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:46.353 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.258168+0000", 2023-11-05T08:33:46.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0992000000000001e-05 2023-11-05T08:33:46.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:46.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.258371+0000", 2023-11-05T08:33:46.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020296299999999999 2023-11-05T08:33:46.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:46.356 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.258404+0000", 2023-11-05T08:33:46.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2765999999999998e-05 2023-11-05T08:33:46.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:46.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.268067+0000", 2023-11-05T08:33:46.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0096629479999999993 2023-11-05T08:33:46.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:46.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.268157+0000", 2023-11-05T08:33:46.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0554000000000001e-05 2023-11-05T08:33:46.358 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.358 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:46.358 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.359 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(osd.14.22:3 3.5s0 3:a287d13d:::smithi09039252-33:head [assert-version v4,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:46.359 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:05.264331+0000", 2023-11-05T08:33:46.359 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.892022799999999, 2023-11-05T08:33:46.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010947936, 2023-11-05T08:33:46.360 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:46.360 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:46.360 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:46.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:46.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.264331+0000", 2023-11-05T08:33:46.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:46.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.264331+0000", 2023-11-05T08:33:46.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:46.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.264334+0000", 2023-11-05T08:33:46.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.467e-06 2023-11-05T08:33:46.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:46.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.264350+0000", 2023-11-05T08:33:46.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6200000000000001e-05 2023-11-05T08:33:46.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:46.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.264387+0000", 2023-11-05T08:33:46.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6684999999999999e-05 2023-11-05T08:33:46.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:46.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.264489+0000", 2023-11-05T08:33:46.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010218 2023-11-05T08:33:46.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:46.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.265518+0000", 2023-11-05T08:33:46.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0010291899999999999 2023-11-05T08:33:46.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-11-05T08:33:46.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.265535+0000", 2023-11-05T08:33:46.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6697e-05 2023-11-05T08:33:46.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:46.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.269530+0000", 2023-11-05T08:33:46.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039951769999999999 2023-11-05T08:33:46.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:46.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.269646+0000", 2023-11-05T08:33:46.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011579299999999999 2023-11-05T08:33:46.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:46.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.275278+0000", 2023-11-05T08:33:46.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0056325469999999999 2023-11-05T08:33:46.371 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.371 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:46.372 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.372 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpReadReply(3.5s0 52/40 ECSubReadReply(tid=27, attrs_read=0))", 2023-11-05T08:33:46.372 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:05.271241+0000", 2023-11-05T08:33:46.372 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.885112384999999, 2023-11-05T08:33:46.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040795400000000004, 2023-11-05T08:33:46.373 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:46.373 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:46.373 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:46.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:46.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.271241+0000", 2023-11-05T08:33:46.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:46.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.271241+0000", 2023-11-05T08:33:46.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:46.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.271242+0000", 2023-11-05T08:33:46.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1370000000000001e-06 2023-11-05T08:33:46.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:46.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.272078+0000", 2023-11-05T08:33:46.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00083553299999999996 2023-11-05T08:33:46.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:46.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.272107+0000", 2023-11-05T08:33:46.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9195000000000001e-05 2023-11-05T08:33:46.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:46.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.272124+0000", 2023-11-05T08:33:46.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6959000000000001e-05 2023-11-05T08:33:46.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:46.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.272391+0000", 2023-11-05T08:33:46.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026688200000000002 2023-11-05T08:33:46.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:46.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.275320+0000", 2023-11-05T08:33:46.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029298340000000001 2023-11-05T08:33:46.381 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.381 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:46.381 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.382 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.16s1 56/45 ECSubWrite(tid=63, reqid=client.4691.0:369, at_version=56'20, trim_to=0'0, roll_forward_to=51'19))", 2023-11-05T08:33:46.382 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:09.292024+0000", 2023-11-05T08:33:46.382 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 36.864329024, 2023-11-05T08:33:46.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0035291490000000001, 2023-11-05T08:33:46.382 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:46.383 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:46.383 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:46.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:46.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.292024+0000", 2023-11-05T08:33:46.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:46.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.292024+0000", 2023-11-05T08:33:46.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:46.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.292025+0000", 2023-11-05T08:33:46.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.083e-06 2023-11-05T08:33:46.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:46.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.292178+0000", 2023-11-05T08:33:46.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015233600000000001 2023-11-05T08:33:46.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:46.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.292213+0000", 2023-11-05T08:33:46.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4779e-05 2023-11-05T08:33:46.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:46.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.292239+0000", 2023-11-05T08:33:46.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6758999999999999e-05 2023-11-05T08:33:46.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.389 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:46.389 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.292487+0000", 2023-11-05T08:33:46.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024808600000000001 2023-11-05T08:33:46.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:46.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.292519+0000", 2023-11-05T08:33:46.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2125999999999997e-05 2023-11-05T08:33:46.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:46.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.295403+0000", 2023-11-05T08:33:46.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028833539999999999 2023-11-05T08:33:46.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:46.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.295553+0000", 2023-11-05T08:33:46.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000150626 2023-11-05T08:33:46.392 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.392 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:46.392 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.393 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.16s1 56/45 ECSubWrite(tid=64, reqid=client.4691.0:370, at_version=56'21, trim_to=0'0, roll_forward_to=51'19))", 2023-11-05T08:33:46.393 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:09.297268+0000", 2023-11-05T08:33:46.393 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 36.859084987999999, 2023-11-05T08:33:46.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053142609999999998, 2023-11-05T08:33:46.394 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:46.394 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:46.394 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:46.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:46.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297268+0000", 2023-11-05T08:33:46.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:46.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297268+0000", 2023-11-05T08:33:46.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:46.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297270+0000", 2023-11-05T08:33:46.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7570000000000001e-06 2023-11-05T08:33:46.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:46.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297426+0000", 2023-11-05T08:33:46.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015544199999999999 2023-11-05T08:33:46.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:46.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297457+0000", 2023-11-05T08:33:46.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1615999999999997e-05 2023-11-05T08:33:46.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:46.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297480+0000", 2023-11-05T08:33:46.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3326e-05 2023-11-05T08:33:46.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:46.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.299414+0000", 2023-11-05T08:33:46.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001933518 2023-11-05T08:33:46.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:46.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.299445+0000", 2023-11-05T08:33:46.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0701999999999998e-05 2023-11-05T08:33:46.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:46.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.302505+0000", 2023-11-05T08:33:46.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030607590000000001 2023-11-05T08:33:46.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:46.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.302583+0000", 2023-11-05T08:33:46.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7140999999999999e-05 2023-11-05T08:33:46.407 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.407 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:46.407 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.408 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.9s4 92/45 ECSubWrite(tid=98, reqid=client.4691.0:662, at_version=92'38, trim_to=0'0, roll_forward_to=68'37))", 2023-11-05T08:33:46.408 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:45.749377+0000", 2023-11-05T08:33:46.408 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.406976273, 2023-11-05T08:33:46.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0045228969999999997, 2023-11-05T08:33:46.408 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:46.409 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:46.409 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:46.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:46.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:45.749377+0000", 2023-11-05T08:33:46.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:46.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:45.749377+0000", 2023-11-05T08:33:46.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:46.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:46.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:45.749377+0000", 2023-11-05T08:33:46.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.48e-07 2023-11-05T08:33:46.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:46.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:45.749892+0000", 2023-11-05T08:33:46.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051456699999999995 2023-11-05T08:33:46.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:46.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:45.749905+0000", 2023-11-05T08:33:46.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2813000000000001e-05 2023-11-05T08:33:46.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:46.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:45.749913+0000", 2023-11-05T08:33:46.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7790000000000001e-06 2023-11-05T08:33:46.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:46.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:45.750008+0000", 2023-11-05T08:33:46.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4896999999999996e-05 2023-11-05T08:33:46.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:46.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:45.750025+0000", 2023-11-05T08:33:46.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6376e-05 2023-11-05T08:33:46.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:46.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:45.753817+0000", 2023-11-05T08:33:46.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037921589999999998 2023-11-05T08:33:46.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:46.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:46.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:46.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:45.753900+0000", 2023-11-05T08:33:46.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3058000000000003e-05 2023-11-05T08:33:46.421 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.422 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:46.422 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.422 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:46.422 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:46.422 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:33:46.424 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5: {} 2023-11-05T08:33:46.424 INFO:teuthology.orchestra.run.smithi002.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-11-05T08:33:46.425 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6: {} 2023-11-05T08:33:46.426 INFO:teuthology.orchestra.run.smithi002.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-11-05T08:33:46.426 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-11-05T08:33:46.427 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7: {} 2023-11-05T08:33:46.427 INFO:teuthology.orchestra.run.smithi002.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-11-05T08:33:46.428 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:33:46.428 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-11-05T08:33:46.428 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-11-05T08:33:46.429 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:33:46.429 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2023-11-05T08:33:46.430 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8: {} 2023-11-05T08:33:46.430 INFO:teuthology.orchestra.run.smithi002.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-11-05T08:33:46.430 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:33:46.420+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:33:46.431 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9: {} 2023-11-05T08:33:46.431 INFO:teuthology.orchestra.run.smithi002.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-11-05T08:33:46.432 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10: {} 2023-11-05T08:33:46.432 INFO:teuthology.orchestra.run.smithi002.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-11-05T08:33:46.432 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11: {} 2023-11-05T08:33:46.433 INFO:teuthology.orchestra.run.smithi002.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-11-05T08:33:46.433 INFO:teuthology.orchestra.run.smithi002.stdout:osd.12: {} 2023-11-05T08:33:46.434 INFO:teuthology.orchestra.run.smithi002.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-11-05T08:33:46.434 INFO:teuthology.orchestra.run.smithi002.stdout:osd.13: {} 2023-11-05T08:33:46.434 INFO:teuthology.orchestra.run.smithi002.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-11-05T08:33:46.435 INFO:teuthology.orchestra.run.smithi002.stdout:osd.14: {} 2023-11-05T08:33:46.435 INFO:teuthology.orchestra.run.smithi002.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-11-05T08:33:46.436 INFO:teuthology.orchestra.run.smithi002.stdout:osd.15: {} 2023-11-05T08:33:46.436 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-05T08:33:46.437 INFO:teuthology.orchestra.run.smithi002.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-11-05T08:33:46.438 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-05T08:33:46.525 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:33:46.521+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:33:46.602 INFO:teuthology.orchestra.run.smithi002.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-05T08:33:46.604 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-05T08:33:46.605 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-11-05T08:33:46.648 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:33:46.645+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:33:46.674 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-11-05T08:33:46.677 INFO:tasks.rados.rados.0.smithi090.stdout:291: done (4 left) 2023-11-05T08:33:46.678 INFO:tasks.rados.rados.0.smithi090.stdout:292: done (3 left) 2023-11-05T08:33:46.678 INFO:tasks.rados.rados.0.smithi090.stdout:293: done (2 left) 2023-11-05T08:33:46.678 INFO:tasks.rados.rados.0.smithi090.stdout:294: done (1 left) 2023-11-05T08:33:46.678 INFO:tasks.rados.rados.0.smithi090.stdout:295: read oid 38 snap -1 2023-11-05T08:33:46.679 INFO:tasks.rados.rados.0.smithi090.stdout:295: expect (ObjNum 65 snap 5 seq_num 65) 2023-11-05T08:33:46.679 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 18 current snap is 30 2023-11-05T08:33:46.679 INFO:tasks.rados.rados.0.smithi090.stdout:296: seq_num 115 ranges {294912=770048,1064960=704512,1769472=458752} 2023-11-05T08:33:46.688 INFO:tasks.rados.rados.0.smithi090.stdout:296: writing smithi09039252-18 from 294912 to 1064960 tid 1 2023-11-05T08:33:46.697 INFO:tasks.rados.rados.0.smithi090.stdout:296: writing smithi09039252-18 from 1064960 to 1769472 tid 2 2023-11-05T08:33:46.703 INFO:tasks.rados.rados.0.smithi090.stdout:296: writing smithi09039252-18 from 1769472 to 2228224 tid 3 2023-11-05T08:33:46.704 INFO:tasks.rados.rados.0.smithi090.stdout:297: read oid 10 snap -1 2023-11-05T08:33:46.704 INFO:tasks.rados.rados.0.smithi090.stdout:297: expect (ObjNum 9 snap 0 seq_num 9) 2023-11-05T08:33:46.704 INFO:tasks.rados.rados.0.smithi090.stdout:298: read oid 50 snap -1 2023-11-05T08:33:46.705 INFO:tasks.rados.rados.0.smithi090.stdout:298: expect (ObjNum 91 snap 16 seq_num 91) 2023-11-05T08:33:46.705 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 5 current snap is 30 2023-11-05T08:33:46.705 INFO:tasks.rados.rados.0.smithi090.stdout:299: seq_num 116 ranges {0=786432,786432=458752,1245184=442368,1687552=262144} 2023-11-05T08:33:46.712 INFO:tasks.rados.rados.0.smithi090.stdout:299: writing smithi09039252-5 from 0 to 786432 tid 1 2023-11-05T08:33:46.717 INFO:tasks.rados.rados.0.smithi090.stdout:299: writing smithi09039252-5 from 786432 to 1245184 tid 2 2023-11-05T08:33:46.722 INFO:tasks.rados.rados.0.smithi090.stdout:299: writing smithi09039252-5 from 1245184 to 1687552 tid 3 2023-11-05T08:33:46.724 INFO:tasks.rados.rados.0.smithi090.stdout:299: writing smithi09039252-5 from 1687552 to 1949696 tid 4 2023-11-05T08:33:46.725 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 2 current snap is 30 2023-11-05T08:33:46.725 INFO:tasks.rados.rados.0.smithi090.stdout:300: seq_num 117 ranges {589824=425984,1015808=606208,1622016=540672,2162688=409600} 2023-11-05T08:33:46.726 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:33:46.724+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:33:46.735 INFO:tasks.rados.rados.0.smithi090.stdout:300: writing smithi09039252-2 from 589824 to 1015808 tid 1 2023-11-05T08:33:46.742 INFO:tasks.rados.rados.0.smithi090.stdout:300: writing smithi09039252-2 from 1015808 to 1622016 tid 2 2023-11-05T08:33:46.747 INFO:tasks.rados.rados.0.smithi090.stdout:300: writing smithi09039252-2 from 1622016 to 2162688 tid 3 2023-11-05T08:33:46.752 INFO:tasks.rados.rados.0.smithi090.stdout:300: writing smithi09039252-2 from 2162688 to 2572288 tid 4 2023-11-05T08:33:46.753 INFO:tasks.rados.rados.0.smithi090.stdout:301: delete oid 27 current snap is 30 2023-11-05T08:33:46.753 INFO:tasks.rados.rados.0.smithi090.stdout:296: finishing write tid 1 to smithi09039252-18 2023-11-05T08:33:46.753 INFO:tasks.rados.rados.0.smithi090.stdout:296: finishing write tid 2 to smithi09039252-18 2023-11-05T08:33:46.753 INFO:tasks.rados.rados.0.smithi090.stdout:296: finishing write tid 3 to smithi09039252-18 2023-11-05T08:33:46.753 INFO:tasks.rados.rados.0.smithi090.stdout:296: finishing write tid 4 to smithi09039252-18 2023-11-05T08:33:46.754 INFO:tasks.rados.rados.0.smithi090.stdout:296: finishing write tid 5 to smithi09039252-18 2023-11-05T08:33:46.754 INFO:tasks.rados.rados.0.smithi090.stdout:296: oid 18 updating version 0 to 37 2023-11-05T08:33:46.754 INFO:tasks.rados.rados.0.smithi090.stdout:296: oid 18 version 37 is already newer than 35 2023-11-05T08:33:46.754 INFO:tasks.rados.rados.0.smithi090.stdout:296: oid 18 version 37 is already newer than 34 2023-11-05T08:33:46.755 INFO:tasks.rados.rados.0.smithi090.stdout:296: oid 18 version 37 is already newer than 36 2023-11-05T08:33:46.755 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 18 v 37 (ObjNum 115 snap 30 seq_num 115) dirty exists 2023-11-05T08:33:46.755 INFO:tasks.rados.rados.0.smithi090.stdout:296: left oid 18 (ObjNum 115 snap 30 seq_num 115) 2023-11-05T08:33:46.755 INFO:tasks.rados.rados.0.smithi090.stdout:299: finishing write tid 1 to smithi09039252-5 2023-11-05T08:33:46.755 INFO:tasks.rados.rados.0.smithi090.stdout:299: finishing write tid 2 to smithi09039252-5 2023-11-05T08:33:46.756 INFO:tasks.rados.rados.0.smithi090.stdout:299: finishing write tid 3 to smithi09039252-5 2023-11-05T08:33:46.756 INFO:tasks.rados.rados.0.smithi090.stdout:299: finishing write tid 4 to smithi09039252-5 2023-11-05T08:33:46.756 INFO:tasks.rados.rados.0.smithi090.stdout:299: finishing write tid 5 to smithi09039252-5 2023-11-05T08:33:46.756 INFO:tasks.rados.rados.0.smithi090.stdout:299: finishing write tid 6 to smithi09039252-5 2023-11-05T08:33:46.756 INFO:tasks.rados.rados.0.smithi090.stdout:299: oid 5 updating version 0 to 54 2023-11-05T08:33:46.756 INFO:tasks.rados.rados.0.smithi090.stdout:299: oid 5 updating version 54 to 56 2023-11-05T08:33:46.757 INFO:tasks.rados.rados.0.smithi090.stdout:299: oid 5 version 56 is already newer than 53 2023-11-05T08:33:46.757 INFO:tasks.rados.rados.0.smithi090.stdout:299: oid 5 version 56 is already newer than 52 2023-11-05T08:33:46.757 INFO:tasks.rados.rados.0.smithi090.stdout:299: oid 5 version 56 is already newer than 55 2023-11-05T08:33:46.757 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 5 v 56 (ObjNum 116 snap 30 seq_num 116) dirty exists 2023-11-05T08:33:46.757 INFO:tasks.rados.rados.0.smithi090.stdout:299: left oid 5 (ObjNum 116 snap 30 seq_num 116) 2023-11-05T08:33:46.758 INFO:tasks.rados.rados.0.smithi090.stdout:300: finishing write tid 1 to smithi09039252-2 2023-11-05T08:33:46.758 INFO:tasks.rados.rados.0.smithi090.stdout:300: finishing write tid 2 to smithi09039252-2 2023-11-05T08:33:46.759 INFO:tasks.rados.rados.0.smithi090.stdout:300: finishing write tid 3 to smithi09039252-2 2023-11-05T08:33:46.760 INFO:tasks.rados.rados.0.smithi090.stdout:296: done (7 left) 2023-11-05T08:33:46.760 INFO:tasks.rados.rados.0.smithi090.stdout:299: done (6 left) 2023-11-05T08:33:46.760 INFO:tasks.rados.rados.0.smithi090.stdout:301: done (5 left) 2023-11-05T08:33:46.760 INFO:tasks.rados.rados.0.smithi090.stdout:302: read oid 48 snap 4 2023-11-05T08:33:46.760 INFO:tasks.rados.rados.0.smithi090.stdout:302: expect (ObjNum 47 snap 0 seq_num 47) 2023-11-05T08:33:46.761 INFO:tasks.rados.rados.0.smithi090.stdout:303: rmattr oid 12 current snap is 30 2023-11-05T08:33:46.761 INFO:tasks.rados.rados.0.smithi090.stdout:303: done (6 left) 2023-11-05T08:33:46.761 INFO:tasks.rados.rados.0.smithi090.stdout:304: snap_remove snap 29 2023-11-05T08:33:46.767 INFO:teuthology.orchestra.run.smithi002.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-05T08:33:46.769 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-05T08:33:46.770 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-11-05T08:33:46.828 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:33:46.825+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:33:46.874 INFO:teuthology.orchestra.run.smithi002.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-05T08:33:46.876 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-05T08:33:46.877 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-11-05T08:33:46.928 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:33:46.924+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:33:46.995 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:46.995 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:47.009 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-11-05T08:33:47.060 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:33:47.057+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:33:47.129 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~51s 2023-11-05T08:33:47.130 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:33:47.125+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:33:47.130 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:47.130 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:47.145 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-11-05T08:33:47.231 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:33:47.225+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:33:47.315 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:47.316 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:47.330 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2023-11-05T08:33:47.331 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:33:47.329+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:33:47.432 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:33:47.428+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:33:47.457 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:47.458 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:47.472 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2023-11-05T08:33:47.533 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:47.528+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:47.599 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:47.599 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:47.614 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2023-11-05T08:33:47.634 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:33:47.629+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:33:47.669 INFO:teuthology.orchestra.run.smithi002.stderr:reweighted osd.0 to 1 (10000) 2023-11-05T08:33:47.669 INFO:teuthology.orchestra.run.smithi002.stderr:noscrub is unset 2023-11-05T08:33:47.678 INFO:tasks.rados.rados.0.smithi090.stdout:304: done (6 left) 2023-11-05T08:33:47.678 INFO:tasks.rados.rados.0.smithi090.stdout:305: snap_create 2023-11-05T08:33:47.678 INFO:tasks.rados.rados.0.smithi090.stdout:300: finishing write tid 4 to smithi09039252-2 2023-11-05T08:33:47.679 INFO:tasks.rados.rados.0.smithi090.stdout:300: finishing write tid 5 to smithi09039252-2 2023-11-05T08:33:47.679 INFO:tasks.rados.rados.0.smithi090.stdout:302: expect (ObjNum 47 snap 0 seq_num 47) 2023-11-05T08:33:47.704 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2023-11-05T08:33:47.735 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:33:47.732+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:33:47.744 INFO:tasks.rados.rados.0.smithi090.stdout:300: finishing write tid 6 to smithi09039252-2 2023-11-05T08:33:47.744 INFO:tasks.rados.rados.0.smithi090.stdout:300: oid 2 updating version 0 to 25 2023-11-05T08:33:47.744 INFO:tasks.rados.rados.0.smithi090.stdout:300: oid 2 version 25 is already newer than 22 2023-11-05T08:33:47.744 INFO:tasks.rados.rados.0.smithi090.stdout:300: oid 2 version 25 is already newer than 21 2023-11-05T08:33:47.745 INFO:tasks.rados.rados.0.smithi090.stdout:300: oid 2 version 25 is already newer than 24 2023-11-05T08:33:47.745 INFO:tasks.rados.rados.0.smithi090.stdout:300: oid 2 version 25 is already newer than 23 2023-11-05T08:33:47.745 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 2 v 25 (ObjNum 117 snap 30 seq_num 117) dirty exists 2023-11-05T08:33:47.745 INFO:tasks.rados.rados.0.smithi090.stdout:300: left oid 2 (ObjNum 117 snap 30 seq_num 117) 2023-11-05T08:33:47.749 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:47.749 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:47.764 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2023-11-05T08:33:47.836 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:33:47.832+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:33:47.928 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:47.928 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:47.937 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:33:47.933+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:33:47.942 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2023-11-05T08:33:48.037 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:33:48.033+0000 7fdc5cd58640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:33:48.104 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:48.104 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:48.118 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2023-11-05T08:33:48.138 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:33:48.133+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:33:48.240 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-11-05T08:33:48.238+0000 7f72b2c5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 42525) UID: 0 2023-11-05T08:33:48.285 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:48.285 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:48.302 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-11-05T08:33:48.340 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:48.338+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:48.438 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:48.438 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:48.439 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-05T08:33:48.441 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:33:48.437+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:33:48.453 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-11-05T08:33:48.541 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:33:48.537+0000 7fdc5cd58640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:33:48.579 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:48.579 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:48.580 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-11-05T08:33:48.594 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-11-05T08:33:48.599 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1: {} 2023-11-05T08:33:48.600 INFO:teuthology.orchestra.run.smithi002.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-11-05T08:33:48.620 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2: {} 2023-11-05T08:33:48.621 INFO:teuthology.orchestra.run.smithi002.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-11-05T08:33:48.641 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3: {} 2023-11-05T08:33:48.641 INFO:teuthology.orchestra.run.smithi002.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-11-05T08:33:48.642 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:48.640+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:48.658 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4: {} 2023-11-05T08:33:48.658 INFO:teuthology.orchestra.run.smithi002.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-11-05T08:33:48.677 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5: {} 2023-11-05T08:33:48.677 INFO:teuthology.orchestra.run.smithi002.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-11-05T08:33:48.682 INFO:teuthology.orchestra.run.smithi002.stderr:reweighted osd.1 to 1 (10000) 2023-11-05T08:33:48.697 INFO:tasks.rados.rados.0.smithi090.stdout:300: done (6 left) 2023-11-05T08:33:48.697 INFO:tasks.rados.rados.0.smithi090.stdout:302: done (5 left) 2023-11-05T08:33:48.697 INFO:tasks.rados.rados.0.smithi090.stdout:305: done (4 left) 2023-11-05T08:33:48.697 INFO:tasks.rados.rados.0.smithi090.stdout:306: snap_create 2023-11-05T08:33:48.704 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2023-11-05T08:33:48.705 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6: {} 2023-11-05T08:33:48.705 INFO:teuthology.orchestra.run.smithi002.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-11-05T08:33:48.724 INFO:tasks.rados.rados.0.smithi090.stdout:297: expect (ObjNum 9 snap 0 seq_num 9) 2023-11-05T08:33:48.732 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7: {} 2023-11-05T08:33:48.732 INFO:teuthology.orchestra.run.smithi002.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-11-05T08:33:48.739 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:33:48.740 INFO:teuthology.orchestra.run.smithi053.stdout: "size": 20, 2023-11-05T08:33:48.740 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 600, 2023-11-05T08:33:48.740 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2023-11-05T08:33:48.740 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.740 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es2 41/30 ECSubWrite(tid=40, reqid=osd.2.0:40, at_version=41'27, trim_to=0'0, roll_forward_to=40'25))", 2023-11-05T08:33:48.741 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.127197+0000", 2023-11-05T08:33:48.741 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 54.606060198000002, 2023-11-05T08:33:48.741 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.019700031, 2023-11-05T08:33:48.741 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:48.741 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:48.742 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:48.742 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.742 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:48.742 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127197+0000", 2023-11-05T08:33:48.742 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.743 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.743 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.743 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:48.743 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127197+0000", 2023-11-05T08:33:48.743 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.743 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.744 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.744 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:48.744 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127198+0000", 2023-11-05T08:33:48.744 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.0999999999999998e-07 2023-11-05T08:33:48.744 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.745 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.745 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:48.745 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127255+0000", 2023-11-05T08:33:48.745 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6762000000000003e-05 2023-11-05T08:33:48.745 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.745 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.746 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:48.746 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127291+0000", 2023-11-05T08:33:48.746 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5948999999999998e-05 2023-11-05T08:33:48.746 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.746 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.747 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:48.747 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127314+0000", 2023-11-05T08:33:48.747 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3686e-05 2023-11-05T08:33:48.747 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.747 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.748 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:48.748 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127534+0000", 2023-11-05T08:33:48.748 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00021922700000000001 2023-11-05T08:33:48.748 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.748 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.748 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:48.749 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.127562+0000", 2023-11-05T08:33:48.749 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8099e-05 2023-11-05T08:33:48.749 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.749 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.749 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:48.750 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.146742+0000", 2023-11-05T08:33:48.750 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.019180135000000001 2023-11-05T08:33:48.750 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.750 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.750 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:48.751 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.146897+0000", 2023-11-05T08:33:48.751 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000155463 2023-11-05T08:33:48.751 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.751 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:48.751 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.751 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.752 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.752 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as2 41/33 ECSubWrite(tid=35, reqid=osd.7.0:35, at_version=41'12, trim_to=0'0, roll_forward_to=40'10))", 2023-11-05T08:33:48.752 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.130596+0000", 2023-11-05T08:33:48.752 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 54.602660974999999, 2023-11-05T08:33:48.752 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022182642999999998, 2023-11-05T08:33:48.753 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:48.753 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:48.753 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:48.753 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.753 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:48.754 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130596+0000", 2023-11-05T08:33:48.754 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.754 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.754 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.754 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:48.754 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130596+0000", 2023-11-05T08:33:48.755 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.755 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.755 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.755 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:48.755 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130597+0000", 2023-11-05T08:33:48.756 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.7600000000000006e-07 2023-11-05T08:33:48.756 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.756 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.756 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:48.756 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130660+0000", 2023-11-05T08:33:48.756 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3015000000000006e-05 2023-11-05T08:33:48.757 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.757 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.757 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:48.757 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130725+0000", 2023-11-05T08:33:48.757 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4988999999999997e-05 2023-11-05T08:33:48.758 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.758 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.758 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:48.758 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130750+0000", 2023-11-05T08:33:48.758 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4802000000000001e-05 2023-11-05T08:33:48.759 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.759 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.759 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:48.759 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.131037+0000", 2023-11-05T08:33:48.759 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00028702900000000001 2023-11-05T08:33:48.759 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.760 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.760 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:48.760 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.131064+0000", 2023-11-05T08:33:48.760 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7022e-05 2023-11-05T08:33:48.760 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.761 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.761 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:48.761 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.152706+0000", 2023-11-05T08:33:48.761 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.021641734999999999 2023-11-05T08:33:48.761 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.761 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.762 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:48.762 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.152779+0000", 2023-11-05T08:33:48.762 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.3075000000000001e-05 2023-11-05T08:33:48.762 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.762 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:48.762 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.763 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.763 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.763 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as2 41/33 ECSubWrite(tid=36, reqid=osd.7.0:36, at_version=41'14, trim_to=0'0, roll_forward_to=40'10))", 2023-11-05T08:33:48.763 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.130806+0000", 2023-11-05T08:33:48.763 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 54.602451094999999, 2023-11-05T08:33:48.764 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.035467152000000002, 2023-11-05T08:33:48.764 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:48.764 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:48.764 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:48.764 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.765 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:48.765 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130806+0000", 2023-11-05T08:33:48.765 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.765 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.765 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.766 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:48.766 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130806+0000", 2023-11-05T08:33:48.766 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.766 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.766 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.766 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:48.767 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130807+0000", 2023-11-05T08:33:48.767 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1000000000000001e-06 2023-11-05T08:33:48.767 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.767 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.767 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:48.768 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130852+0000", 2023-11-05T08:33:48.768 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4428999999999998e-05 2023-11-05T08:33:48.768 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.768 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.768 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:48.769 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130876+0000", 2023-11-05T08:33:48.769 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3975999999999999e-05 2023-11-05T08:33:48.769 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.769 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.769 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:48.769 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.130896+0000", 2023-11-05T08:33:48.770 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9905999999999999e-05 2023-11-05T08:33:48.770 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.770 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.770 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:48.770 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.143384+0000", 2023-11-05T08:33:48.771 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012488727999999999 2023-11-05T08:33:48.771 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.771 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.771 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:48.771 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.143412+0000", 2023-11-05T08:33:48.772 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7988999999999999e-05 2023-11-05T08:33:48.772 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.772 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.772 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:48.772 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.166186+0000", 2023-11-05T08:33:48.772 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022773787 2023-11-05T08:33:48.773 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.773 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.773 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:48.773 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.166273+0000", 2023-11-05T08:33:48.773 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.7237000000000005e-05 2023-11-05T08:33:48.774 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.774 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:48.774 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.774 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.774 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.775 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es2 41/30 ECSubWrite(tid=41, reqid=osd.2.0:40, at_version=0'0, trim_to=0'0, roll_forward_to=41'27))", 2023-11-05T08:33:48.775 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.148654+0000", 2023-11-05T08:33:48.775 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 54.584603451, 2023-11-05T08:33:48.775 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017394370999999999, 2023-11-05T08:33:48.775 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:48.776 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:48.776 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:48.776 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.776 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:48.776 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148654+0000", 2023-11-05T08:33:48.777 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.777 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.777 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.777 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:48.777 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148654+0000", 2023-11-05T08:33:48.778 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.778 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.778 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.778 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:48.778 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148656+0000", 2023-11-05T08:33:48.779 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.677e-06 2023-11-05T08:33:48.779 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.779 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.779 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:48.779 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148683+0000", 2023-11-05T08:33:48.779 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7246e-05 2023-11-05T08:33:48.780 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.780 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.780 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:48.780 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148715+0000", 2023-11-05T08:33:48.780 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2518999999999998e-05 2023-11-05T08:33:48.781 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.781 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.781 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:48.781 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148742+0000", 2023-11-05T08:33:48.781 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6876000000000002e-05 2023-11-05T08:33:48.781 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.782 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.782 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:48.782 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148890+0000", 2023-11-05T08:33:48.782 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014771599999999999 2023-11-05T08:33:48.782 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.783 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.783 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:48.783 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148915+0000", 2023-11-05T08:33:48.783 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5099000000000001e-05 2023-11-05T08:33:48.783 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.784 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.784 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:48.784 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.165914+0000", 2023-11-05T08:33:48.784 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.016999018000000001 2023-11-05T08:33:48.784 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.785 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.785 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:48.785 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.166048+0000", 2023-11-05T08:33:48.785 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013422 2023-11-05T08:33:48.785 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.786 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:48.786 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.786 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.786 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.786 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.2s5 41/40 ECSubWrite(tid=187, reqid=client.4691.0:292, at_version=41'9, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:48.787 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.162642+0000", 2023-11-05T08:33:48.787 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 54.570615052000001, 2023-11-05T08:33:48.787 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010859549, 2023-11-05T08:33:48.787 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:48.787 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:48.788 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:48.788 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.788 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:48.788 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.162642+0000", 2023-11-05T08:33:48.788 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.789 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.789 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.789 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:48.789 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.162642+0000", 2023-11-05T08:33:48.789 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.789 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.790 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.790 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:48.790 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.162644+0000", 2023-11-05T08:33:48.790 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.823e-06 2023-11-05T08:33:48.790 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.791 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.791 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:48.791 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.162700+0000", 2023-11-05T08:33:48.791 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5863999999999999e-05 2023-11-05T08:33:48.791 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.792 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.792 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:48.792 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.162731+0000", 2023-11-05T08:33:48.792 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0731000000000001e-05 2023-11-05T08:33:48.792 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.792 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.793 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:48.793 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.162754+0000", 2023-11-05T08:33:48.793 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3234000000000002e-05 2023-11-05T08:33:48.793 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.794 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.794 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:48.794 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.162988+0000", 2023-11-05T08:33:48.794 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00023393399999999999 2023-11-05T08:33:48.794 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.795 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.795 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:48.795 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.163016+0000", 2023-11-05T08:33:48.795 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8309e-05 2023-11-05T08:33:48.795 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.795 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.796 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:48.796 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.173420+0000", 2023-11-05T08:33:48.796 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010403802 2023-11-05T08:33:48.796 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.796 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.797 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:48.797 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.173502+0000", 2023-11-05T08:33:48.797 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.1852000000000002e-05 2023-11-05T08:33:48.797 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.797 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:48.797 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.798 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.798 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.798 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as2 41/33 ECSubWrite(tid=37, reqid=osd.7.0:36, at_version=0'0, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:33:48.798 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.167511+0000", 2023-11-05T08:33:48.799 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 54.565746613999998, 2023-11-05T08:33:48.799 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011732691, 2023-11-05T08:33:48.799 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:48.799 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:48.799 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:48.800 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.800 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:48.800 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167511+0000", 2023-11-05T08:33:48.800 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.800 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.800 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.801 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:48.801 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167511+0000", 2023-11-05T08:33:48.801 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.801 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.801 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.802 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:48.802 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167513+0000", 2023-11-05T08:33:48.802 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.0829999999999998e-06 2023-11-05T08:33:48.802 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.802 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.803 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:48.803 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167540+0000", 2023-11-05T08:33:48.803 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7095999999999999e-05 2023-11-05T08:33:48.803 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.803 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.804 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:48.804 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167572+0000", 2023-11-05T08:33:48.804 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2462000000000002e-05 2023-11-05T08:33:48.804 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.804 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.805 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:48.805 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167594+0000", 2023-11-05T08:33:48.805 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1316999999999999e-05 2023-11-05T08:33:48.805 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.805 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.806 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:48.806 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167791+0000", 2023-11-05T08:33:48.806 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00019748199999999999 2023-11-05T08:33:48.806 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.806 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.806 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:48.807 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.167815+0000", 2023-11-05T08:33:48.807 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3536999999999998e-05 2023-11-05T08:33:48.807 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.807 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.807 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:48.808 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.179154+0000", 2023-11-05T08:33:48.808 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011339373999999999 2023-11-05T08:33:48.808 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.808 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.808 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:48.809 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.179244+0000", 2023-11-05T08:33:48.809 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9339999999999995e-05 2023-11-05T08:33:48.809 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.809 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:48.809 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.809 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.810 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.810 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.2s5 41/40 ECSubWrite(tid=188, reqid=osd.13.0:188, at_version=41'11, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:48.810 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.171651+0000", 2023-11-05T08:33:48.810 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 54.561606027000003, 2023-11-05T08:33:48.810 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0076644390000000003, 2023-11-05T08:33:48.811 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:48.811 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:48.811 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:48.811 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.811 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:48.812 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171651+0000", 2023-11-05T08:33:48.812 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.812 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.812 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.812 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:48.813 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171651+0000", 2023-11-05T08:33:48.813 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.813 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.813 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.813 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:48.813 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171653+0000", 2023-11-05T08:33:48.814 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5039999999999999e-06 2023-11-05T08:33:48.814 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.814 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.814 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:48.814 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171700+0000", 2023-11-05T08:33:48.815 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.6989e-05 2023-11-05T08:33:48.815 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.815 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.815 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:48.815 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171731+0000", 2023-11-05T08:33:48.816 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1310999999999999e-05 2023-11-05T08:33:48.816 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.816 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.816 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:48.816 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171750+0000", 2023-11-05T08:33:48.817 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9188e-05 2023-11-05T08:33:48.817 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.817 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.817 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:48.817 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171900+0000", 2023-11-05T08:33:48.818 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00015010699999999999 2023-11-05T08:33:48.818 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.818 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.818 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:48.818 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.171920+0000", 2023-11-05T08:33:48.819 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9626000000000001e-05 2023-11-05T08:33:48.819 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.819 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.819 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:48.819 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.179250+0000", 2023-11-05T08:33:48.820 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0073298900000000004 2023-11-05T08:33:48.820 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.820 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.820 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:48.820 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.179316+0000", 2023-11-05T08:33:48.820 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5823999999999999e-05 2023-11-05T08:33:48.821 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.821 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:48.821 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.821 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.821 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.822 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as2 42/33 ECSubWrite(tid=38, reqid=osd.7.0:38, at_version=42'16, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:33:48.822 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:55.121755+0000", 2023-11-05T08:33:48.822 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 53.611502027, 2023-11-05T08:33:48.822 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011609566, 2023-11-05T08:33:48.822 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:48.823 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:48.823 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:48.823 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.823 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:48.823 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.121755+0000", 2023-11-05T08:33:48.824 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.824 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.824 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.824 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:48.824 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.121755+0000", 2023-11-05T08:33:48.824 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.825 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.825 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.825 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:48.825 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.121756+0000", 2023-11-05T08:33:48.825 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.0299999999999998e-07 2023-11-05T08:33:48.826 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.826 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.826 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:48.826 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.121810+0000", 2023-11-05T08:33:48.826 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3918000000000001e-05 2023-11-05T08:33:48.827 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.827 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.827 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:48.827 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.121861+0000", 2023-11-05T08:33:48.827 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.0846000000000001e-05 2023-11-05T08:33:48.827 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.828 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.828 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:48.828 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.121889+0000", 2023-11-05T08:33:48.828 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8359000000000001e-05 2023-11-05T08:33:48.828 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.829 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.829 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:48.829 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.122158+0000", 2023-11-05T08:33:48.829 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00026898800000000001 2023-11-05T08:33:48.829 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.829 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.830 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:48.830 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.122192+0000", 2023-11-05T08:33:48.830 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3259000000000001e-05 2023-11-05T08:33:48.830 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.830 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.831 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:48.831 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.133269+0000", 2023-11-05T08:33:48.831 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011076979000000001 2023-11-05T08:33:48.831 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.831 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.832 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:48.832 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.133365+0000", 2023-11-05T08:33:48.832 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6414000000000002e-05 2023-11-05T08:33:48.832 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.832 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:48.833 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.833 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.833 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.833 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as2 42/33 ECSubWrite(tid=39, reqid=osd.7.0:39, at_version=42'18, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:33:48.833 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:55.124648+0000", 2023-11-05T08:33:48.833 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 53.608609166999997, 2023-11-05T08:33:48.834 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.025426339999999999, 2023-11-05T08:33:48.834 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:48.834 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:48.834 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:48.834 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.835 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:48.835 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.124648+0000", 2023-11-05T08:33:48.835 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.835 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.835 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.835 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:48.836 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.124648+0000", 2023-11-05T08:33:48.836 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.836 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.836 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.836 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:48.837 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.124650+0000", 2023-11-05T08:33:48.837 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.463e-06 2023-11-05T08:33:48.837 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.837 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.837 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:48.838 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.124710+0000", 2023-11-05T08:33:48.838 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0001999999999999e-05 2023-11-05T08:33:48.838 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.838 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.838 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:48.838 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.124739+0000", 2023-11-05T08:33:48.839 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9278999999999999e-05 2023-11-05T08:33:48.839 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.839 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.839 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:48.839 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.124763+0000", 2023-11-05T08:33:48.839 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4036000000000002e-05 2023-11-05T08:33:48.840 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.840 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.840 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:48.840 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.125122+0000", 2023-11-05T08:33:48.840 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00035926900000000001 2023-11-05T08:33:48.841 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.841 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.841 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:48.841 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.125149+0000", 2023-11-05T08:33:48.841 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7192999999999999e-05 2023-11-05T08:33:48.842 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.842 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.842 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:48.842 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.149969+0000", 2023-11-05T08:33:48.842 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.024819074 2023-11-05T08:33:48.843 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.843 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.843 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:48.843 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.150075+0000", 2023-11-05T08:33:48.843 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000106024 2023-11-05T08:33:48.843 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.844 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:48.844 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.844 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.844 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.844 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es2 42/30 ECSubWrite(tid=43, reqid=osd.2.0:43, at_version=42'29, trim_to=0'0, roll_forward_to=41'27))", 2023-11-05T08:33:48.845 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:55.134544+0000", 2023-11-05T08:33:48.845 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 53.598713246999999, 2023-11-05T08:33:48.845 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.02091324, 2023-11-05T08:33:48.845 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:48.845 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:48.845 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:48.846 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.846 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:48.846 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134544+0000", 2023-11-05T08:33:48.846 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.846 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.847 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.847 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:48.847 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134544+0000", 2023-11-05T08:33:48.847 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.847 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.848 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.848 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:48.848 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134545+0000", 2023-11-05T08:33:48.848 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3499999999999996e-07 2023-11-05T08:33:48.848 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.849 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.849 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:48.849 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134593+0000", 2023-11-05T08:33:48.849 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.8538999999999997e-05 2023-11-05T08:33:48.849 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.849 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.850 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:48.850 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134621+0000", 2023-11-05T08:33:48.850 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7939000000000001e-05 2023-11-05T08:33:48.850 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.850 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.851 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:48.851 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134641+0000", 2023-11-05T08:33:48.851 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9352999999999999e-05 2023-11-05T08:33:48.851 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.851 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.852 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:48.852 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134788+0000", 2023-11-05T08:33:48.852 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000147806 2023-11-05T08:33:48.852 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.852 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.853 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:48.853 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134850+0000", 2023-11-05T08:33:48.853 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1614999999999999e-05 2023-11-05T08:33:48.853 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.853 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.854 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:48.854 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.155400+0000", 2023-11-05T08:33:48.854 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.020550023000000001 2023-11-05T08:33:48.854 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.854 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.854 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:48.855 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.155457+0000", 2023-11-05T08:33:48.855 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7330000000000002e-05 2023-11-05T08:33:48.855 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.855 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:48.855 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.856 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.856 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.856 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.2s5 42/40 ECSubWrite(tid=192, reqid=osd.13.0:192, at_version=42'13, trim_to=0'0, roll_forward_to=41'11))", 2023-11-05T08:33:48.856 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:55.150316+0000", 2023-11-05T08:33:48.856 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 53.582941751, 2023-11-05T08:33:48.856 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0088926939999999996, 2023-11-05T08:33:48.857 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:48.857 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:48.857 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:48.857 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.857 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:48.858 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.150316+0000", 2023-11-05T08:33:48.858 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.858 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.858 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.858 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:48.858 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.150316+0000", 2023-11-05T08:33:48.859 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.859 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.859 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.859 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:48.859 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.150318+0000", 2023-11-05T08:33:48.860 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.0140000000000001e-06 2023-11-05T08:33:48.860 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.860 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.860 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:48.860 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.150377+0000", 2023-11-05T08:33:48.861 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9231000000000003e-05 2023-11-05T08:33:48.861 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.861 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.861 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:48.861 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.150411+0000", 2023-11-05T08:33:48.861 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4279000000000002e-05 2023-11-05T08:33:48.862 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.862 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.862 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:48.862 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.150435+0000", 2023-11-05T08:33:48.862 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4165999999999999e-05 2023-11-05T08:33:48.863 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.863 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.863 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:48.863 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.152989+0000", 2023-11-05T08:33:48.863 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0025534889999999999 2023-11-05T08:33:48.863 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.864 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.864 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:48.864 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.153035+0000", 2023-11-05T08:33:48.864 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.6078000000000002e-05 2023-11-05T08:33:48.864 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.865 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.865 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:48.865 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.159134+0000", 2023-11-05T08:33:48.865 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0060986069999999998 2023-11-05T08:33:48.865 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.865 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.866 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:48.866 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.159208+0000", 2023-11-05T08:33:48.866 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.483e-05 2023-11-05T08:33:48.866 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.866 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:48.867 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.867 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.867 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.867 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as2 42/33 ECSubWrite(tid=40, reqid=osd.7.0:39, at_version=0'0, trim_to=0'0, roll_forward_to=42'18))", 2023-11-05T08:33:48.867 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:55.151263+0000", 2023-11-05T08:33:48.868 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 53.581994318, 2023-11-05T08:33:48.868 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0080059020000000005, 2023-11-05T08:33:48.868 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:48.868 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:48.868 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:48.869 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.869 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:48.869 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.151263+0000", 2023-11-05T08:33:48.869 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.869 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.869 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.870 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:48.870 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.151263+0000", 2023-11-05T08:33:48.870 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.870 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.870 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.871 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:48.871 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.151265+0000", 2023-11-05T08:33:48.871 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9939999999999999e-06 2023-11-05T08:33:48.871 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.871 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.871 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:48.872 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.151295+0000", 2023-11-05T08:33:48.872 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0355e-05 2023-11-05T08:33:48.872 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.872 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.872 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:48.873 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.151328+0000", 2023-11-05T08:33:48.873 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2246000000000003e-05 2023-11-05T08:33:48.873 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.873 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.873 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:48.874 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.151359+0000", 2023-11-05T08:33:48.874 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1458999999999998e-05 2023-11-05T08:33:48.874 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.874 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.874 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:48.875 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.153854+0000", 2023-11-05T08:33:48.875 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002495115 2023-11-05T08:33:48.875 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.875 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.875 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:48.875 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.153879+0000", 2023-11-05T08:33:48.876 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5065000000000001e-05 2023-11-05T08:33:48.876 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.876 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.876 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:48.876 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.159215+0000", 2023-11-05T08:33:48.877 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0053359690000000003 2023-11-05T08:33:48.877 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.877 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.877 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:48.877 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.159269+0000", 2023-11-05T08:33:48.878 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3699000000000002e-05 2023-11-05T08:33:48.878 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.878 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:48.878 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.878 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.878 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.879 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as2 52/45 ECSubWrite(tid=45, reqid=client.4691.0:349, at_version=52'21, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:48.879 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:05.262984+0000", 2023-11-05T08:33:48.879 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 43.470273675000001, 2023-11-05T08:33:48.879 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0076861730000000001, 2023-11-05T08:33:48.879 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:48.880 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:48.880 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:48.880 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.880 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:48.880 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.262984+0000", 2023-11-05T08:33:48.881 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.881 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.881 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.881 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:48.881 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.262984+0000", 2023-11-05T08:33:48.882 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.882 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.882 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.882 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:48.882 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.262985+0000", 2023-11-05T08:33:48.883 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.8400000000000002e-07 2023-11-05T08:33:48.883 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.883 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.883 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:48.883 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.263110+0000", 2023-11-05T08:33:48.884 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012501999999999999 2023-11-05T08:33:48.884 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.884 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.884 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:48.884 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.263149+0000", 2023-11-05T08:33:48.884 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8899000000000001e-05 2023-11-05T08:33:48.885 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.885 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.885 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:48.885 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.263173+0000", 2023-11-05T08:33:48.885 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4243000000000002e-05 2023-11-05T08:33:48.886 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.886 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.886 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:48.886 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.263816+0000", 2023-11-05T08:33:48.886 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00064262900000000003 2023-11-05T08:33:48.887 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.887 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.887 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:48.887 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.263830+0000", 2023-11-05T08:33:48.887 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4676e-05 2023-11-05T08:33:48.887 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.888 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.888 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:48.888 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.270569+0000", 2023-11-05T08:33:48.888 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0067386939999999999 2023-11-05T08:33:48.888 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.889 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.889 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:48.889 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.270670+0000", 2023-11-05T08:33:48.889 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000101028 2023-11-05T08:33:48.889 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.889 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:48.890 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.890 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.890 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.890 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.bs2 66/37 ECSubWrite(tid=108, reqid=client.4691.0:496, at_version=66'23, trim_to=0'0, roll_forward_to=65'21))", 2023-11-05T08:33:48.890 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.398619+0000", 2023-11-05T08:33:48.891 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 29.334638772000002, 2023-11-05T08:33:48.891 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.016555743000000001, 2023-11-05T08:33:48.891 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:48.891 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:48.891 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:48.891 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.892 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:48.892 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.398619+0000", 2023-11-05T08:33:48.892 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.892 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.892 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.893 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:48.893 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.398619+0000", 2023-11-05T08:33:48.893 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.893 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.893 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.893 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:48.894 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.398629+0000", 2023-11-05T08:33:48.894 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.9529999999999994e-06 2023-11-05T08:33:48.894 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.894 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.894 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:48.895 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.399216+0000", 2023-11-05T08:33:48.895 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00058774000000000001 2023-11-05T08:33:48.895 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.895 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.895 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:48.896 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.399255+0000", 2023-11-05T08:33:48.896 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8902000000000001e-05 2023-11-05T08:33:48.896 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.896 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.896 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:48.896 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.399280+0000", 2023-11-05T08:33:48.897 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4827000000000001e-05 2023-11-05T08:33:48.897 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.897 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.897 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:48.897 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.399530+0000", 2023-11-05T08:33:48.898 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00024994500000000001 2023-11-05T08:33:48.898 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.898 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.898 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:48.898 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.399558+0000", 2023-11-05T08:33:48.899 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8370000000000001e-05 2023-11-05T08:33:48.899 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.899 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.899 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:48.899 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.415065+0000", 2023-11-05T08:33:48.900 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.015506565 2023-11-05T08:33:48.900 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.900 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.900 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:48.900 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.415174+0000", 2023-11-05T08:33:48.901 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000109441 2023-11-05T08:33:48.901 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.901 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:48.901 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.901 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.902 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.902 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.bs2 66/37 ECSubWrite(tid=109, reqid=client.4691.0:497, at_version=66'24, trim_to=0'0, roll_forward_to=65'21))", 2023-11-05T08:33:48.902 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.403242+0000", 2023-11-05T08:33:48.902 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 29.330015024000001, 2023-11-05T08:33:48.902 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.018077743, 2023-11-05T08:33:48.903 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:48.903 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:48.903 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:48.903 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.903 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:48.904 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.403242+0000", 2023-11-05T08:33:48.904 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.904 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.904 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.904 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:48.904 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.403242+0000", 2023-11-05T08:33:48.905 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.905 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.905 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.905 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:48.905 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.403245+0000", 2023-11-05T08:33:48.906 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6000000000000001e-06 2023-11-05T08:33:48.906 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.906 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.906 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:48.906 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.403863+0000", 2023-11-05T08:33:48.907 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000617955 2023-11-05T08:33:48.907 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.907 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.907 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:48.907 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.403904+0000", 2023-11-05T08:33:48.908 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1542999999999998e-05 2023-11-05T08:33:48.908 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.908 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.908 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:48.908 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.403931+0000", 2023-11-05T08:33:48.909 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6016e-05 2023-11-05T08:33:48.909 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.909 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.909 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:48.909 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.404129+0000", 2023-11-05T08:33:48.909 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000198397 2023-11-05T08:33:48.910 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.910 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.910 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:48.910 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.404161+0000", 2023-11-05T08:33:48.910 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2039000000000003e-05 2023-11-05T08:33:48.911 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.911 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.911 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:48.911 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.421200+0000", 2023-11-05T08:33:48.911 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017038604999999998 2023-11-05T08:33:48.911 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.912 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.912 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:48.912 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.421320+0000", 2023-11-05T08:33:48.912 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000120588 2023-11-05T08:33:48.912 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.913 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:48.913 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.913 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.913 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.913 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.bs2 66/37 ECSubWrite(tid=110, reqid=client.4691.0:498, at_version=66'25, trim_to=0'0, roll_forward_to=65'21))", 2023-11-05T08:33:48.914 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.408787+0000", 2023-11-05T08:33:48.914 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 29.324470015999999, 2023-11-05T08:33:48.914 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012658930000000001, 2023-11-05T08:33:48.914 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:48.914 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:48.915 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:48.915 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.915 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:48.915 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.408787+0000", 2023-11-05T08:33:48.915 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.915 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.916 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.916 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:48.916 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.408787+0000", 2023-11-05T08:33:48.916 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.916 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.917 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.917 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:48.917 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.408789+0000", 2023-11-05T08:33:48.917 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.0169999999999999e-06 2023-11-05T08:33:48.917 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.918 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.918 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:48.918 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.408927+0000", 2023-11-05T08:33:48.918 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000137811 2023-11-05T08:33:48.918 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.918 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.919 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:48.919 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.408961+0000", 2023-11-05T08:33:48.919 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4173e-05 2023-11-05T08:33:48.919 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.919 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.920 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:48.920 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.408985+0000", 2023-11-05T08:33:48.920 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3961999999999999e-05 2023-11-05T08:33:48.920 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.920 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.921 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:48.921 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.410719+0000", 2023-11-05T08:33:48.921 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0017334729999999999 2023-11-05T08:33:48.921 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.921 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.921 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:48.922 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.410744+0000", 2023-11-05T08:33:48.922 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4956999999999998e-05 2023-11-05T08:33:48.922 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.922 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.922 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:48.923 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.421379+0000", 2023-11-05T08:33:48.923 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010635627999999999 2023-11-05T08:33:48.923 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.923 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.923 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:48.924 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.421446+0000", 2023-11-05T08:33:48.924 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.6909000000000006e-05 2023-11-05T08:33:48.924 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.924 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:48.924 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.925 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.925 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.925 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.bs2 66/37 ECSubWrite(tid=111, reqid=client.4691.0:499, at_version=66'26, trim_to=0'0, roll_forward_to=65'21))", 2023-11-05T08:33:48.925 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.409025+0000", 2023-11-05T08:33:48.925 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 29.324232855999998, 2023-11-05T08:33:48.926 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012479762, 2023-11-05T08:33:48.926 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:48.926 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:48.926 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:48.926 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.926 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:48.927 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.409025+0000", 2023-11-05T08:33:48.927 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.927 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.927 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.927 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:48.928 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.409025+0000", 2023-11-05T08:33:48.928 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.928 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.928 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.928 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:48.929 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.409026+0000", 2023-11-05T08:33:48.929 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1599999999999999e-06 2023-11-05T08:33:48.929 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.929 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.929 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:48.929 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.409067+0000", 2023-11-05T08:33:48.930 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1189000000000002e-05 2023-11-05T08:33:48.930 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.930 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.930 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:48.930 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.409087+0000", 2023-11-05T08:33:48.931 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9905999999999999e-05 2023-11-05T08:33:48.931 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.931 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.931 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:48.931 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.409098+0000", 2023-11-05T08:33:48.932 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1643e-05 2023-11-05T08:33:48.932 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.932 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.932 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:48.932 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.412893+0000", 2023-11-05T08:33:48.932 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0037942829999999999 2023-11-05T08:33:48.933 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.933 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.933 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:48.933 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.412919+0000", 2023-11-05T08:33:48.933 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6083000000000001e-05 2023-11-05T08:33:48.934 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.934 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.934 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:48.934 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.421452+0000", 2023-11-05T08:33:48.934 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0085328890000000001 2023-11-05T08:33:48.935 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.935 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.935 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:48.935 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.421504+0000", 2023-11-05T08:33:48.935 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2608999999999997e-05 2023-11-05T08:33:48.936 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.936 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:48.936 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.936 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.936 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.937 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es2 66/30 ECSubWrite(tid=87, reqid=osd.2.0:87, at_version=66'65, trim_to=0'0, roll_forward_to=65'63))", 2023-11-05T08:33:48.937 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.411058+0000", 2023-11-05T08:33:48.937 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 29.322198928999999, 2023-11-05T08:33:48.937 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010528946000000001, 2023-11-05T08:33:48.937 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:48.937 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:48.938 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:48.938 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.938 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:48.938 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.411058+0000", 2023-11-05T08:33:48.938 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.939 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.939 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.939 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:48.939 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.411058+0000", 2023-11-05T08:33:48.939 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.940 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.940 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.940 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:48.940 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.411059+0000", 2023-11-05T08:33:48.940 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5300000000000004e-07 2023-11-05T08:33:48.941 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.941 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.941 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:48.941 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.411112+0000", 2023-11-05T08:33:48.941 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2676000000000002e-05 2023-11-05T08:33:48.941 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.942 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.942 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:48.942 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.411142+0000", 2023-11-05T08:33:48.942 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9796000000000001e-05 2023-11-05T08:33:48.942 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.943 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.943 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:48.943 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.411164+0000", 2023-11-05T08:33:48.943 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.2249999999999999e-05 2023-11-05T08:33:48.943 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.944 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.944 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:48.944 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.411322+0000", 2023-11-05T08:33:48.944 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00015797700000000001 2023-11-05T08:33:48.944 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.944 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.945 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:48.945 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.411353+0000", 2023-11-05T08:33:48.945 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0682999999999997e-05 2023-11-05T08:33:48.945 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.945 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.946 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:48.946 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.421482+0000", 2023-11-05T08:33:48.946 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.01012939 2023-11-05T08:33:48.946 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.946 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.947 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:48.947 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.421587+0000", 2023-11-05T08:33:48.947 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010552099999999999 2023-11-05T08:33:48.947 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.947 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:48.948 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.948 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.948 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.948 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es2 66/30 ECSubWrite(tid=88, reqid=osd.2.0:88, at_version=66'67, trim_to=0'0, roll_forward_to=65'63))", 2023-11-05T08:33:48.948 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.414905+0000", 2023-11-05T08:33:48.949 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 29.318352150999999, 2023-11-05T08:33:48.949 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0083023899999999998, 2023-11-05T08:33:48.949 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:48.949 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:48.949 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:48.949 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.950 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:48.950 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.414905+0000", 2023-11-05T08:33:48.950 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.950 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.950 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.951 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:48.951 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.414905+0000", 2023-11-05T08:33:48.951 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.951 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.951 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.952 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:48.952 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.414907+0000", 2023-11-05T08:33:48.952 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.81e-06 2023-11-05T08:33:48.952 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.952 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.953 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:48.953 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.414960+0000", 2023-11-05T08:33:48.953 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3386e-05 2023-11-05T08:33:48.953 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.953 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.953 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:48.954 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.414992+0000", 2023-11-05T08:33:48.954 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1115999999999998e-05 2023-11-05T08:33:48.954 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.954 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.954 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:48.955 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.415012+0000", 2023-11-05T08:33:48.955 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.0843e-05 2023-11-05T08:33:48.955 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.955 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.955 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:48.955 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.415228+0000", 2023-11-05T08:33:48.956 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00021531600000000001 2023-11-05T08:33:48.956 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.956 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.956 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:48.956 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.415259+0000", 2023-11-05T08:33:48.957 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1399000000000002e-05 2023-11-05T08:33:48.957 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.957 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.957 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:48.957 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.423080+0000", 2023-11-05T08:33:48.958 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0078204220000000005 2023-11-05T08:33:48.958 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.958 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.958 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:48.958 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.423208+0000", 2023-11-05T08:33:48.958 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012809799999999999 2023-11-05T08:33:48.959 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.959 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:48.959 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.959 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.959 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.960 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.19s6 86/45 ECSubWrite(tid=51, reqid=client.4691.0:635, at_version=86'33, trim_to=0'0, roll_forward_to=65'31))", 2023-11-05T08:33:48.960 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:39.619629+0000", 2023-11-05T08:33:48.960 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 9.1136284459999999, 2023-11-05T08:33:48.960 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0081971470000000001, 2023-11-05T08:33:48.960 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:48.961 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:48.961 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:48.961 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.961 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:48.961 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.619629+0000", 2023-11-05T08:33:48.961 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.962 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.962 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.962 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:48.962 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.619629+0000", 2023-11-05T08:33:48.962 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:48.963 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.963 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.963 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:48.963 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.619629+0000", 2023-11-05T08:33:48.963 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4299999999999998e-07 2023-11-05T08:33:48.964 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.964 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.964 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:48.964 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.619925+0000", 2023-11-05T08:33:48.964 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00029548399999999999 2023-11-05T08:33:48.964 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.965 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.965 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:48.965 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.619966+0000", 2023-11-05T08:33:48.965 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1142e-05 2023-11-05T08:33:48.965 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.966 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.966 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:48.966 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.619998+0000", 2023-11-05T08:33:48.966 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1576999999999999e-05 2023-11-05T08:33:48.966 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.967 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.967 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:48.967 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.620107+0000", 2023-11-05T08:33:48.967 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000109158 2023-11-05T08:33:48.967 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.968 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.968 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:48.968 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.620122+0000", 2023-11-05T08:33:48.968 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5594999999999999e-05 2023-11-05T08:33:48.968 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.968 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.969 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:48.969 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.627726+0000", 2023-11-05T08:33:48.969 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.007603233 2023-11-05T08:33:48.969 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:48.969 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:48.970 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:48.970 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.627826+0000", 2023-11-05T08:33:48.970 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000100515 2023-11-05T08:33:48.970 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.970 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:48.971 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.971 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:48.971 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:48.971 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:33:48.972 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8: {} 2023-11-05T08:33:48.973 INFO:teuthology.orchestra.run.smithi002.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-11-05T08:33:48.973 INFO:tasks.rados.rados.0.smithi090.stdout:117: expect (ObjNum 0 snap 0 seq_num 0) 2023-11-05T08:33:48.974 INFO:tasks.rados.rados.0.smithi090.stdout:298: expect (ObjNum 91 snap 16 seq_num 91) 2023-11-05T08:33:48.974 INFO:tasks.rados.rados.0.smithi090.stdout:295: expect (ObjNum 65 snap 5 seq_num 65) 2023-11-05T08:33:48.975 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9: {} 2023-11-05T08:33:48.975 INFO:teuthology.orchestra.run.smithi002.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-11-05T08:33:48.976 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-11-05T08:33:48.976 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:48.968+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:48.977 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10: {} 2023-11-05T08:33:48.977 INFO:teuthology.orchestra.run.smithi002.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-11-05T08:33:48.977 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11: {} 2023-11-05T08:33:48.978 INFO:teuthology.orchestra.run.smithi002.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-11-05T08:33:48.978 INFO:teuthology.orchestra.run.smithi002.stdout:osd.12: {} 2023-11-05T08:33:48.979 INFO:teuthology.orchestra.run.smithi002.stderr:osd.12: osd_enable_op_tracker = 'true' 2023-11-05T08:33:48.979 INFO:teuthology.orchestra.run.smithi002.stdout:osd.13: {} 2023-11-05T08:33:48.979 INFO:teuthology.orchestra.run.smithi002.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-11-05T08:33:48.980 INFO:teuthology.orchestra.run.smithi002.stdout:osd.14: {} 2023-11-05T08:33:48.980 INFO:teuthology.orchestra.run.smithi002.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-11-05T08:33:48.981 INFO:teuthology.orchestra.run.smithi002.stdout:osd.15: {} 2023-11-05T08:33:48.981 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-05T08:33:48.982 INFO:teuthology.orchestra.run.smithi002.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-11-05T08:33:48.982 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-05T08:33:49.073 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:33:49.068+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:33:49.151 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:33:49.151 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2023-11-05T08:33:49.151 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 0 2023-11-05T08:33:49.151 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:33:49.165 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-11-05T08:33:49.175 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:33:49.170+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:33:49.275 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:33:49.274+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:33:49.333 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:33:49.333 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2023-11-05T08:33:49.333 INFO:teuthology.orchestra.run.smithi053.stdout: "complaint_time": 30, 2023-11-05T08:33:49.334 INFO:teuthology.orchestra.run.smithi053.stdout: "num_blocked_ops": 0 2023-11-05T08:33:49.334 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:33:49.349 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-11-05T08:33:49.377 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:33:49.373+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:33:49.472 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:33:49.473 INFO:teuthology.orchestra.run.smithi053.stdout: "size": 20, 2023-11-05T08:33:49.473 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 600, 2023-11-05T08:33:49.473 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2023-11-05T08:33:49.473 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.474 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:271 3.18s0 3:1aa38538:::smithi09039252-11:5 [read 0~1097728,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:33:49.474 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:53.109072+0000", 2023-11-05T08:33:49.474 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 56.360122234000002, 2023-11-05T08:33:49.474 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.069197392, 2023-11-05T08:33:49.474 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:49.475 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2023-11-05T08:33:49.475 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:49.475 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:49.475 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:49.475 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 271 2023-11-05T08:33:49.476 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.476 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:49.476 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.476 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:49.476 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109072+0000", 2023-11-05T08:33:49.477 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.477 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.477 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.477 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:49.477 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109072+0000", 2023-11-05T08:33:49.478 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.478 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.478 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.478 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:49.479 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109076+0000", 2023-11-05T08:33:49.479 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.9639999999999997e-06 2023-11-05T08:33:49.479 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.479 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.479 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:49.480 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109094+0000", 2023-11-05T08:33:49.480 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7995e-05 2023-11-05T08:33:49.480 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.480 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.480 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:49.481 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109134+0000", 2023-11-05T08:33:49.481 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.0729000000000003e-05 2023-11-05T08:33:49.481 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.481 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.481 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:49.482 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109160+0000", 2023-11-05T08:33:49.482 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5276e-05 2023-11-05T08:33:49.482 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.482 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.482 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:49.483 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.134337+0000", 2023-11-05T08:33:49.483 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.025177609 2023-11-05T08:33:49.483 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.483 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.483 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2023-11-05T08:33:49.484 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.134667+0000", 2023-11-05T08:33:49.484 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00032950600000000003 2023-11-05T08:33:49.484 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.484 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.484 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:49.485 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.166070+0000", 2023-11-05T08:33:49.485 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0314031800000001 2023-11-05T08:33:49.485 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.485 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.485 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:49.486 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.166695+0000", 2023-11-05T08:33:49.486 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000625137 2023-11-05T08:33:49.486 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.486 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.486 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:49.487 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.178269+0000", 2023-11-05T08:33:49.487 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011573996 2023-11-05T08:33:49.487 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.487 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:49.487 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.488 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.488 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.488 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:272 3.18s0 3:1aa38538:::smithi09039252-11:5 [read 0~1097728,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:33:49.488 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:53.109221+0000", 2023-11-05T08:33:49.488 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 56.359972622000001, 2023-11-05T08:33:49.489 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0781554369999999, 2023-11-05T08:33:49.489 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:49.489 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2023-11-05T08:33:49.489 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:49.489 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:49.490 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:49.490 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 272 2023-11-05T08:33:49.490 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.490 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:49.490 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.491 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:49.491 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109221+0000", 2023-11-05T08:33:49.491 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.491 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.491 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.492 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:49.492 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109221+0000", 2023-11-05T08:33:49.492 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.492 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.492 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.493 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:49.493 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109223+0000", 2023-11-05T08:33:49.493 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4330000000000001e-06 2023-11-05T08:33:49.493 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.493 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.493 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:49.494 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109232+0000", 2023-11-05T08:33:49.494 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.234e-06 2023-11-05T08:33:49.494 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.494 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.494 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:49.495 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109254+0000", 2023-11-05T08:33:49.495 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1576000000000001e-05 2023-11-05T08:33:49.495 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.495 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.495 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:49.496 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109270+0000", 2023-11-05T08:33:49.496 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6396e-05 2023-11-05T08:33:49.496 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.496 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.496 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:49.496 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.134760+0000", 2023-11-05T08:33:49.497 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.025490292000000001 2023-11-05T08:33:49.497 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.497 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.497 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2023-11-05T08:33:49.497 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.134764+0000", 2023-11-05T08:33:49.498 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5470000000000001e-06 2023-11-05T08:33:49.498 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.498 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.498 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:49.498 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.178397+0000", 2023-11-05T08:33:49.499 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.04363341 2023-11-05T08:33:49.499 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.499 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.499 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:49.499 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.178606+0000", 2023-11-05T08:33:49.500 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00020828200000000001 2023-11-05T08:33:49.500 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.500 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.500 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:49.500 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.187377+0000", 2023-11-05T08:33:49.501 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0087712669999999993 2023-11-05T08:33:49.501 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.501 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:49.501 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.501 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.502 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.502 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:273 3.18s0 3:1aa38538:::smithi09039252-11:5 [read 0~1097728,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:33:49.502 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:53.109291+0000", 2023-11-05T08:33:49.502 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 56.359902851000001, 2023-11-05T08:33:49.502 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.08037483, 2023-11-05T08:33:49.503 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:49.503 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2023-11-05T08:33:49.503 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:49.503 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:49.503 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:49.503 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 273 2023-11-05T08:33:49.504 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.504 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:49.504 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.504 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:49.504 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109291+0000", 2023-11-05T08:33:49.505 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.505 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.505 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.505 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:49.505 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109291+0000", 2023-11-05T08:33:49.506 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.506 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.506 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.506 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:49.506 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109292+0000", 2023-11-05T08:33:49.507 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.054e-06 2023-11-05T08:33:49.507 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.507 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.507 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:49.507 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109299+0000", 2023-11-05T08:33:49.507 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.7700000000000004e-06 2023-11-05T08:33:49.508 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.508 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.508 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:49.508 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109317+0000", 2023-11-05T08:33:49.508 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.8529000000000002e-05 2023-11-05T08:33:49.509 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.509 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.509 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:49.509 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.109332+0000", 2023-11-05T08:33:49.509 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4905999999999999e-05 2023-11-05T08:33:49.510 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.510 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.510 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:49.510 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.134830+0000", 2023-11-05T08:33:49.510 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.025497912000000001 2023-11-05T08:33:49.511 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.511 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.511 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2023-11-05T08:33:49.511 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:53.134832+0000", 2023-11-05T08:33:49.511 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1399999999999998e-06 2023-11-05T08:33:49.512 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.512 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.512 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:49.512 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.179990+0000", 2023-11-05T08:33:49.512 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0451579209999999 2023-11-05T08:33:49.513 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.513 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.513 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:49.513 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.180250+0000", 2023-11-05T08:33:49.513 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000259595 2023-11-05T08:33:49.514 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.514 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.514 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:49.514 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.189666+0000", 2023-11-05T08:33:49.514 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0094160029999999992 2023-11-05T08:33:49.515 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.515 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:49.515 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.515 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.515 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.516 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.11s3 41/40 ECSubWrite(tid=65, reqid=client.4691.0:287, at_version=41'13, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:49.516 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.173108+0000", 2023-11-05T08:33:49.516 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 55.296085560999998, 2023-11-05T08:33:49.516 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012254233999999999, 2023-11-05T08:33:49.516 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:49.517 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:49.517 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:49.517 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.517 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:49.517 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.173108+0000", 2023-11-05T08:33:49.517 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.518 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.518 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.518 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:49.518 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.173108+0000", 2023-11-05T08:33:49.518 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.519 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.519 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.519 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:49.519 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.173109+0000", 2023-11-05T08:33:49.519 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.7700000000000004e-07 2023-11-05T08:33:49.520 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.520 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.520 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:49.520 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.174570+0000", 2023-11-05T08:33:49.520 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00146053 2023-11-05T08:33:49.521 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.521 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.521 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:49.521 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.174598+0000", 2023-11-05T08:33:49.521 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8404000000000001e-05 2023-11-05T08:33:49.522 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.522 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.522 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:49.522 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.174614+0000", 2023-11-05T08:33:49.523 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6436999999999999e-05 2023-11-05T08:33:49.523 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.523 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.523 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:49.523 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.174752+0000", 2023-11-05T08:33:49.523 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013757799999999999 2023-11-05T08:33:49.524 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.524 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.524 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:49.524 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.174775+0000", 2023-11-05T08:33:49.524 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.287e-05 2023-11-05T08:33:49.525 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.525 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.525 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:49.525 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.185293+0000", 2023-11-05T08:33:49.525 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010518192000000001 2023-11-05T08:33:49.526 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.526 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.526 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:49.526 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.185363+0000", 2023-11-05T08:33:49.526 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9546000000000001e-05 2023-11-05T08:33:49.527 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.527 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:49.527 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.527 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.527 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.528 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.11s3 41/40 ECSubWrite(tid=66, reqid=client.4691.0:288, at_version=41'14, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:49.528 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.177594+0000", 2023-11-05T08:33:49.528 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 55.291600365000001, 2023-11-05T08:33:49.528 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0090366969999999998, 2023-11-05T08:33:49.528 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:49.529 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:49.529 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:49.529 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.529 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:49.529 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.177594+0000", 2023-11-05T08:33:49.529 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.530 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.530 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.530 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:49.530 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.177594+0000", 2023-11-05T08:33:49.530 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.531 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.531 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.531 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:49.531 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.177595+0000", 2023-11-05T08:33:49.531 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6599999999999994e-07 2023-11-05T08:33:49.531 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.532 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.532 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:49.532 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.180867+0000", 2023-11-05T08:33:49.532 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.003272264 2023-11-05T08:33:49.532 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.532 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.533 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:49.533 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.180893+0000", 2023-11-05T08:33:49.533 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5956e-05 2023-11-05T08:33:49.533 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.533 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.534 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:49.534 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.180915+0000", 2023-11-05T08:33:49.534 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1756999999999999e-05 2023-11-05T08:33:49.534 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.534 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.534 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:49.535 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.181066+0000", 2023-11-05T08:33:49.535 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000151213 2023-11-05T08:33:49.535 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.535 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.535 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:49.536 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.181085+0000", 2023-11-05T08:33:49.536 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9389999999999999e-05 2023-11-05T08:33:49.536 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.536 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.536 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:49.536 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.186564+0000", 2023-11-05T08:33:49.537 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0054784569999999999 2023-11-05T08:33:49.537 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.537 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.537 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:49.537 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.186630+0000", 2023-11-05T08:33:49.538 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.6693999999999995e-05 2023-11-05T08:33:49.538 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.538 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:49.538 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.538 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.539 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.539 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:317 3.17s0 3:e93e8c6a:::smithi09039252-17:head [delete] snapc b=[7,5] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e45)", 2023-11-05T08:33:49.539 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:58.238377+0000", 2023-11-05T08:33:49.539 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 51.230817160000001, 2023-11-05T08:33:49.539 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9937814299999999, 2023-11-05T08:33:49.539 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:49.540 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:49.540 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:49.540 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:49.540 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:49.540 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 317 2023-11-05T08:33:49.541 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.541 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:49.541 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.541 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:49.541 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:58.238377+0000", 2023-11-05T08:33:49.541 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.542 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.542 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.542 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:49.542 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:58.238377+0000", 2023-11-05T08:33:49.542 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.543 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.543 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.543 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:49.543 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:58.238379+0000", 2023-11-05T08:33:49.543 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9460000000000001e-06 2023-11-05T08:33:49.544 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.544 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.544 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:49.544 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:58.238390+0000", 2023-11-05T08:33:49.544 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1065e-05 2023-11-05T08:33:49.544 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.545 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.545 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:49.545 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:58.238412+0000", 2023-11-05T08:33:49.545 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.2529999999999999e-05 2023-11-05T08:33:49.545 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.546 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.546 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:49.546 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:58.238425+0000", 2023-11-05T08:33:49.546 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3077e-05 2023-11-05T08:33:49.546 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.547 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.547 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:49.547 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:58.238536+0000", 2023-11-05T08:33:49.547 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00011045899999999999 2023-11-05T08:33:49.547 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.548 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.548 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2023-11-05T08:33:49.548 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:58.238541+0000", 2023-11-05T08:33:49.548 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9710000000000003e-06 2023-11-05T08:33:49.548 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.549 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.549 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:49.549 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:59.210323+0000", 2023-11-05T08:33:49.549 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.97178187500000002 2023-11-05T08:33:49.549 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.550 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.550 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2023-11-05T08:33:49.550 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:59.210329+0000", 2023-11-05T08:33:49.550 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5100000000000004e-06 2023-11-05T08:33:49.550 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.551 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.551 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:49.551 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:00.215041+0000", 2023-11-05T08:33:49.551 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.004711436 2023-11-05T08:33:49.551 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.551 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.552 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for missing object", 2023-11-05T08:33:49.552 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:00.216219+0000", 2023-11-05T08:33:49.552 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0011779609999999999 2023-11-05T08:33:49.552 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.552 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.553 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:49.553 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:00.226127+0000", 2023-11-05T08:33:49.553 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0099082130000000008 2023-11-05T08:33:49.553 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.553 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.554 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:49.554 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:00.226269+0000", 2023-11-05T08:33:49.554 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000142032 2023-11-05T08:33:49.554 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.554 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.555 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:49.555 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:00.227010+0000", 2023-11-05T08:33:49.555 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00074093 2023-11-05T08:33:49.555 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.555 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.556 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:49.556 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:00.231106+0000", 2023-11-05T08:33:49.556 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0040965589999999996 2023-11-05T08:33:49.556 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.556 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.557 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:49.557 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:00.231612+0000", 2023-11-05T08:33:49.557 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00050578000000000005 2023-11-05T08:33:49.557 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.557 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.558 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:49.558 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:00.232158+0000", 2023-11-05T08:33:49.558 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00054608599999999999 2023-11-05T08:33:49.558 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.558 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:49.558 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.559 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.559 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.559 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.16s5 51/45 ECSubWrite(tid=58, reqid=osd.11.0:58, at_version=51'19, trim_to=0'0, roll_forward_to=49'17))", 2023-11-05T08:33:49.559 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:04.226078+0000", 2023-11-05T08:33:49.559 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 45.243116280999999, 2023-11-05T08:33:49.560 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.019914247999999999, 2023-11-05T08:33:49.560 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:49.560 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:49.560 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:49.560 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.561 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:49.561 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.226078+0000", 2023-11-05T08:33:49.561 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.561 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.561 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.561 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:49.562 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.226078+0000", 2023-11-05T08:33:49.562 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.562 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.562 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.562 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:49.563 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.226079+0000", 2023-11-05T08:33:49.563 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.133e-06 2023-11-05T08:33:49.563 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.563 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.563 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:49.563 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.226140+0000", 2023-11-05T08:33:49.564 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0809e-05 2023-11-05T08:33:49.564 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.564 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.564 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:49.564 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.226171+0000", 2023-11-05T08:33:49.565 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0889000000000002e-05 2023-11-05T08:33:49.565 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.565 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.565 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:49.565 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.226196+0000", 2023-11-05T08:33:49.565 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5945999999999999e-05 2023-11-05T08:33:49.566 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.566 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.566 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:49.566 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.226413+0000", 2023-11-05T08:33:49.566 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000216379 2023-11-05T08:33:49.567 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.567 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.567 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:49.567 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.226442+0000", 2023-11-05T08:33:49.567 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9323e-05 2023-11-05T08:33:49.568 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.568 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.568 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:49.568 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.245889+0000", 2023-11-05T08:33:49.568 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.019446938 2023-11-05T08:33:49.569 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.569 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.569 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:49.569 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.245992+0000", 2023-11-05T08:33:49.569 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010283100000000001 2023-11-05T08:33:49.570 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.570 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:49.570 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.570 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.570 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.571 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:338 3.17s0 3:e93e8c6a:::smithi09039252-17:head [read 0~0,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:49.571 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:05.250110+0000", 2023-11-05T08:33:49.572 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 44.219083920999999, 2023-11-05T08:33:49.573 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0080018280000000008, 2023-11-05T08:33:49.573 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:49.573 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:49.573 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:49.573 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:49.574 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:49.574 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 338 2023-11-05T08:33:49.574 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.574 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:49.575 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.575 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:49.575 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250110+0000", 2023-11-05T08:33:49.575 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.575 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.575 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.576 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:49.576 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250110+0000", 2023-11-05T08:33:49.576 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.576 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.576 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.577 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:49.577 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250113+0000", 2023-11-05T08:33:49.577 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3230000000000002e-06 2023-11-05T08:33:49.577 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.577 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.578 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:49.578 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250128+0000", 2023-11-05T08:33:49.578 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4749999999999999e-05 2023-11-05T08:33:49.578 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.578 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.579 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:49.579 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250163+0000", 2023-11-05T08:33:49.579 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5079000000000001e-05 2023-11-05T08:33:49.579 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.579 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.580 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:49.580 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250186+0000", 2023-11-05T08:33:49.580 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.2566e-05 2023-11-05T08:33:49.580 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.580 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.581 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:49.581 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250320+0000", 2023-11-05T08:33:49.581 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000134544 2023-11-05T08:33:49.581 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.581 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.582 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op must wait for map", 2023-11-05T08:33:49.582 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250338+0000", 2023-11-05T08:33:49.582 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7813e-05 2023-11-05T08:33:49.582 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.582 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.583 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:49.583 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.257738+0000", 2023-11-05T08:33:49.583 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0074003050000000003 2023-11-05T08:33:49.583 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.583 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.584 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:49.584 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258112+0000", 2023-11-05T08:33:49.584 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00037344800000000002 2023-11-05T08:33:49.584 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.584 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:49.585 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.585 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.585 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.585 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:339 3.17s0 3:e93e8c6a:::smithi09039252-17:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:49.585 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:05.250225+0000", 2023-11-05T08:33:49.585 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 44.218968973999999, 2023-11-05T08:33:49.586 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0084669749999999998, 2023-11-05T08:33:49.586 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:49.586 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:49.586 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:49.586 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:49.587 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:49.587 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 339 2023-11-05T08:33:49.587 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.587 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:49.587 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.588 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:49.588 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250225+0000", 2023-11-05T08:33:49.588 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.588 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.588 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.589 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:49.589 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250225+0000", 2023-11-05T08:33:49.589 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.589 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.589 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.590 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:49.590 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250226+0000", 2023-11-05T08:33:49.590 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.8400000000000003e-07 2023-11-05T08:33:49.590 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.590 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.591 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:49.591 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250232+0000", 2023-11-05T08:33:49.591 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.2099999999999998e-06 2023-11-05T08:33:49.591 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.591 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.592 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:49.592 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250248+0000", 2023-11-05T08:33:49.592 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5639000000000001e-05 2023-11-05T08:33:49.592 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.592 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.593 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:49.593 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250257+0000", 2023-11-05T08:33:49.593 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.8459999999999997e-06 2023-11-05T08:33:49.593 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.593 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.593 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:49.594 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250524+0000", 2023-11-05T08:33:49.594 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00026785099999999999 2023-11-05T08:33:49.594 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.594 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.594 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:33:49.595 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250553+0000", 2023-11-05T08:33:49.595 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8649999999999998e-05 2023-11-05T08:33:49.595 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.595 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.595 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:49.596 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258255+0000", 2023-11-05T08:33:49.596 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0077024119999999996 2023-11-05T08:33:49.596 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.596 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.596 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:49.597 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258692+0000", 2023-11-05T08:33:49.597 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00043648299999999999 2023-11-05T08:33:49.597 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.597 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:49.597 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.598 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.598 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.598 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:340 3.17s0 3:e93e8c6a:::smithi09039252-17:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:49.598 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:05.250298+0000", 2023-11-05T08:33:49.598 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 44.218896235999999, 2023-11-05T08:33:49.599 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0088381339999999992, 2023-11-05T08:33:49.599 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:49.599 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:49.599 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:49.599 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:49.600 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:49.600 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 340 2023-11-05T08:33:49.600 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.600 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:49.600 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.601 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:49.601 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250298+0000", 2023-11-05T08:33:49.601 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.601 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.601 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.601 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:49.602 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250298+0000", 2023-11-05T08:33:49.602 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.602 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.602 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.602 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:49.603 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250298+0000", 2023-11-05T08:33:49.603 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.3e-07 2023-11-05T08:33:49.603 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.603 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.604 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:49.604 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250306+0000", 2023-11-05T08:33:49.604 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.0400000000000004e-06 2023-11-05T08:33:49.604 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.604 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.605 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:49.605 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250317+0000", 2023-11-05T08:33:49.605 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1914e-05 2023-11-05T08:33:49.605 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.605 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.606 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:49.606 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250325+0000", 2023-11-05T08:33:49.606 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.7200000000000006e-06 2023-11-05T08:33:49.606 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.606 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.607 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:49.607 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250639+0000", 2023-11-05T08:33:49.607 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00031349200000000001 2023-11-05T08:33:49.607 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.607 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.607 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:33:49.608 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.250664+0000", 2023-11-05T08:33:49.608 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5786e-05 2023-11-05T08:33:49.608 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.608 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.608 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:49.609 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258785+0000", 2023-11-05T08:33:49.609 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0081202930000000006 2023-11-05T08:33:49.609 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.609 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.609 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:49.610 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.259136+0000", 2023-11-05T08:33:49.610 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00035115899999999998 2023-11-05T08:33:49.610 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.610 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:49.610 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.610 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.611 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.611 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:342 3.8s0 3:11e42f5e:::smithi09039252-35:head [copy-from ver 10] snapc 10=[7] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:49.611 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:05.254724+0000", 2023-11-05T08:33:49.611 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 44.214469499000003, 2023-11-05T08:33:49.611 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.043497013000000001, 2023-11-05T08:33:49.612 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:49.612 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:49.612 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:49.612 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:49.612 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:49.613 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 342 2023-11-05T08:33:49.613 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.613 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:49.613 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.613 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:49.613 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.254724+0000", 2023-11-05T08:33:49.614 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.614 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.614 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.614 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:49.614 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.254724+0000", 2023-11-05T08:33:49.615 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.615 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.615 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.615 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:49.615 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.254727+0000", 2023-11-05T08:33:49.616 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.2570000000000002e-06 2023-11-05T08:33:49.616 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.616 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.616 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:49.616 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.254741+0000", 2023-11-05T08:33:49.616 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4746000000000001e-05 2023-11-05T08:33:49.617 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.617 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.617 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:49.617 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.254771+0000", 2023-11-05T08:33:49.617 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0015999999999999e-05 2023-11-05T08:33:49.618 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.618 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.618 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:49.618 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.254796+0000", 2023-11-05T08:33:49.618 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4536e-05 2023-11-05T08:33:49.618 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.619 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.619 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:49.619 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.256414+0000", 2023-11-05T08:33:49.619 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0016180369999999999 2023-11-05T08:33:49.619 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.620 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.620 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:49.620 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.256671+0000", 2023-11-05T08:33:49.620 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00025643699999999998 2023-11-05T08:33:49.620 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.620 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.621 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:49.621 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.291461+0000", 2023-11-05T08:33:49.621 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.034790465999999999 2023-11-05T08:33:49.621 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.621 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.622 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:49.622 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.297059+0000", 2023-11-05T08:33:49.622 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0055981729999999997 2023-11-05T08:33:49.622 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.622 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.623 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:49.623 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.297172+0000", 2023-11-05T08:33:49.623 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00011328900000000001 2023-11-05T08:33:49.623 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.623 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.623 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:49.624 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.298221+0000", 2023-11-05T08:33:49.624 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001049056 2023-11-05T08:33:49.624 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.624 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:49.624 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.625 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.625 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.625 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:343 3.8s0 3:11e42f5e:::smithi09039252-35:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:49.625 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:05.254831+0000", 2023-11-05T08:33:49.625 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 44.214362577999999, 2023-11-05T08:33:49.626 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.041641958, 2023-11-05T08:33:49.626 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:49.626 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2023-11-05T08:33:49.626 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:49.626 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:49.626 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:49.627 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 343 2023-11-05T08:33:49.627 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.627 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:49.627 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.627 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:49.628 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.254831+0000", 2023-11-05T08:33:49.628 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.628 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.628 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.628 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:49.629 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.254831+0000", 2023-11-05T08:33:49.629 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.629 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.629 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.629 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:49.630 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.254833+0000", 2023-11-05T08:33:49.630 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9069999999999999e-06 2023-11-05T08:33:49.630 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.630 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.630 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:49.631 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.254841+0000", 2023-11-05T08:33:49.631 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.5959999999999998e-06 2023-11-05T08:33:49.631 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.631 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.631 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:49.631 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.254864+0000", 2023-11-05T08:33:49.632 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3499999999999999e-05 2023-11-05T08:33:49.632 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.632 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.632 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:49.632 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.254875+0000", 2023-11-05T08:33:49.633 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1032999999999999e-05 2023-11-05T08:33:49.633 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.633 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.633 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:49.633 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.257178+0000", 2023-11-05T08:33:49.634 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002302784 2023-11-05T08:33:49.634 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.634 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.634 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for blocked object", 2023-11-05T08:33:49.634 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.257359+0000", 2023-11-05T08:33:49.635 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00018044600000000001 2023-11-05T08:33:49.635 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.635 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.635 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:49.635 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.295697+0000", 2023-11-05T08:33:49.635 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.038338192 2023-11-05T08:33:49.636 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.636 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.636 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:49.636 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.295953+0000", 2023-11-05T08:33:49.636 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00025650899999999998 2023-11-05T08:33:49.637 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.637 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.637 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:49.637 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.296473+0000", 2023-11-05T08:33:49.637 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00051999100000000003 2023-11-05T08:33:49.637 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.638 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:49.638 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.638 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.638 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.638 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.18s4 52/45 ECSubWrite(tid=205, reqid=client.4691.0:351, at_version=52'24, trim_to=0'0, roll_forward_to=51'23))", 2023-11-05T08:33:49.639 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:05.281620+0000", 2023-11-05T08:33:49.639 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 44.187574237, 2023-11-05T08:33:49.639 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0073326679999999997, 2023-11-05T08:33:49.639 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:49.639 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:49.640 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:49.640 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.640 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:49.640 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.281620+0000", 2023-11-05T08:33:49.640 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.641 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.641 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.641 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:49.641 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.281620+0000", 2023-11-05T08:33:49.641 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.642 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.642 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.642 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:49.642 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.281621+0000", 2023-11-05T08:33:49.642 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.7400000000000002e-07 2023-11-05T08:33:49.643 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.643 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.643 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:49.643 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.282585+0000", 2023-11-05T08:33:49.643 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00096495399999999996 2023-11-05T08:33:49.644 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.644 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.644 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:49.644 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.282639+0000", 2023-11-05T08:33:49.644 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3282000000000002e-05 2023-11-05T08:33:49.645 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.645 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.645 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:49.645 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.282670+0000", 2023-11-05T08:33:49.645 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0825999999999999e-05 2023-11-05T08:33:49.646 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.646 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.646 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:49.646 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.282933+0000", 2023-11-05T08:33:49.646 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000263758 2023-11-05T08:33:49.647 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.647 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.647 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:49.647 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.282966+0000", 2023-11-05T08:33:49.647 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3121999999999999e-05 2023-11-05T08:33:49.648 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.648 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.648 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:49.648 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.288887+0000", 2023-11-05T08:33:49.648 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0059202129999999997 2023-11-05T08:33:49.648 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.649 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.649 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:49.649 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.288952+0000", 2023-11-05T08:33:49.649 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5739000000000002e-05 2023-11-05T08:33:49.649 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.650 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:49.650 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.650 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.650 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.650 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:412 3.8s0 3:11e42f5e:::smithi09039252-35:head [append 0~0] snapc 1b=[1b,1a,17,7] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2023-11-05T08:33:49.651 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:14.389862+0000", 2023-11-05T08:33:49.651 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 35.079331746999998, 2023-11-05T08:33:49.651 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010619417000000001, 2023-11-05T08:33:49.651 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:49.651 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:49.652 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:49.652 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:49.652 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:49.652 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 412 2023-11-05T08:33:49.652 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.653 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:49.653 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.653 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:49.653 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.389862+0000", 2023-11-05T08:33:49.653 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.653 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.654 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.654 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:49.654 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.389862+0000", 2023-11-05T08:33:49.654 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.654 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.655 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.655 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:49.655 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.389865+0000", 2023-11-05T08:33:49.655 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3599999999999999e-06 2023-11-05T08:33:49.655 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.656 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.656 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:49.656 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.389875+0000", 2023-11-05T08:33:49.656 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0489000000000001e-05 2023-11-05T08:33:49.656 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.656 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.657 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:49.657 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.389905+0000", 2023-11-05T08:33:49.657 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0020000000000001e-05 2023-11-05T08:33:49.657 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.657 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.658 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:49.658 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.389921+0000", 2023-11-05T08:33:49.658 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5682999999999999e-05 2023-11-05T08:33:49.658 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.658 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.659 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:49.659 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.390092+0000", 2023-11-05T08:33:49.659 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00017081600000000001 2023-11-05T08:33:49.659 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.659 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.659 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:49.660 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.390428+0000", 2023-11-05T08:33:49.660 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00033592099999999998 2023-11-05T08:33:49.660 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.660 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.660 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:49.660 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.392907+0000", 2023-11-05T08:33:49.661 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0024797439999999999 2023-11-05T08:33:49.661 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.661 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.661 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:49.661 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.399481+0000", 2023-11-05T08:33:49.662 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0065741369999999999 2023-11-05T08:33:49.662 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.662 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.662 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:49.662 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.400140+0000", 2023-11-05T08:33:49.663 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00065843399999999995 2023-11-05T08:33:49.663 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.663 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.663 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:49.663 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.400482+0000", 2023-11-05T08:33:49.663 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000341813 2023-11-05T08:33:49.664 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.664 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:49.664 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.664 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.664 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.665 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:413 3.8s0 3:11e42f5e:::smithi09039252-35:head [rollback 7] snapc 1b=[1b,1a,17,7] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2023-11-05T08:33:49.665 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:14.389956+0000", 2023-11-05T08:33:49.665 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 35.079238316000001, 2023-11-05T08:33:49.665 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011423634, 2023-11-05T08:33:49.665 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:49.665 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:49.666 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:49.666 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:49.666 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:49.666 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 413 2023-11-05T08:33:49.666 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.667 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:49.667 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.667 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:49.667 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.389956+0000", 2023-11-05T08:33:49.667 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.668 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.668 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.668 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:49.668 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.389956+0000", 2023-11-05T08:33:49.668 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.668 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.669 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.669 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:49.669 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.389957+0000", 2023-11-05T08:33:49.669 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.22e-06 2023-11-05T08:33:49.669 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.670 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.670 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:49.670 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.389964+0000", 2023-11-05T08:33:49.670 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.3000000000000004e-06 2023-11-05T08:33:49.670 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.671 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.671 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:49.671 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.390000+0000", 2023-11-05T08:33:49.671 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5982999999999998e-05 2023-11-05T08:33:49.671 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.671 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.672 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:49.672 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.390011+0000", 2023-11-05T08:33:49.672 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.099e-05 2023-11-05T08:33:49.672 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.672 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.673 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:49.673 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.394506+0000", 2023-11-05T08:33:49.673 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0044950939999999998 2023-11-05T08:33:49.673 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.673 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.674 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:49.674 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.394702+0000", 2023-11-05T08:33:49.674 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00019568 2023-11-05T08:33:49.674 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.674 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.675 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:49.675 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.395966+0000", 2023-11-05T08:33:49.675 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0012637379999999999 2023-11-05T08:33:49.675 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.675 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.675 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:49.676 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.399485+0000", 2023-11-05T08:33:49.676 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.003518864 2023-11-05T08:33:49.676 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.676 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.676 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:49.677 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.401053+0000", 2023-11-05T08:33:49.677 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0015684620000000001 2023-11-05T08:33:49.677 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.677 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.677 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:49.678 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.401379+0000", 2023-11-05T08:33:49.678 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00032630300000000001 2023-11-05T08:33:49.678 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.678 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:49.678 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.679 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.679 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.679 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:414 3.8s0 3:11e42f5e:::smithi09039252-35:head [append 0~0] snapc 1b=[1b,1a,17,7] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2023-11-05T08:33:49.679 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:14.390035+0000", 2023-11-05T08:33:49.679 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 35.079158988000003, 2023-11-05T08:33:49.679 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.01234273, 2023-11-05T08:33:49.680 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:49.680 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:49.680 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:49.680 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:49.680 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:49.681 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 414 2023-11-05T08:33:49.681 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.681 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:49.681 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.681 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:49.682 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.390035+0000", 2023-11-05T08:33:49.682 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.682 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.682 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.682 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:49.683 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.390035+0000", 2023-11-05T08:33:49.683 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.683 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.683 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.683 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:49.684 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.390036+0000", 2023-11-05T08:33:49.684 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2500000000000001e-06 2023-11-05T08:33:49.684 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.684 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.684 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:49.685 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.390043+0000", 2023-11-05T08:33:49.685 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3640000000000004e-06 2023-11-05T08:33:49.685 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.685 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.685 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:49.686 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.390063+0000", 2023-11-05T08:33:49.686 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9902999999999999e-05 2023-11-05T08:33:49.686 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.686 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.686 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:49.686 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.390081+0000", 2023-11-05T08:33:49.687 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.8369e-05 2023-11-05T08:33:49.687 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.687 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.687 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:49.687 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.396827+0000", 2023-11-05T08:33:49.688 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0067464500000000002 2023-11-05T08:33:49.688 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.688 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.688 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:49.688 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.397166+0000", 2023-11-05T08:33:49.689 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00033854000000000002 2023-11-05T08:33:49.689 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.689 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.689 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:49.689 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.398388+0000", 2023-11-05T08:33:49.689 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0012222859999999999 2023-11-05T08:33:49.690 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.690 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.690 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:49.690 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.400877+0000", 2023-11-05T08:33:49.690 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002489144 2023-11-05T08:33:49.691 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.691 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.691 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:49.691 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.401956+0000", 2023-11-05T08:33:49.691 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0010789389999999999 2023-11-05T08:33:49.691 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.692 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.692 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:49.692 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.402378+0000", 2023-11-05T08:33:49.692 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00042148499999999998 2023-11-05T08:33:49.692 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.693 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:49.693 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.693 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.693 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.693 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.16s5 66/45 ECSubWrite(tid=112, reqid=osd.11.0:112, at_version=66'34, trim_to=0'0, roll_forward_to=65'32))", 2023-11-05T08:33:49.694 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.400318+0000", 2023-11-05T08:33:49.694 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 30.068876330999998, 2023-11-05T08:33:49.694 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.015622313000000001, 2023-11-05T08:33:49.694 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:49.694 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:49.695 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:49.695 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.695 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:49.695 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.400318+0000", 2023-11-05T08:33:49.695 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.695 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.696 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.696 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:49.696 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.400318+0000", 2023-11-05T08:33:49.696 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.696 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.697 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.697 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:49.697 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.400318+0000", 2023-11-05T08:33:49.697 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8599999999999998e-07 2023-11-05T08:33:49.697 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.698 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.698 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:49.698 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.400371+0000", 2023-11-05T08:33:49.698 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2636000000000002e-05 2023-11-05T08:33:49.698 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.699 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.699 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:49.699 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.400408+0000", 2023-11-05T08:33:49.699 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.7015999999999999e-05 2023-11-05T08:33:49.699 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.699 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.700 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:49.700 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.400436+0000", 2023-11-05T08:33:49.700 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8113e-05 2023-11-05T08:33:49.700 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.700 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.701 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:49.701 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.400633+0000", 2023-11-05T08:33:49.701 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000197147 2023-11-05T08:33:49.701 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.701 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.702 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:49.702 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.400665+0000", 2023-11-05T08:33:49.702 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1498999999999998e-05 2023-11-05T08:33:49.702 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.702 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.702 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:49.703 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.415850+0000", 2023-11-05T08:33:49.703 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.015185067999999999 2023-11-05T08:33:49.703 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.703 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.703 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:49.704 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.415940+0000", 2023-11-05T08:33:49.704 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.0247999999999994e-05 2023-11-05T08:33:49.704 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.704 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:49.704 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.705 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.705 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.705 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.16s5 66/45 ECSubWrite(tid=115, reqid=osd.11.0:112, at_version=0'0, trim_to=0'0, roll_forward_to=66'34))", 2023-11-05T08:33:49.705 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.416614+0000", 2023-11-05T08:33:49.705 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 30.052579949999998, 2023-11-05T08:33:49.705 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0087502250000000004, 2023-11-05T08:33:49.706 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:49.706 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:49.706 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:49.706 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.706 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:49.707 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.416614+0000", 2023-11-05T08:33:49.707 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.707 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.707 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.707 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:49.708 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.416614+0000", 2023-11-05T08:33:49.708 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.708 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.708 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.708 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:49.709 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.416616+0000", 2023-11-05T08:33:49.709 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7e-06 2023-11-05T08:33:49.709 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.709 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.709 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:49.709 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.416644+0000", 2023-11-05T08:33:49.710 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8549000000000001e-05 2023-11-05T08:33:49.710 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.710 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.710 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:49.710 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.416678+0000", 2023-11-05T08:33:49.711 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3266000000000003e-05 2023-11-05T08:33:49.711 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.711 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.711 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:49.711 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.416701+0000", 2023-11-05T08:33:49.712 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3632999999999999e-05 2023-11-05T08:33:49.712 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.712 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.712 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:49.712 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.416913+0000", 2023-11-05T08:33:49.713 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00021194700000000001 2023-11-05T08:33:49.713 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.713 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.713 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:49.713 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.416946+0000", 2023-11-05T08:33:49.714 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2869e-05 2023-11-05T08:33:49.714 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.714 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.714 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:49.714 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.425257+0000", 2023-11-05T08:33:49.715 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0083110960000000005 2023-11-05T08:33:49.715 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.715 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.715 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:49.715 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.425364+0000", 2023-11-05T08:33:49.715 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010716499999999999 2023-11-05T08:33:49.716 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.716 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:49.716 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.716 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.716 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.717 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.18s4 66/45 ECSubWrite(tid=250, reqid=osd.13.0:250, at_version=66'38, trim_to=0'0, roll_forward_to=65'36))", 2023-11-05T08:33:49.717 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.418433+0000", 2023-11-05T08:33:49.717 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 30.050761239, 2023-11-05T08:33:49.717 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0069943790000000002, 2023-11-05T08:33:49.717 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:49.718 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:49.718 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:49.718 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.718 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:49.718 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.418433+0000", 2023-11-05T08:33:49.719 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.719 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.719 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.719 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:49.719 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.418433+0000", 2023-11-05T08:33:49.720 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.720 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.720 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.720 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:49.720 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.418433+0000", 2023-11-05T08:33:49.721 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.6300000000000005e-07 2023-11-05T08:33:49.721 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.721 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.721 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:49.721 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.418511+0000", 2023-11-05T08:33:49.722 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.7712000000000004e-05 2023-11-05T08:33:49.722 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.722 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.722 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:49.722 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.418542+0000", 2023-11-05T08:33:49.722 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0436000000000001e-05 2023-11-05T08:33:49.723 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.723 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.723 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:49.723 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.418564+0000", 2023-11-05T08:33:49.723 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.2175999999999999e-05 2023-11-05T08:33:49.724 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.724 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.724 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:49.724 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.418777+0000", 2023-11-05T08:33:49.724 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00021367700000000001 2023-11-05T08:33:49.725 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.725 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.725 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:49.725 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.418823+0000", 2023-11-05T08:33:49.725 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5791999999999998e-05 2023-11-05T08:33:49.725 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.726 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.726 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:49.726 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.425287+0000", 2023-11-05T08:33:49.726 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0064638330000000004 2023-11-05T08:33:49.726 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.727 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.727 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:49.727 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.425427+0000", 2023-11-05T08:33:49.727 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014009 2023-11-05T08:33:49.727 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.728 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:49.728 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.728 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.728 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.728 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(osd.8.22:3 3.17s0 3:e9ea46d0:::smithi09039252-22:head [assert-version v19,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2023-11-05T08:33:49.728 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:39.605255+0000", 2023-11-05T08:33:49.729 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 9.8639386610000006, 2023-11-05T08:33:49.729 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010434637, 2023-11-05T08:33:49.729 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:49.729 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2023-11-05T08:33:49.729 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:49.730 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.730 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:49.730 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.605255+0000", 2023-11-05T08:33:49.730 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.730 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.731 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.731 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:49.731 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.605255+0000", 2023-11-05T08:33:49.731 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:49.731 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.731 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.732 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:49.732 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.605258+0000", 2023-11-05T08:33:49.732 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3870000000000002e-06 2023-11-05T08:33:49.732 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.732 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.733 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:49.733 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.605269+0000", 2023-11-05T08:33:49.733 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0946e-05 2023-11-05T08:33:49.733 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.733 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.733 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:49.734 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.605292+0000", 2023-11-05T08:33:49.734 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.2906000000000001e-05 2023-11-05T08:33:49.734 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.734 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.734 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:49.735 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.605309+0000", 2023-11-05T08:33:49.735 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7557000000000001e-05 2023-11-05T08:33:49.735 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.735 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.735 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:49.736 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.605473+0000", 2023-11-05T08:33:49.736 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00016349999999999999 2023-11-05T08:33:49.736 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.736 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.736 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op must wait for map", 2023-11-05T08:33:49.736 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.605493+0000", 2023-11-05T08:33:49.737 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.0506e-05 2023-11-05T08:33:49.737 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.737 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.737 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:49.737 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.611114+0000", 2023-11-05T08:33:49.738 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0056206540000000001 2023-11-05T08:33:49.738 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.738 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.738 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:49.738 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.611328+0000", 2023-11-05T08:33:49.739 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00021375800000000001 2023-11-05T08:33:49.739 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:49.739 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:49.739 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:49.739 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.615690+0000", 2023-11-05T08:33:49.740 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0043624229999999998 2023-11-05T08:33:49.740 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.740 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:49.740 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.740 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:49.741 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:49.741 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:33:49.741 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-11-05T08:33:49.743 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:33:49.737+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:33:49.744 INFO:tasks.rados.rados.0.smithi090.stdout:117: done (4 left) 2023-11-05T08:33:49.744 INFO:tasks.rados.rados.0.smithi090.stdout:295: done (3 left) 2023-11-05T08:33:49.744 INFO:tasks.rados.rados.0.smithi090.stdout:297: done (2 left) 2023-11-05T08:33:49.745 INFO:tasks.rados.rados.0.smithi090.stdout:298: done (1 left) 2023-11-05T08:33:49.745 INFO:tasks.rados.rados.0.smithi090.stdout:306: done (0 left) 2023-11-05T08:33:49.745 INFO:tasks.rados.rados.0.smithi090.stdout:307: snap_remove snap 4 2023-11-05T08:33:49.746 INFO:teuthology.orchestra.run.smithi002.stderr:reweighted osd.2 to 1 (10000) 2023-11-05T08:33:49.746 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2023-11-05T08:33:49.747 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2023-11-05T08:33:49.843 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:33:49.840+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:33:49.882 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:33:49.882 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2023-11-05T08:33:49.883 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 0 2023-11-05T08:33:49.883 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:33:49.896 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2023-11-05T08:33:49.986 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:33:49.984+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:33:50.045 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-11-05T08:33:50.042+0000 7f72b2c5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 42525) UID: 0 2023-11-05T08:33:50.068 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:33:50.068 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2023-11-05T08:33:50.068 INFO:teuthology.orchestra.run.smithi053.stdout: "complaint_time": 30, 2023-11-05T08:33:50.068 INFO:teuthology.orchestra.run.smithi053.stdout: "num_blocked_ops": 0 2023-11-05T08:33:50.069 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:33:50.082 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2023-11-05T08:33:50.146 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:33:50.141+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:33:50.247 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:33:50.245+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:33:50.258 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:33:50.258 INFO:teuthology.orchestra.run.smithi053.stdout: "size": 20, 2023-11-05T08:33:50.258 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 600, 2023-11-05T08:33:50.258 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2023-11-05T08:33:50.259 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.259 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as3 41/33 ECSubWrite(tid=35, reqid=osd.7.0:35, at_version=41'12, trim_to=0'0, roll_forward_to=40'10))", 2023-11-05T08:33:50.259 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.129446+0000", 2023-11-05T08:33:50.259 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 56.125626216000001, 2023-11-05T08:33:50.259 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010453017, 2023-11-05T08:33:50.260 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:50.260 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:50.260 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:50.260 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.260 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:50.261 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.129446+0000", 2023-11-05T08:33:50.261 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.261 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.261 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.261 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:50.262 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.129446+0000", 2023-11-05T08:33:50.262 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.262 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.262 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.262 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:50.262 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.129446+0000", 2023-11-05T08:33:50.263 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.5000000000000002e-07 2023-11-05T08:33:50.263 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.263 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.263 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:50.263 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.129503+0000", 2023-11-05T08:33:50.264 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6286000000000003e-05 2023-11-05T08:33:50.264 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.264 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.264 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:50.264 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.129536+0000", 2023-11-05T08:33:50.265 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3036e-05 2023-11-05T08:33:50.265 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.265 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.265 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:50.265 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.129559+0000", 2023-11-05T08:33:50.265 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3091999999999998e-05 2023-11-05T08:33:50.266 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.266 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.266 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:50.266 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.129756+0000", 2023-11-05T08:33:50.266 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000196918 2023-11-05T08:33:50.267 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.267 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.267 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:50.267 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.129782+0000", 2023-11-05T08:33:50.267 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6013e-05 2023-11-05T08:33:50.268 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.268 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.268 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:50.268 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.139822+0000", 2023-11-05T08:33:50.268 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010040261 2023-11-05T08:33:50.268 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.269 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.269 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:50.269 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.139899+0000", 2023-11-05T08:33:50.269 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.6661000000000003e-05 2023-11-05T08:33:50.269 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.270 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:50.270 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.270 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.270 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.270 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as3 41/33 ECSubWrite(tid=36, reqid=osd.7.0:36, at_version=41'14, trim_to=0'0, roll_forward_to=40'10))", 2023-11-05T08:33:50.271 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.131534+0000", 2023-11-05T08:33:50.271 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 56.123537792, 2023-11-05T08:33:50.271 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.016827443000000001, 2023-11-05T08:33:50.271 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:50.271 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:50.272 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:50.272 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.272 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:50.272 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.131534+0000", 2023-11-05T08:33:50.272 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.273 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.273 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.273 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:50.273 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.131534+0000", 2023-11-05T08:33:50.273 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.274 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.274 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.274 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:50.274 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.131536+0000", 2023-11-05T08:33:50.274 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3769999999999999e-06 2023-11-05T08:33:50.275 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.275 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.275 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:50.275 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.131586+0000", 2023-11-05T08:33:50.275 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.0778000000000001e-05 2023-11-05T08:33:50.275 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.276 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.276 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:50.276 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.131619+0000", 2023-11-05T08:33:50.276 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2793000000000003e-05 2023-11-05T08:33:50.276 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.276 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.277 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:50.277 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.131642+0000", 2023-11-05T08:33:50.277 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3241999999999999e-05 2023-11-05T08:33:50.277 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.277 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.277 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:50.278 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.131806+0000", 2023-11-05T08:33:50.278 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000163843 2023-11-05T08:33:50.278 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.278 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.278 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:50.278 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.131832+0000", 2023-11-05T08:33:50.279 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5689000000000001e-05 2023-11-05T08:33:50.279 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.279 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.279 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:50.279 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148268+0000", 2023-11-05T08:33:50.279 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.016435924000000001 2023-11-05T08:33:50.280 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.280 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.280 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:50.280 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148362+0000", 2023-11-05T08:33:50.280 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.3796999999999997e-05 2023-11-05T08:33:50.280 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.281 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:50.281 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.281 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.281 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.281 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es1 41/30 ECSubWrite(tid=41, reqid=osd.2.0:40, at_version=0'0, trim_to=0'0, roll_forward_to=41'27))", 2023-11-05T08:33:50.281 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.148780+0000", 2023-11-05T08:33:50.282 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 56.106292187000001, 2023-11-05T08:33:50.282 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0090092869999999995, 2023-11-05T08:33:50.282 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:50.282 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:50.282 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:50.283 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.283 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:50.283 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148780+0000", 2023-11-05T08:33:50.283 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.283 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.284 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.284 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:50.284 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148780+0000", 2023-11-05T08:33:50.284 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.284 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.285 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.285 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:50.285 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148789+0000", 2023-11-05T08:33:50.285 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.2029999999999998e-06 2023-11-05T08:33:50.285 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.285 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.286 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:50.286 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148816+0000", 2023-11-05T08:33:50.286 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7291999999999999e-05 2023-11-05T08:33:50.286 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.286 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.287 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:50.287 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148847+0000", 2023-11-05T08:33:50.287 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1118999999999998e-05 2023-11-05T08:33:50.287 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.287 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.288 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:50.288 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.148871+0000", 2023-11-05T08:33:50.288 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3448999999999999e-05 2023-11-05T08:33:50.288 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.288 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.288 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:50.289 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.149090+0000", 2023-11-05T08:33:50.289 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00021928799999999999 2023-11-05T08:33:50.289 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.289 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.289 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:50.290 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.149116+0000", 2023-11-05T08:33:50.290 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6239000000000001e-05 2023-11-05T08:33:50.290 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.290 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.290 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:50.291 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.157693+0000", 2023-11-05T08:33:50.291 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0085768340000000002 2023-11-05T08:33:50.291 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.291 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.291 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:50.291 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.157789+0000", 2023-11-05T08:33:50.292 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.5863e-05 2023-11-05T08:33:50.292 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.292 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:50.292 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.292 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.293 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.293 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.11s2 41/40 ECSubWrite(tid=65, reqid=client.4691.0:287, at_version=41'13, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:50.293 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.173568+0000", 2023-11-05T08:33:50.293 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 56.081504090000003, 2023-11-05T08:33:50.293 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0078777859999999995, 2023-11-05T08:33:50.294 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:50.294 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:50.294 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:50.294 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.294 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:50.294 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.173568+0000", 2023-11-05T08:33:50.295 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.295 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.295 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.295 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:50.295 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.173568+0000", 2023-11-05T08:33:50.296 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.296 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.296 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.296 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:50.296 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.173570+0000", 2023-11-05T08:33:50.296 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9460000000000001e-06 2023-11-05T08:33:50.297 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.297 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.297 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:50.297 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.174495+0000", 2023-11-05T08:33:50.297 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00092541300000000004 2023-11-05T08:33:50.298 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.298 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.298 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:50.298 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.174540+0000", 2023-11-05T08:33:50.298 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5170999999999998e-05 2023-11-05T08:33:50.299 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.299 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.299 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:50.299 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.174564+0000", 2023-11-05T08:33:50.299 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3159e-05 2023-11-05T08:33:50.300 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.300 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.300 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:50.300 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.175041+0000", 2023-11-05T08:33:50.300 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00047766399999999999 2023-11-05T08:33:50.301 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.301 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.301 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:50.301 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.175062+0000", 2023-11-05T08:33:50.301 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1064999999999999e-05 2023-11-05T08:33:50.302 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.302 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.302 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:50.302 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.181334+0000", 2023-11-05T08:33:50.302 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0062722159999999997 2023-11-05T08:33:50.303 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.303 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.303 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:50.303 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.181446+0000", 2023-11-05T08:33:50.303 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000111152 2023-11-05T08:33:50.304 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.304 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:50.304 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.304 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.304 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.305 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as3 42/33 ECSubWrite(tid=38, reqid=osd.7.0:38, at_version=42'16, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:33:50.305 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:55.126927+0000", 2023-11-05T08:33:50.305 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 55.128145205000003, 2023-11-05T08:33:50.305 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.013340393000000001, 2023-11-05T08:33:50.305 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:50.306 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:50.306 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:50.306 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.306 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:50.306 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.126927+0000", 2023-11-05T08:33:50.306 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.307 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.307 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.307 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:50.307 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.126927+0000", 2023-11-05T08:33:50.307 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.308 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.308 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.308 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:50.308 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.126927+0000", 2023-11-05T08:33:50.308 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7e-07 2023-11-05T08:33:50.308 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.309 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.309 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:50.309 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.127040+0000", 2023-11-05T08:33:50.309 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00011244299999999999 2023-11-05T08:33:50.309 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.310 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.310 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:50.310 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.127069+0000", 2023-11-05T08:33:50.310 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9255999999999999e-05 2023-11-05T08:33:50.310 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.310 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.311 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:50.311 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.127093+0000", 2023-11-05T08:33:50.311 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3839e-05 2023-11-05T08:33:50.311 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.311 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.312 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:50.312 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.129369+0000", 2023-11-05T08:33:50.312 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0022765250000000002 2023-11-05T08:33:50.312 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.312 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.312 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:50.313 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.129395+0000", 2023-11-05T08:33:50.313 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5755999999999999e-05 2023-11-05T08:33:50.313 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.313 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.313 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:50.314 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.140178+0000", 2023-11-05T08:33:50.314 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010782523 2023-11-05T08:33:50.314 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.314 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.314 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:50.315 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.140267+0000", 2023-11-05T08:33:50.315 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9580999999999995e-05 2023-11-05T08:33:50.315 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.315 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:50.315 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.316 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.316 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.316 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as3 42/33 ECSubWrite(tid=39, reqid=osd.7.0:39, at_version=42'18, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:33:50.316 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:55.128574+0000", 2023-11-05T08:33:50.316 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 55.126497972999999, 2023-11-05T08:33:50.317 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.013189111, 2023-11-05T08:33:50.317 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:50.317 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:50.317 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:50.317 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.317 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:50.318 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.128574+0000", 2023-11-05T08:33:50.318 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.318 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.318 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.318 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:50.319 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.128574+0000", 2023-11-05T08:33:50.319 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.319 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.319 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.319 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:50.319 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.128575+0000", 2023-11-05T08:33:50.320 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1370000000000001e-06 2023-11-05T08:33:50.320 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.320 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.320 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:50.320 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.128665+0000", 2023-11-05T08:33:50.321 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9571e-05 2023-11-05T08:33:50.321 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.321 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.321 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:50.321 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.128693+0000", 2023-11-05T08:33:50.321 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8822e-05 2023-11-05T08:33:50.322 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.322 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.322 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:50.322 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.128737+0000", 2023-11-05T08:33:50.322 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3469e-05 2023-11-05T08:33:50.323 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.323 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.323 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:50.323 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.130720+0000", 2023-11-05T08:33:50.323 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0019833229999999999 2023-11-05T08:33:50.324 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.324 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.324 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:50.324 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.130746+0000", 2023-11-05T08:33:50.325 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5326000000000001e-05 2023-11-05T08:33:50.325 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.325 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.325 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:50.325 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.141675+0000", 2023-11-05T08:33:50.325 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010929589 2023-11-05T08:33:50.326 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.326 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.326 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:50.326 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.141763+0000", 2023-11-05T08:33:50.326 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.7874000000000006e-05 2023-11-05T08:33:50.327 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.327 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:50.327 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.327 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.327 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.327 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es1 42/30 ECSubWrite(tid=43, reqid=osd.2.0:43, at_version=42'29, trim_to=0'0, roll_forward_to=41'27))", 2023-11-05T08:33:50.328 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:55.134325+0000", 2023-11-05T08:33:50.328 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 55.120746771, 2023-11-05T08:33:50.328 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.008930136, 2023-11-05T08:33:50.328 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:50.328 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:50.329 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:50.329 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.329 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:50.329 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134325+0000", 2023-11-05T08:33:50.329 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.329 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.330 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.330 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:50.330 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134325+0000", 2023-11-05T08:33:50.330 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.330 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.331 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.331 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:50.331 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134327+0000", 2023-11-05T08:33:50.331 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6899999999999999e-06 2023-11-05T08:33:50.331 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.331 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.332 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:50.332 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134380+0000", 2023-11-05T08:33:50.332 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3458000000000002e-05 2023-11-05T08:33:50.332 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.332 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.333 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:50.333 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134410+0000", 2023-11-05T08:33:50.333 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9405999999999999e-05 2023-11-05T08:33:50.333 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.333 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.333 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:50.334 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134433+0000", 2023-11-05T08:33:50.334 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3745999999999999e-05 2023-11-05T08:33:50.334 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.334 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.334 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:50.335 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134900+0000", 2023-11-05T08:33:50.335 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00046645999999999998 2023-11-05T08:33:50.335 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.335 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.335 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:50.335 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.134925+0000", 2023-11-05T08:33:50.336 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5196000000000001e-05 2023-11-05T08:33:50.336 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.336 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.336 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:50.336 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.143157+0000", 2023-11-05T08:33:50.337 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0082321760000000008 2023-11-05T08:33:50.337 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.337 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.337 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:50.337 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:55.143255+0000", 2023-11-05T08:33:50.337 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.8004000000000006e-05 2023-11-05T08:33:50.338 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.338 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:50.338 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.338 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.338 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.341 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as3 52/45 ECSubWrite(tid=45, reqid=client.4691.0:349, at_version=52'21, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:50.341 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:05.263635+0000", 2023-11-05T08:33:50.341 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 44.991436555999996, 2023-11-05T08:33:50.342 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0081552580000000003, 2023-11-05T08:33:50.342 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:50.342 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:50.342 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:50.342 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.343 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:50.343 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.263635+0000", 2023-11-05T08:33:50.343 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.343 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.343 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.344 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:50.344 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.263635+0000", 2023-11-05T08:33:50.344 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.344 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.344 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.345 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:50.345 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.263637+0000", 2023-11-05T08:33:50.345 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.897e-06 2023-11-05T08:33:50.345 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.345 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.345 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:50.346 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.264062+0000", 2023-11-05T08:33:50.346 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00042438099999999999 2023-11-05T08:33:50.346 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.346 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.346 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:50.347 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.264102+0000", 2023-11-05T08:33:50.347 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.0058999999999997e-05 2023-11-05T08:33:50.347 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.347 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.347 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:50.348 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.264138+0000", 2023-11-05T08:33:50.348 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.6208999999999998e-05 2023-11-05T08:33:50.348 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.348 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.348 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:50.348 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.264324+0000", 2023-11-05T08:33:50.349 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00018639199999999999 2023-11-05T08:33:50.349 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.349 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.349 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:50.349 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.264351+0000", 2023-11-05T08:33:50.350 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6673e-05 2023-11-05T08:33:50.350 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.350 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.350 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:50.350 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.271652+0000", 2023-11-05T08:33:50.351 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0073011170000000002 2023-11-05T08:33:50.351 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.351 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.351 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:50.351 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.271791+0000", 2023-11-05T08:33:50.352 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013852999999999999 2023-11-05T08:33:50.352 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.352 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:50.352 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.352 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.352 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.353 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as3 61/45 ECSubWrite(tid=57, reqid=client.4691.0:405, at_version=61'23, trim_to=0'0, roll_forward_to=52'21))", 2023-11-05T08:33:50.353 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:14.365006+0000", 2023-11-05T08:33:50.353 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 35.890066181999998, 2023-11-05T08:33:50.353 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0064071809999999996, 2023-11-05T08:33:50.353 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:50.354 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:50.354 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:50.354 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.354 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:50.354 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.365006+0000", 2023-11-05T08:33:50.355 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.355 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.355 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.355 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:50.355 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.365006+0000", 2023-11-05T08:33:50.355 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.356 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.356 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.356 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:50.356 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.365007+0000", 2023-11-05T08:33:50.356 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.72e-06 2023-11-05T08:33:50.357 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.357 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.357 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:50.357 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.365501+0000", 2023-11-05T08:33:50.357 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00049378099999999999 2023-11-05T08:33:50.358 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.358 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.358 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:50.358 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.365533+0000", 2023-11-05T08:33:50.358 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1835999999999998e-05 2023-11-05T08:33:50.358 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.359 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.359 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:50.359 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.365550+0000", 2023-11-05T08:33:50.359 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6813e-05 2023-11-05T08:33:50.359 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.360 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.360 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:50.360 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.365689+0000", 2023-11-05T08:33:50.360 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000138784 2023-11-05T08:33:50.360 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.360 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.361 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:50.361 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.365722+0000", 2023-11-05T08:33:50.361 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3509000000000001e-05 2023-11-05T08:33:50.361 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.361 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.362 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:50.362 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.371344+0000", 2023-11-05T08:33:50.362 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.005621366 2023-11-05T08:33:50.362 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.362 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.362 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:50.363 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.371413+0000", 2023-11-05T08:33:50.363 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9371999999999999e-05 2023-11-05T08:33:50.363 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.363 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:50.363 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.364 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.364 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.364 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.as3 61/45 ECSubWrite(tid=58, reqid=client.4691.0:406, at_version=61'24, trim_to=0'0, roll_forward_to=52'21))", 2023-11-05T08:33:50.364 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:14.371923+0000", 2023-11-05T08:33:50.364 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 35.883149351, 2023-11-05T08:33:50.365 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0074408499999999997, 2023-11-05T08:33:50.365 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:50.365 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:50.365 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:50.365 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.366 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:50.366 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.371923+0000", 2023-11-05T08:33:50.366 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.366 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.366 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.366 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:50.367 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.371923+0000", 2023-11-05T08:33:50.367 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.367 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.367 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.367 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:50.368 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.371925+0000", 2023-11-05T08:33:50.368 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.153e-06 2023-11-05T08:33:50.368 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.368 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.368 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:50.368 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.372289+0000", 2023-11-05T08:33:50.369 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00036441300000000001 2023-11-05T08:33:50.369 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.369 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.369 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:50.369 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.372329+0000", 2023-11-05T08:33:50.370 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.9906e-05 2023-11-05T08:33:50.370 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.370 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.370 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:50.370 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.372353+0000", 2023-11-05T08:33:50.371 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3856e-05 2023-11-05T08:33:50.371 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.371 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.371 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:50.371 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.372522+0000", 2023-11-05T08:33:50.371 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00016944 2023-11-05T08:33:50.372 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.372 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.372 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:50.372 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.372551+0000", 2023-11-05T08:33:50.372 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8856e-05 2023-11-05T08:33:50.373 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.373 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.373 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:50.373 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.379263+0000", 2023-11-05T08:33:50.373 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0067113049999999999 2023-11-05T08:33:50.374 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.374 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.374 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:50.374 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:14.379363+0000", 2023-11-05T08:33:50.374 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000100921 2023-11-05T08:33:50.375 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.375 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:50.375 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.375 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.375 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.376 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.bs6 66/37 ECSubWrite(tid=108, reqid=client.4691.0:496, at_version=66'23, trim_to=0'0, roll_forward_to=65'21))", 2023-11-05T08:33:50.376 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.399096+0000", 2023-11-05T08:33:50.376 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 30.855975987000001, 2023-11-05T08:33:50.376 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010759829, 2023-11-05T08:33:50.376 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:50.377 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:50.377 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:50.377 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.377 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:50.378 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.399096+0000", 2023-11-05T08:33:50.378 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.378 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.378 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.379 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:50.379 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.399096+0000", 2023-11-05T08:33:50.379 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.379 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.379 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.380 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:50.380 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.399098+0000", 2023-11-05T08:33:50.380 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.0899999999999999e-06 2023-11-05T08:33:50.380 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.380 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.381 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:50.381 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.399777+0000", 2023-11-05T08:33:50.381 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000678468 2023-11-05T08:33:50.381 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.381 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.382 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:50.382 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.399810+0000", 2023-11-05T08:33:50.382 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3646000000000003e-05 2023-11-05T08:33:50.382 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.382 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.383 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:50.383 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.399834+0000", 2023-11-05T08:33:50.383 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3782999999999999e-05 2023-11-05T08:33:50.383 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.383 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.384 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:50.384 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.399991+0000", 2023-11-05T08:33:50.384 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000157317 2023-11-05T08:33:50.384 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.384 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.384 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:50.385 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.400017+0000", 2023-11-05T08:33:50.385 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5996999999999999e-05 2023-11-05T08:33:50.385 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.385 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.385 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:50.386 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.409770+0000", 2023-11-05T08:33:50.386 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0097524240000000009 2023-11-05T08:33:50.386 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.386 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.386 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:50.387 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.409856+0000", 2023-11-05T08:33:50.387 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.6104000000000001e-05 2023-11-05T08:33:50.387 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.387 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:50.387 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.388 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.388 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.388 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.16s4 66/45 ECSubWrite(tid=112, reqid=osd.11.0:112, at_version=66'34, trim_to=0'0, roll_forward_to=65'32))", 2023-11-05T08:33:50.388 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.401373+0000", 2023-11-05T08:33:50.388 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 30.853698841, 2023-11-05T08:33:50.388 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0064814720000000003, 2023-11-05T08:33:50.389 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:50.389 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:50.389 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:50.389 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.389 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:50.390 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.401373+0000", 2023-11-05T08:33:50.390 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.390 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.390 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.390 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:50.391 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.401373+0000", 2023-11-05T08:33:50.391 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.391 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.391 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.391 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:50.392 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.401374+0000", 2023-11-05T08:33:50.392 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9699999999999996e-07 2023-11-05T08:33:50.392 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.392 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.392 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:50.393 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.401420+0000", 2023-11-05T08:33:50.393 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.6683e-05 2023-11-05T08:33:50.393 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.393 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.393 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:50.394 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.401450+0000", 2023-11-05T08:33:50.394 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9788999999999999e-05 2023-11-05T08:33:50.394 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.394 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.394 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:50.394 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.401473+0000", 2023-11-05T08:33:50.395 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.2682999999999999e-05 2023-11-05T08:33:50.395 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.395 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.395 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:50.395 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.401644+0000", 2023-11-05T08:33:50.396 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00017150999999999999 2023-11-05T08:33:50.396 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.396 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.396 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:50.396 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.401674+0000", 2023-11-05T08:33:50.397 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9723e-05 2023-11-05T08:33:50.397 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.397 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.397 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:50.397 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.407763+0000", 2023-11-05T08:33:50.397 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0060884950000000002 2023-11-05T08:33:50.398 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.398 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.398 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:50.398 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.407855+0000", 2023-11-05T08:33:50.398 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1991999999999996e-05 2023-11-05T08:33:50.399 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.399 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:50.399 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.399 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.399 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.400 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.bs6 66/37 ECSubWrite(tid=109, reqid=client.4691.0:497, at_version=66'24, trim_to=0'0, roll_forward_to=65'21))", 2023-11-05T08:33:50.400 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.404475+0000", 2023-11-05T08:33:50.400 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 30.850596736, 2023-11-05T08:33:50.400 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.006663576, 2023-11-05T08:33:50.400 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:50.400 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:50.401 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:50.401 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.401 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:50.401 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.404475+0000", 2023-11-05T08:33:50.401 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.402 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.402 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.402 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:50.402 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.404475+0000", 2023-11-05T08:33:50.402 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.403 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.403 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.403 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:50.403 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.404477+0000", 2023-11-05T08:33:50.403 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.0030000000000001e-06 2023-11-05T08:33:50.403 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.404 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.404 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:50.404 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.404894+0000", 2023-11-05T08:33:50.404 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000416697 2023-11-05T08:33:50.404 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.404 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.405 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:50.405 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.404928+0000", 2023-11-05T08:33:50.405 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4329000000000003e-05 2023-11-05T08:33:50.405 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.405 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.406 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:50.406 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.404989+0000", 2023-11-05T08:33:50.406 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0424999999999998e-05 2023-11-05T08:33:50.406 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.406 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.407 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:50.407 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.405674+0000", 2023-11-05T08:33:50.407 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000685678 2023-11-05T08:33:50.407 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.407 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.407 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:50.408 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.405700+0000", 2023-11-05T08:33:50.408 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5313e-05 2023-11-05T08:33:50.408 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.408 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.408 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:50.409 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.411060+0000", 2023-11-05T08:33:50.409 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0053600990000000001 2023-11-05T08:33:50.409 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.409 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.409 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:50.409 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.411139+0000", 2023-11-05T08:33:50.410 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.9031999999999998e-05 2023-11-05T08:33:50.410 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.410 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:50.410 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.410 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.411 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.411 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.bs6 66/37 ECSubWrite(tid=110, reqid=client.4691.0:498, at_version=66'25, trim_to=0'0, roll_forward_to=65'21))", 2023-11-05T08:33:50.411 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.411793+0000", 2023-11-05T08:33:50.411 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 30.843279052, 2023-11-05T08:33:50.411 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0078269380000000003, 2023-11-05T08:33:50.412 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:50.412 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:50.412 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:50.412 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.412 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:50.412 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.411793+0000", 2023-11-05T08:33:50.413 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.413 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.413 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.413 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:50.413 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.411793+0000", 2023-11-05T08:33:50.414 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.414 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.414 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.414 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:50.414 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.411795+0000", 2023-11-05T08:33:50.414 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.827e-06 2023-11-05T08:33:50.415 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.415 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.415 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:50.415 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.411946+0000", 2023-11-05T08:33:50.415 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00015118999999999999 2023-11-05T08:33:50.416 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.416 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.416 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:50.416 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.411995+0000", 2023-11-05T08:33:50.416 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9175999999999998e-05 2023-11-05T08:33:50.416 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.417 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.417 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:50.417 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.412024+0000", 2023-11-05T08:33:50.417 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8739999999999999e-05 2023-11-05T08:33:50.417 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.418 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.418 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:50.418 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.412505+0000", 2023-11-05T08:33:50.418 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00048081399999999998 2023-11-05T08:33:50.418 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.419 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.419 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:50.419 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.412530+0000", 2023-11-05T08:33:50.419 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5374000000000001e-05 2023-11-05T08:33:50.419 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.419 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.420 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:50.420 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.419518+0000", 2023-11-05T08:33:50.420 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0069876119999999998 2023-11-05T08:33:50.420 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.420 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.421 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:50.421 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.419620+0000", 2023-11-05T08:33:50.421 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000102205 2023-11-05T08:33:50.421 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.421 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:50.421 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.422 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.422 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.422 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.bs6 66/37 ECSubWrite(tid=111, reqid=client.4691.0:499, at_version=66'26, trim_to=0'0, roll_forward_to=65'21))", 2023-11-05T08:33:50.422 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.412085+0000", 2023-11-05T08:33:50.422 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 30.842987260000001, 2023-11-05T08:33:50.423 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0076042080000000003, 2023-11-05T08:33:50.423 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:50.423 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:50.423 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:50.423 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.424 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:50.424 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.412085+0000", 2023-11-05T08:33:50.424 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.424 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.424 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.424 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:50.425 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.412085+0000", 2023-11-05T08:33:50.425 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.425 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.425 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.425 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:50.426 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.412086+0000", 2023-11-05T08:33:50.426 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0359999999999999e-06 2023-11-05T08:33:50.426 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.426 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.426 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:50.426 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.412133+0000", 2023-11-05T08:33:50.427 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7076000000000001e-05 2023-11-05T08:33:50.427 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.427 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.427 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:50.427 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.412161+0000", 2023-11-05T08:33:50.428 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8518999999999999e-05 2023-11-05T08:33:50.428 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.428 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.428 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:50.428 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.412181+0000", 2023-11-05T08:33:50.428 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9347e-05 2023-11-05T08:33:50.429 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.429 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.429 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:50.429 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.413827+0000", 2023-11-05T08:33:50.429 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001646034 2023-11-05T08:33:50.430 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.430 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.430 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:50.430 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.413851+0000", 2023-11-05T08:33:50.430 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4332999999999999e-05 2023-11-05T08:33:50.431 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.431 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.431 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:50.431 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.419630+0000", 2023-11-05T08:33:50.431 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0057790309999999996 2023-11-05T08:33:50.431 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.432 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.432 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:50.432 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.419689+0000", 2023-11-05T08:33:50.432 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8832000000000002e-05 2023-11-05T08:33:50.432 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.433 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:50.433 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.433 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.433 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.433 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es1 66/30 ECSubWrite(tid=87, reqid=osd.2.0:87, at_version=66'65, trim_to=0'0, roll_forward_to=65'63))", 2023-11-05T08:33:50.433 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.413863+0000", 2023-11-05T08:33:50.434 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 30.841209049, 2023-11-05T08:33:50.434 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0091784380000000006, 2023-11-05T08:33:50.434 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:50.434 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:50.434 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:50.435 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.435 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:50.435 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.413863+0000", 2023-11-05T08:33:50.435 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.435 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.435 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.436 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:50.436 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.413863+0000", 2023-11-05T08:33:50.436 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.436 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.436 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.437 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:50.437 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.413864+0000", 2023-11-05T08:33:50.437 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.407e-06 2023-11-05T08:33:50.437 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.437 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.437 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:50.438 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.413917+0000", 2023-11-05T08:33:50.438 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2785999999999999e-05 2023-11-05T08:33:50.438 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.438 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.438 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:50.439 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.413947+0000", 2023-11-05T08:33:50.439 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9422999999999999e-05 2023-11-05T08:33:50.439 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.439 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.439 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:50.439 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.413969+0000", 2023-11-05T08:33:50.440 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.2582000000000002e-05 2023-11-05T08:33:50.440 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.440 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.440 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:50.440 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.414151+0000", 2023-11-05T08:33:50.441 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00018211099999999999 2023-11-05T08:33:50.441 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.441 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.441 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:50.441 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.414182+0000", 2023-11-05T08:33:50.441 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1072000000000003e-05 2023-11-05T08:33:50.442 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.442 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.442 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:50.442 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.422947+0000", 2023-11-05T08:33:50.442 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0087647520000000007 2023-11-05T08:33:50.442 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.443 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.443 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:50.443 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.423041+0000", 2023-11-05T08:33:50.443 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.4304999999999999e-05 2023-11-05T08:33:50.443 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.444 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:50.444 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.444 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.444 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.444 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.1s2 66/40 ECSubWrite(tid=112, reqid=osd.4.0:107, at_version=0'0, trim_to=0'0, roll_forward_to=66'26))", 2023-11-05T08:33:50.445 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.415009+0000", 2023-11-05T08:33:50.445 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 30.840063162, 2023-11-05T08:33:50.445 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0079745690000000008, 2023-11-05T08:33:50.445 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:50.445 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:50.445 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:50.446 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.446 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:50.446 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.415009+0000", 2023-11-05T08:33:50.446 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.446 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.447 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.447 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:50.447 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.415009+0000", 2023-11-05T08:33:50.447 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.447 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.448 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.448 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:50.448 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.415010+0000", 2023-11-05T08:33:50.448 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.68e-06 2023-11-05T08:33:50.448 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.449 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.449 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:50.449 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.415039+0000", 2023-11-05T08:33:50.449 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8076e-05 2023-11-05T08:33:50.449 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.449 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.450 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:50.450 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.415262+0000", 2023-11-05T08:33:50.450 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000223019 2023-11-05T08:33:50.450 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.450 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.451 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:50.451 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.415286+0000", 2023-11-05T08:33:50.451 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4162999999999999e-05 2023-11-05T08:33:50.451 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.451 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.452 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:50.452 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.415463+0000", 2023-11-05T08:33:50.452 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00017772400000000001 2023-11-05T08:33:50.452 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.452 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.453 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:50.453 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.415494+0000", 2023-11-05T08:33:50.453 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1025999999999997e-05 2023-11-05T08:33:50.453 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.453 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.453 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:50.454 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.422878+0000", 2023-11-05T08:33:50.454 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0073837720000000003 2023-11-05T08:33:50.454 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.454 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.454 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:50.455 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.422983+0000", 2023-11-05T08:33:50.455 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000105109 2023-11-05T08:33:50.455 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.455 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:50.455 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.456 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.456 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.456 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.es1 66/30 ECSubWrite(tid=88, reqid=osd.2.0:88, at_version=66'67, trim_to=0'0, roll_forward_to=65'63))", 2023-11-05T08:33:50.456 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.416186+0000", 2023-11-05T08:33:50.456 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 30.838886420000001, 2023-11-05T08:33:50.456 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.008119823, 2023-11-05T08:33:50.457 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:50.457 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:50.457 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:50.457 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.457 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:50.458 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.416186+0000", 2023-11-05T08:33:50.458 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.458 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.458 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.458 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:50.459 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.416186+0000", 2023-11-05T08:33:50.459 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.459 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.459 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.459 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:50.460 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.416187+0000", 2023-11-05T08:33:50.460 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.77e-06 2023-11-05T08:33:50.460 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.460 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.460 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:50.460 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.416239+0000", 2023-11-05T08:33:50.461 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1229e-05 2023-11-05T08:33:50.461 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.461 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.461 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:50.461 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.416297+0000", 2023-11-05T08:33:50.462 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8856e-05 2023-11-05T08:33:50.462 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.462 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.462 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:50.462 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.416324+0000", 2023-11-05T08:33:50.463 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6202000000000001e-05 2023-11-05T08:33:50.463 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.463 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.463 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:50.463 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.416570+0000", 2023-11-05T08:33:50.463 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00024649999999999997 2023-11-05T08:33:50.464 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.464 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.464 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:50.464 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.416602+0000", 2023-11-05T08:33:50.464 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1829000000000003e-05 2023-11-05T08:33:50.465 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.465 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.465 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:50.465 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.424215+0000", 2023-11-05T08:33:50.465 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0076131990000000002 2023-11-05T08:33:50.465 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.466 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.466 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:50.466 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.424305+0000", 2023-11-05T08:33:50.466 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.0237999999999999e-05 2023-11-05T08:33:50.466 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.467 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:50.467 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.467 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.467 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.467 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.16s4 66/45 ECSubWrite(tid=115, reqid=osd.11.0:112, at_version=0'0, trim_to=0'0, roll_forward_to=66'34))", 2023-11-05T08:33:50.468 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:19.416982+0000", 2023-11-05T08:33:50.468 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 30.838089964000002, 2023-11-05T08:33:50.468 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0061497449999999999, 2023-11-05T08:33:50.468 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:50.468 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:50.468 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:50.469 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.469 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:50.469 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.416982+0000", 2023-11-05T08:33:50.469 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.469 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.470 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.470 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:50.470 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.416982+0000", 2023-11-05T08:33:50.470 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.470 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.471 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.471 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:50.471 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.416984+0000", 2023-11-05T08:33:50.471 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.8959999999999999e-06 2023-11-05T08:33:50.471 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.471 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.472 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:50.472 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.417010+0000", 2023-11-05T08:33:50.472 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6587000000000001e-05 2023-11-05T08:33:50.472 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.472 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.473 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:50.473 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.417042+0000", 2023-11-05T08:33:50.473 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1816000000000002e-05 2023-11-05T08:33:50.473 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.473 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.474 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:50.474 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.417065+0000", 2023-11-05T08:33:50.474 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.2753000000000001e-05 2023-11-05T08:33:50.474 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.474 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.475 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:50.475 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.417225+0000", 2023-11-05T08:33:50.475 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00015948700000000001 2023-11-05T08:33:50.475 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.475 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.476 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:50.476 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.417256+0000", 2023-11-05T08:33:50.476 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1563000000000003e-05 2023-11-05T08:33:50.476 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.476 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.477 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:50.477 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.423049+0000", 2023-11-05T08:33:50.477 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0057928370000000003 2023-11-05T08:33:50.477 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.477 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.477 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:50.478 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:19.423132+0000", 2023-11-05T08:33:50.478 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.2806000000000006e-05 2023-11-05T08:33:50.478 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.478 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:50.478 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.479 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.479 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.479 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "MOSDECSubOpWrite(3.19s3 86/45 ECSubWrite(tid=51, reqid=client.4691.0:635, at_version=86'33, trim_to=0'0, roll_forward_to=65'31))", 2023-11-05T08:33:50.479 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:39.619852+0000", 2023-11-05T08:33:50.479 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 10.635219848, 2023-11-05T08:33:50.479 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0072255499999999999, 2023-11-05T08:33:50.480 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:50.480 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2023-11-05T08:33:50.480 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:50.480 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.480 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:50.481 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.619852+0000", 2023-11-05T08:33:50.481 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.481 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.481 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.481 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:50.482 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.619852+0000", 2023-11-05T08:33:50.482 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.482 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.482 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.482 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:50.482 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.619853+0000", 2023-11-05T08:33:50.483 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.0299999999999998e-07 2023-11-05T08:33:50.483 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.483 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.483 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:50.483 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.620307+0000", 2023-11-05T08:33:50.484 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000454559 2023-11-05T08:33:50.484 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.484 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.484 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:50.484 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.620321+0000", 2023-11-05T08:33:50.484 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3485e-05 2023-11-05T08:33:50.485 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.485 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.485 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:50.485 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.620331+0000", 2023-11-05T08:33:50.485 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0271e-05 2023-11-05T08:33:50.486 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.486 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.486 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:50.486 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.620588+0000", 2023-11-05T08:33:50.486 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00025698099999999998 2023-11-05T08:33:50.486 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.487 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.487 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:50.487 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.620608+0000", 2023-11-05T08:33:50.487 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9902999999999999e-05 2023-11-05T08:33:50.487 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.488 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.488 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:50.488 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.627033+0000", 2023-11-05T08:33:50.488 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0064250230000000002 2023-11-05T08:33:50.488 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.488 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.489 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:50.489 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:39.627078+0000", 2023-11-05T08:33:50.489 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4524999999999998e-05 2023-11-05T08:33:50.489 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.489 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:50.490 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.490 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:50.490 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:50.490 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:33:50.492 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2023-11-05T08:33:50.492 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:33:50.489+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:33:50.592 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:33:50.589+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:33:50.623 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:33:50.623 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2023-11-05T08:33:50.624 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 0 2023-11-05T08:33:50.624 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:33:50.637 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2023-11-05T08:33:50.651 INFO:teuthology.orchestra.run.smithi002.stderr:nodeep-scrub is unset 2023-11-05T08:33:50.652 INFO:teuthology.orchestra.run.smithi002.stderr:reweighted osd.3 to 1 (10000) 2023-11-05T08:33:50.663 INFO:tasks.rados.rados.0.smithi090.stdout:307: done (0 left) 2023-11-05T08:33:50.664 INFO:tasks.rados.rados.0.smithi090.stdout:308: snap_remove snap 31 2023-11-05T08:33:50.692 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2023-11-05T08:33:50.693 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:33:50.689+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:33:50.794 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:50.788+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:50.811 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:33:50.812 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2023-11-05T08:33:50.812 INFO:teuthology.orchestra.run.smithi053.stdout: "complaint_time": 30, 2023-11-05T08:33:50.812 INFO:teuthology.orchestra.run.smithi053.stdout: "num_blocked_ops": 0 2023-11-05T08:33:50.812 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:33:50.826 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2023-11-05T08:33:50.895 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:50.890+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:50.984 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-05T08:33:50.993 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:33:50.993 INFO:teuthology.orchestra.run.smithi053.stdout: "size": 20, 2023-11-05T08:33:50.993 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 600, 2023-11-05T08:33:50.994 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2023-11-05T08:33:50.994 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.994 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:50.994 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:28.710073+0000", 2023-11-05T08:33:50.995 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 82.278661635000006, 2023-11-05T08:33:50.995 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.052834791999999998, 2023-11-05T08:33:50.995 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:50.995 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:50.995 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:50.996 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:50.996 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:50.996 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 1 2023-11-05T08:33:50.996 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.996 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:50.997 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.997 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:50.997 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710073+0000", 2023-11-05T08:33:50.997 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.997 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.997 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.998 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:50.998 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710073+0000", 2023-11-05T08:33:50.998 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:50.998 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:50.999 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:50.999 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:50.999 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710076+0000", 2023-11-05T08:33:50.999 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1729999999999999e-06 2023-11-05T08:33:50.999 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.000 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.001 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:51.001 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710108+0000", 2023-11-05T08:33:51.001 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1458000000000003e-05 2023-11-05T08:33:51.001 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.001 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.002 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:51.002 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710140+0000", 2023-11-05T08:33:51.002 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2532e-05 2023-11-05T08:33:51.002 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.002 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.003 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:51.003 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710165+0000", 2023-11-05T08:33:51.003 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5205e-05 2023-11-05T08:33:51.003 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.003 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.004 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:51.004 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710311+0000", 2023-11-05T08:33:51.004 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014532799999999999 2023-11-05T08:33:51.004 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.004 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.005 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2023-11-05T08:33:51.005 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710327+0000", 2023-11-05T08:33:51.005 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6059e-05 2023-11-05T08:33:51.005 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.005 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.006 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:51.006 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.759348+0000", 2023-11-05T08:33:51.006 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.049021481999999998 2023-11-05T08:33:51.006 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.006 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.007 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:51.007 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.759880+0000", 2023-11-05T08:33:51.007 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00053197400000000005 2023-11-05T08:33:51.007 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.007 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.008 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:51.008 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.760478+0000", 2023-11-05T08:33:51.008 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00059839300000000002 2023-11-05T08:33:51.008 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.008 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.009 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:51.009 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.761955+0000", 2023-11-05T08:33:51.009 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001476991 2023-11-05T08:33:51.009 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.009 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.010 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:51.010 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.762611+0000", 2023-11-05T08:33:51.010 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00065598700000000002 2023-11-05T08:33:51.010 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.010 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.011 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:51.011 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.762782+0000", 2023-11-05T08:33:51.011 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00017064000000000001 2023-11-05T08:33:51.011 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.011 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.012 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:51.012 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.762908+0000", 2023-11-05T08:33:51.012 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012557000000000001 2023-11-05T08:33:51.012 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.012 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:51.013 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.013 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.013 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.013 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:51.013 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.151583+0000", 2023-11-05T08:33:51.014 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 81.837151841999997, 2023-11-05T08:33:51.014 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.020286660000000001, 2023-11-05T08:33:51.014 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:51.014 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:51.014 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:51.015 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:51.015 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:51.015 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 114 2023-11-05T08:33:51.015 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.015 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:51.016 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.016 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:51.016 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151583+0000", 2023-11-05T08:33:51.016 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:51.016 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.017 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.017 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:51.017 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151583+0000", 2023-11-05T08:33:51.017 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:51.017 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.017 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.018 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:51.018 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151583+0000", 2023-11-05T08:33:51.018 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.2600000000000002e-07 2023-11-05T08:33:51.018 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.018 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.019 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:51.019 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151589+0000", 2023-11-05T08:33:51.019 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5099999999999998e-06 2023-11-05T08:33:51.019 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.019 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.020 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:51.020 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151598+0000", 2023-11-05T08:33:51.020 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1730000000000001e-06 2023-11-05T08:33:51.020 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.020 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.020 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:51.021 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151608+0000", 2023-11-05T08:33:51.021 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0059e-05 2023-11-05T08:33:51.021 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.021 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.021 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:51.021 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.165015+0000", 2023-11-05T08:33:51.022 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.013407207000000001 2023-11-05T08:33:51.022 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.022 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.022 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:51.022 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.165268+0000", 2023-11-05T08:33:51.023 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00025284999999999999 2023-11-05T08:33:51.023 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.023 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.023 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:51.023 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.165635+0000", 2023-11-05T08:33:51.024 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00036695899999999998 2023-11-05T08:33:51.024 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.024 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.024 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:51.024 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.170105+0000", 2023-11-05T08:33:51.024 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0044696960000000004 2023-11-05T08:33:51.025 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.025 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.025 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:51.025 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.171719+0000", 2023-11-05T08:33:51.025 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001613855 2023-11-05T08:33:51.026 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.026 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.026 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:51.026 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.171810+0000", 2023-11-05T08:33:51.026 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1625000000000005e-05 2023-11-05T08:33:51.027 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.027 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.027 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:51.027 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.171869+0000", 2023-11-05T08:33:51.027 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.91e-05 2023-11-05T08:33:51.028 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.028 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:51.028 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.028 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.028 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.028 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:51.029 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.151660+0000", 2023-11-05T08:33:51.029 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 81.837074043000001, 2023-11-05T08:33:51.029 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022553719, 2023-11-05T08:33:51.029 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:51.029 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:51.030 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:51.030 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:51.030 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:51.030 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 115 2023-11-05T08:33:51.030 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.031 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:51.031 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.031 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:51.031 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151660+0000", 2023-11-05T08:33:51.031 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:51.031 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.032 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.032 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:51.032 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151660+0000", 2023-11-05T08:33:51.032 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:51.032 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.033 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.033 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:51.033 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151661+0000", 2023-11-05T08:33:51.033 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.5600000000000004e-07 2023-11-05T08:33:51.033 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.033 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.034 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:51.034 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151799+0000", 2023-11-05T08:33:51.034 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000137207 2023-11-05T08:33:51.034 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.034 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.035 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:51.035 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151814+0000", 2023-11-05T08:33:51.035 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4885000000000001e-05 2023-11-05T08:33:51.035 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.035 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.036 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:51.036 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151827+0000", 2023-11-05T08:33:51.036 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3413e-05 2023-11-05T08:33:51.036 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.036 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.037 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:51.037 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.167513+0000", 2023-11-05T08:33:51.037 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.015686091999999999 2023-11-05T08:33:51.037 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.037 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.038 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:51.038 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.167652+0000", 2023-11-05T08:33:51.038 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013890500000000001 2023-11-05T08:33:51.038 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.038 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.039 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:51.039 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.167860+0000", 2023-11-05T08:33:51.039 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00020838599999999999 2023-11-05T08:33:51.039 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.039 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.040 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:51.040 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.171602+0000", 2023-11-05T08:33:51.040 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0037417449999999999 2023-11-05T08:33:51.040 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.040 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.040 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:51.041 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174068+0000", 2023-11-05T08:33:51.041 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0024658010000000001 2023-11-05T08:33:51.041 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.041 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.041 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:51.042 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174152+0000", 2023-11-05T08:33:51.042 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.3752999999999996e-05 2023-11-05T08:33:51.042 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.042 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.042 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:51.043 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174214+0000", 2023-11-05T08:33:51.043 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.2576000000000005e-05 2023-11-05T08:33:51.043 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.043 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:51.043 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.044 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.044 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.044 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:51.044 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.151849+0000", 2023-11-05T08:33:51.044 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 81.836885687000006, 2023-11-05T08:33:51.044 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022629176000000001, 2023-11-05T08:33:51.045 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:51.045 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:51.045 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:51.045 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:51.045 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:51.046 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 116 2023-11-05T08:33:51.046 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.046 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:51.046 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.046 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:51.047 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151849+0000", 2023-11-05T08:33:51.047 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:51.047 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.047 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.047 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:51.048 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151849+0000", 2023-11-05T08:33:51.048 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:51.048 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.048 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.048 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:51.049 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151850+0000", 2023-11-05T08:33:51.049 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.037e-06 2023-11-05T08:33:51.049 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.049 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.049 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:51.049 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151858+0000", 2023-11-05T08:33:51.050 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.3389999999999993e-06 2023-11-05T08:33:51.050 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.050 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.050 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:51.050 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151868+0000", 2023-11-05T08:33:51.051 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6670000000000008e-06 2023-11-05T08:33:51.051 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.051 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.051 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:51.051 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151875+0000", 2023-11-05T08:33:51.052 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9229999999999996e-06 2023-11-05T08:33:51.052 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.052 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.052 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:51.052 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.168777+0000", 2023-11-05T08:33:51.052 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.016901884999999998 2023-11-05T08:33:51.053 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.053 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.053 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:51.053 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.169009+0000", 2023-11-05T08:33:51.053 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00023236400000000001 2023-11-05T08:33:51.054 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.054 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.054 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:51.054 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.169365+0000", 2023-11-05T08:33:51.054 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00035625300000000001 2023-11-05T08:33:51.055 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.055 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.055 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:51.055 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.171624+0000", 2023-11-05T08:33:51.055 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0022587169999999999 2023-11-05T08:33:51.056 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.056 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.056 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:51.056 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174314+0000", 2023-11-05T08:33:51.056 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002690263 2023-11-05T08:33:51.057 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.057 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.057 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:51.057 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174405+0000", 2023-11-05T08:33:51.057 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1064999999999996e-05 2023-11-05T08:33:51.057 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.058 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.058 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:51.058 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174478+0000", 2023-11-05T08:33:51.058 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.2663000000000005e-05 2023-11-05T08:33:51.058 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.059 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:51.059 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.059 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.059 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.059 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:51.060 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.151886+0000", 2023-11-05T08:33:51.060 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 81.836848271999997, 2023-11-05T08:33:51.060 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.023174162000000002, 2023-11-05T08:33:51.060 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:51.060 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:51.061 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:51.061 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:51.061 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:51.061 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 117 2023-11-05T08:33:51.061 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.062 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:51.062 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.062 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:51.062 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151886+0000", 2023-11-05T08:33:51.062 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:51.063 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.063 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.063 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:51.063 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151886+0000", 2023-11-05T08:33:51.063 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:51.064 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.064 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.064 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:51.064 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151887+0000", 2023-11-05T08:33:51.064 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.2300000000000001e-07 2023-11-05T08:33:51.064 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.065 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.065 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:51.065 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151893+0000", 2023-11-05T08:33:51.065 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.19e-06 2023-11-05T08:33:51.065 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.066 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.066 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:51.066 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151905+0000", 2023-11-05T08:33:51.066 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1883e-05 2023-11-05T08:33:51.066 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.067 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.067 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:51.067 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151912+0000", 2023-11-05T08:33:51.067 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.0690000000000004e-06 2023-11-05T08:33:51.067 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.068 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.068 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:51.068 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.170509+0000", 2023-11-05T08:33:51.068 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.018596686000000001 2023-11-05T08:33:51.068 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.069 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.069 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:51.069 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.170791+0000", 2023-11-05T08:33:51.069 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00028221799999999999 2023-11-05T08:33:51.069 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.070 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.070 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:51.070 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.171137+0000", 2023-11-05T08:33:51.070 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000346454 2023-11-05T08:33:51.070 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.071 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.071 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:51.071 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.172413+0000", 2023-11-05T08:33:51.071 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0012752340000000001 2023-11-05T08:33:51.071 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.072 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.072 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:51.072 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174926+0000", 2023-11-05T08:33:51.072 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0025137050000000002 2023-11-05T08:33:51.072 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.073 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.073 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:51.073 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.175007+0000", 2023-11-05T08:33:51.073 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.0413000000000004e-05 2023-11-05T08:33:51.073 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.073 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.074 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:51.074 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.175060+0000", 2023-11-05T08:33:51.074 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3687000000000003e-05 2023-11-05T08:33:51.074 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.074 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:51.075 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.075 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.075 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.075 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:163 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:51.075 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.670890+0000", 2023-11-05T08:33:51.076 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 81.317844042999994, 2023-11-05T08:33:51.076 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.02132307, 2023-11-05T08:33:51.076 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:51.076 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:51.076 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:51.077 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:51.077 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:51.077 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 163 2023-11-05T08:33:51.077 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.077 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:51.078 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.078 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:51.078 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670890+0000", 2023-11-05T08:33:51.078 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:51.078 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.079 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.079 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:51.079 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670890+0000", 2023-11-05T08:33:51.079 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:51.079 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.080 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.080 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:51.080 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670891+0000", 2023-11-05T08:33:51.080 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3300000000000002e-07 2023-11-05T08:33:51.080 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.080 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.081 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:51.081 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670897+0000", 2023-11-05T08:33:51.081 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.4600000000000002e-06 2023-11-05T08:33:51.081 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.081 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.082 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:51.082 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670906+0000", 2023-11-05T08:33:51.082 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.3130000000000004e-06 2023-11-05T08:33:51.082 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.082 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.083 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:51.083 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670913+0000", 2023-11-05T08:33:51.083 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9859999999999997e-06 2023-11-05T08:33:51.083 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.083 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.083 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:51.084 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.687360+0000", 2023-11-05T08:33:51.084 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.016447244 2023-11-05T08:33:51.084 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.084 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.084 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:51.085 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.687588+0000", 2023-11-05T08:33:51.085 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00022757699999999999 2023-11-05T08:33:51.085 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.085 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.085 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:51.086 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.687946+0000", 2023-11-05T08:33:51.086 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00035789399999999999 2023-11-05T08:33:51.086 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.086 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.086 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:51.087 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.691929+0000", 2023-11-05T08:33:51.087 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0039835690000000002 2023-11-05T08:33:51.087 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.087 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.087 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:51.088 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.691970+0000", 2023-11-05T08:33:51.088 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.0351e-05 2023-11-05T08:33:51.088 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.088 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.088 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:51.088 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692111+0000", 2023-11-05T08:33:51.089 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014114499999999999 2023-11-05T08:33:51.089 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.089 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.089 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:51.089 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692214+0000", 2023-11-05T08:33:51.090 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000102898 2023-11-05T08:33:51.090 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.090 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:51.090 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.090 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.091 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.091 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:51.091 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.670924+0000", 2023-11-05T08:33:51.091 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 81.317810635000001, 2023-11-05T08:33:51.091 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.021621676999999999, 2023-11-05T08:33:51.092 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:51.092 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:51.092 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:51.092 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:51.092 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:51.093 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 164 2023-11-05T08:33:51.093 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.093 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:51.093 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.093 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:51.094 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670924+0000", 2023-11-05T08:33:51.094 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:51.094 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.094 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.094 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:51.095 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670924+0000", 2023-11-05T08:33:51.095 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:51.095 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.095 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.095 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:51.096 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670925+0000", 2023-11-05T08:33:51.096 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.8299999999999996e-07 2023-11-05T08:33:51.096 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.096 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.096 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:51.097 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671022+0000", 2023-11-05T08:33:51.097 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.7445e-05 2023-11-05T08:33:51.097 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.097 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.097 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:51.098 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671033+0000", 2023-11-05T08:33:51.098 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1206000000000001e-05 2023-11-05T08:33:51.098 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.098 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.098 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:51.099 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671045+0000", 2023-11-05T08:33:51.099 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1797000000000001e-05 2023-11-05T08:33:51.099 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.099 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.099 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:51.099 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.688874+0000", 2023-11-05T08:33:51.100 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017829424 2023-11-05T08:33:51.100 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.100 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.100 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:51.100 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.689097+0000", 2023-11-05T08:33:51.101 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000222828 2023-11-05T08:33:51.101 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.101 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.101 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:51.101 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.689444+0000", 2023-11-05T08:33:51.102 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00034672099999999998 2023-11-05T08:33:51.102 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.102 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.102 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:51.102 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692240+0000", 2023-11-05T08:33:51.103 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0027959030000000002 2023-11-05T08:33:51.103 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.103 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.103 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:51.103 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692339+0000", 2023-11-05T08:33:51.103 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.8801000000000005e-05 2023-11-05T08:33:51.104 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.104 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.104 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:51.104 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692435+0000", 2023-11-05T08:33:51.104 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6311999999999995e-05 2023-11-05T08:33:51.105 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.105 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.105 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:51.105 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692546+0000", 2023-11-05T08:33:51.105 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00011055700000000001 2023-11-05T08:33:51.106 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.106 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:51.106 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.106 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.106 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.107 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:51.107 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.671060+0000", 2023-11-05T08:33:51.107 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 81.317674968999995, 2023-11-05T08:33:51.107 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022080292000000001, 2023-11-05T08:33:51.107 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:51.108 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:51.108 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:51.108 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:51.108 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:51.108 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 165 2023-11-05T08:33:51.109 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.109 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:51.109 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.109 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:51.110 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671060+0000", 2023-11-05T08:33:51.110 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:51.110 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.110 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.110 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:51.111 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671060+0000", 2023-11-05T08:33:51.111 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:51.111 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.111 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.111 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:51.112 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671060+0000", 2023-11-05T08:33:51.112 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.1699999999999997e-07 2023-11-05T08:33:51.112 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.112 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.112 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:51.113 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671067+0000", 2023-11-05T08:33:51.113 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5690000000000001e-06 2023-11-05T08:33:51.113 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.113 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.113 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:51.113 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671076+0000", 2023-11-05T08:33:51.114 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.5270000000000004e-06 2023-11-05T08:33:51.114 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.114 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.114 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:51.114 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671084+0000", 2023-11-05T08:33:51.115 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.3459999999999996e-06 2023-11-05T08:33:51.115 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.115 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.115 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:51.115 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.690832+0000", 2023-11-05T08:33:51.116 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.019747793 2023-11-05T08:33:51.116 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.116 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.116 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:51.116 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.690971+0000", 2023-11-05T08:33:51.117 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000139245 2023-11-05T08:33:51.117 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.117 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.117 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:51.117 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.691178+0000", 2023-11-05T08:33:51.118 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000206892 2023-11-05T08:33:51.118 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.118 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.118 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:51.118 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692704+0000", 2023-11-05T08:33:51.119 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00152655 2023-11-05T08:33:51.119 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.119 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.119 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:51.119 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692904+0000", 2023-11-05T08:33:51.119 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000199376 2023-11-05T08:33:51.120 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.120 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.120 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:51.120 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.693044+0000", 2023-11-05T08:33:51.120 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013989199999999999 2023-11-05T08:33:51.121 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.121 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.121 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:51.121 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.693140+0000", 2023-11-05T08:33:51.121 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6285000000000004e-05 2023-11-05T08:33:51.122 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.122 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:51.122 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.122 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.122 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.123 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:184 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:51.123 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.866152+0000", 2023-11-05T08:33:51.123 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 81.122582694000002, 2023-11-05T08:33:51.123 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.019116587000000001, 2023-11-05T08:33:51.123 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:51.123 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:51.124 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:51.124 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:51.124 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:51.124 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 184 2023-11-05T08:33:51.124 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.125 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:51.125 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.125 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:51.125 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866152+0000", 2023-11-05T08:33:51.125 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:51.125 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.126 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.126 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:51.126 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866152+0000", 2023-11-05T08:33:51.126 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:51.126 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.126 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.127 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:51.127 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866152+0000", 2023-11-05T08:33:51.127 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4600000000000004e-07 2023-11-05T08:33:51.127 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.127 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.128 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:51.128 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866158+0000", 2023-11-05T08:33:51.128 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5629999999999999e-06 2023-11-05T08:33:51.128 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.128 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.128 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:51.129 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866167+0000", 2023-11-05T08:33:51.129 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.2229999999999996e-06 2023-11-05T08:33:51.129 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.129 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.129 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:51.130 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866175+0000", 2023-11-05T08:33:51.130 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.3699999999999997e-06 2023-11-05T08:33:51.130 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.130 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.130 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:51.131 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.878285+0000", 2023-11-05T08:33:51.131 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012110164 2023-11-05T08:33:51.131 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.131 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.131 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:51.132 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.878584+0000", 2023-11-05T08:33:51.132 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00029961300000000001 2023-11-05T08:33:51.132 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.132 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.132 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:51.132 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.878942+0000", 2023-11-05T08:33:51.133 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00035795100000000002 2023-11-05T08:33:51.133 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.133 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.133 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:51.133 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.883845+0000", 2023-11-05T08:33:51.134 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0049021450000000001 2023-11-05T08:33:51.134 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.134 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.134 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:51.134 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885018+0000", 2023-11-05T08:33:51.135 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0011738160000000001 2023-11-05T08:33:51.135 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.135 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.135 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:51.135 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885167+0000", 2023-11-05T08:33:51.135 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014895800000000001 2023-11-05T08:33:51.136 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.136 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.136 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:51.136 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885268+0000", 2023-11-05T08:33:51.136 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010113800000000001 2023-11-05T08:33:51.137 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.137 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:51.137 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.137 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.137 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.138 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:185 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:51.138 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.866212+0000", 2023-11-05T08:33:51.138 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 81.122522974000006, 2023-11-05T08:33:51.138 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.019448929, 2023-11-05T08:33:51.138 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:51.139 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:51.139 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:51.139 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:51.139 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:51.139 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 185 2023-11-05T08:33:51.140 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.140 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:51.140 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.140 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:51.140 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866212+0000", 2023-11-05T08:33:51.140 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:51.141 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.141 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.141 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:51.141 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866212+0000", 2023-11-05T08:33:51.141 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:51.142 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.142 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.142 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:51.142 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866213+0000", 2023-11-05T08:33:51.142 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0559999999999999e-06 2023-11-05T08:33:51.143 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.143 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.143 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:51.143 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866308+0000", 2023-11-05T08:33:51.143 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.4984999999999999e-05 2023-11-05T08:33:51.143 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.144 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.144 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:51.144 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866320+0000", 2023-11-05T08:33:51.144 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2639e-05 2023-11-05T08:33:51.144 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.145 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.145 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:51.145 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866328+0000", 2023-11-05T08:33:51.145 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.1240000000000005e-06 2023-11-05T08:33:51.145 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.146 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.146 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:51.146 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.879899+0000", 2023-11-05T08:33:51.146 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.013571059999999999 2023-11-05T08:33:51.146 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.147 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.147 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:51.147 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.880034+0000", 2023-11-05T08:33:51.147 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000134676 2023-11-05T08:33:51.147 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.148 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.148 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:51.148 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.880258+0000", 2023-11-05T08:33:51.148 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00022358100000000001 2023-11-05T08:33:51.148 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.148 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.149 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:51.149 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885299+0000", 2023-11-05T08:33:51.149 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.005041678 2023-11-05T08:33:51.149 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.149 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.150 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:51.150 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885436+0000", 2023-11-05T08:33:51.150 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000136472 2023-11-05T08:33:51.150 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.150 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.151 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:51.151 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885557+0000", 2023-11-05T08:33:51.151 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000121644 2023-11-05T08:33:51.151 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.151 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.152 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:51.152 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885660+0000", 2023-11-05T08:33:51.152 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000103014 2023-11-05T08:33:51.152 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.152 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:51.153 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.153 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.153 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.153 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:186 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:51.153 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.866343+0000", 2023-11-05T08:33:51.154 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 81.122391031000006, 2023-11-05T08:33:51.154 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.021691589000000001, 2023-11-05T08:33:51.154 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:51.154 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:51.154 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:51.155 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:51.155 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:51.155 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 186 2023-11-05T08:33:51.155 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.155 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:51.155 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.156 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:51.156 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866343+0000", 2023-11-05T08:33:51.156 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:51.156 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.156 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.157 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:51.157 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866343+0000", 2023-11-05T08:33:51.157 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:51.157 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.157 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.158 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:51.158 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866344+0000", 2023-11-05T08:33:51.158 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.6000000000000003e-07 2023-11-05T08:33:51.158 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.158 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.159 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:51.159 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866351+0000", 2023-11-05T08:33:51.159 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3090000000000001e-06 2023-11-05T08:33:51.159 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.159 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.160 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:51.160 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866360+0000", 2023-11-05T08:33:51.160 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.2599999999999994e-06 2023-11-05T08:33:51.160 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.160 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.161 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:51.161 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866367+0000", 2023-11-05T08:33:51.161 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.7599999999999997e-06 2023-11-05T08:33:51.161 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.161 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.162 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:51.162 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.882269+0000", 2023-11-05T08:33:51.162 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.015902218999999999 2023-11-05T08:33:51.162 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.162 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.163 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:51.163 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.882545+0000", 2023-11-05T08:33:51.163 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00027617199999999998 2023-11-05T08:33:51.163 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.163 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.163 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:51.164 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.882925+0000", 2023-11-05T08:33:51.164 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00037995900000000003 2023-11-05T08:33:51.164 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.164 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.164 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:51.165 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.887718+0000", 2023-11-05T08:33:51.165 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0047935069999999998 2023-11-05T08:33:51.165 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.165 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.165 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:51.166 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.887836+0000", 2023-11-05T08:33:51.166 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000117401 2023-11-05T08:33:51.166 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.166 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.166 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:51.167 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.887931+0000", 2023-11-05T08:33:51.167 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.4802000000000005e-05 2023-11-05T08:33:51.167 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.167 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.167 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:51.167 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888035+0000", 2023-11-05T08:33:51.168 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010444 2023-11-05T08:33:51.168 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.168 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:51.168 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.168 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.169 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.169 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:187 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:51.169 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.866377+0000", 2023-11-05T08:33:51.169 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 81.122357070000007, 2023-11-05T08:33:51.169 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022216751, 2023-11-05T08:33:51.170 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:51.170 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:51.170 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:51.170 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:51.170 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:51.171 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 187 2023-11-05T08:33:51.171 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.171 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:51.171 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.171 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:51.171 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866377+0000", 2023-11-05T08:33:51.172 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:51.172 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.172 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.172 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:51.172 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866377+0000", 2023-11-05T08:33:51.173 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:51.173 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.173 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.173 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:51.173 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866378+0000", 2023-11-05T08:33:51.173 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9999999999999997e-07 2023-11-05T08:33:51.174 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.174 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.174 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:51.174 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866384+0000", 2023-11-05T08:33:51.174 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.4099999999999999e-06 2023-11-05T08:33:51.175 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.175 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.175 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:51.175 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866393+0000", 2023-11-05T08:33:51.175 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9719999999999998e-06 2023-11-05T08:33:51.176 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.176 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.176 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:51.176 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866400+0000", 2023-11-05T08:33:51.176 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.2740000000000002e-06 2023-11-05T08:33:51.177 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.177 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.177 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:51.177 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.883936+0000", 2023-11-05T08:33:51.177 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017536513 2023-11-05T08:33:51.178 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.178 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.178 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:51.178 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.884061+0000", 2023-11-05T08:33:51.178 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012435300000000001 2023-11-05T08:33:51.179 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.179 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.179 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:51.179 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.884277+0000", 2023-11-05T08:33:51.179 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00021665500000000001 2023-11-05T08:33:51.180 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.180 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.180 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:51.180 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888193+0000", 2023-11-05T08:33:51.180 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0039160319999999998 2023-11-05T08:33:51.181 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.181 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.181 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:51.181 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888235+0000", 2023-11-05T08:33:51.181 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1267999999999999e-05 2023-11-05T08:33:51.181 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.182 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.182 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:51.182 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888373+0000", 2023-11-05T08:33:51.182 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000138789 2023-11-05T08:33:51.182 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.183 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.183 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:51.183 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888594+0000", 2023-11-05T08:33:51.183 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00022078500000000001 2023-11-05T08:33:51.183 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.184 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:51.184 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.184 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.184 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.184 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:188 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:51.185 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.866411+0000", 2023-11-05T08:33:51.185 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 81.122323808999994, 2023-11-05T08:33:51.185 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.023678187999999999, 2023-11-05T08:33:51.185 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:51.185 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:51.186 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:51.186 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:51.186 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:51.186 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 188 2023-11-05T08:33:51.186 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.187 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:51.187 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.187 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:51.187 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866411+0000", 2023-11-05T08:33:51.187 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:51.187 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.188 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.188 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:51.188 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866411+0000", 2023-11-05T08:33:51.188 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:51.188 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.189 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.189 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:51.189 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866411+0000", 2023-11-05T08:33:51.189 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.44e-07 2023-11-05T08:33:51.189 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.189 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.190 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:51.190 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866531+0000", 2023-11-05T08:33:51.190 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000119664 2023-11-05T08:33:51.190 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.190 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.191 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:51.191 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866545+0000", 2023-11-05T08:33:51.191 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3875e-05 2023-11-05T08:33:51.191 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.191 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.192 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:51.192 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866554+0000", 2023-11-05T08:33:51.192 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9099999999999994e-06 2023-11-05T08:33:51.192 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.192 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.193 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:51.193 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885784+0000", 2023-11-05T08:33:51.193 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.019230651000000001 2023-11-05T08:33:51.193 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.193 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.194 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:51.194 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.886020+0000", 2023-11-05T08:33:51.194 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00023538300000000001 2023-11-05T08:33:51.194 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.194 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.195 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:51.195 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.886386+0000", 2023-11-05T08:33:51.195 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00036659399999999998 2023-11-05T08:33:51.195 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.195 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.196 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:51.196 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888662+0000", 2023-11-05T08:33:51.196 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0022756019999999998 2023-11-05T08:33:51.196 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.196 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.197 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:51.197 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.889922+0000", 2023-11-05T08:33:51.197 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0012603149999999999 2023-11-05T08:33:51.197 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.197 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.198 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:51.198 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.890034+0000", 2023-11-05T08:33:51.198 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000111283 2023-11-05T08:33:51.198 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.198 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.199 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:51.199 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.890089+0000", 2023-11-05T08:33:51.199 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5266999999999998e-05 2023-11-05T08:33:51.199 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.199 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:51.200 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.200 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.200 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.200 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:189 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:51.200 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.866572+0000", 2023-11-05T08:33:51.201 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 81.122162850999999, 2023-11-05T08:33:51.201 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.024895937999999999, 2023-11-05T08:33:51.201 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:51.201 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:51.201 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:51.201 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:51.202 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:51.202 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 189 2023-11-05T08:33:51.202 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.202 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:51.203 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.203 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:51.203 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866572+0000", 2023-11-05T08:33:51.203 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:51.203 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.204 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.204 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:51.204 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866572+0000", 2023-11-05T08:33:51.204 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:51.204 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.205 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.205 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:51.205 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866572+0000", 2023-11-05T08:33:51.205 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.8000000000000005e-07 2023-11-05T08:33:51.205 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.206 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.206 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:51.206 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866581+0000", 2023-11-05T08:33:51.206 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9800000000000004e-06 2023-11-05T08:33:51.206 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.207 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.207 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:51.207 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866591+0000", 2023-11-05T08:33:51.207 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6059999999999992e-06 2023-11-05T08:33:51.207 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.207 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.208 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:51.208 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866598+0000", 2023-11-05T08:33:51.208 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.1029999999999996e-06 2023-11-05T08:33:51.208 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.208 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.209 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:51.209 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888850+0000", 2023-11-05T08:33:51.209 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022251615999999998 2023-11-05T08:33:51.209 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.209 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.210 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:51.210 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888988+0000", 2023-11-05T08:33:51.210 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000138436 2023-11-05T08:33:51.210 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.210 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.211 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:51.211 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.889206+0000", 2023-11-05T08:33:51.211 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00021818099999999999 2023-11-05T08:33:51.211 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.211 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.212 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:51.212 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.890621+0000", 2023-11-05T08:33:51.212 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001414219 2023-11-05T08:33:51.212 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.212 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.212 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:51.213 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.891316+0000", 2023-11-05T08:33:51.213 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00069570500000000002 2023-11-05T08:33:51.213 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.213 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.213 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:51.214 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.891406+0000", 2023-11-05T08:33:51.214 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9901999999999994e-05 2023-11-05T08:33:51.214 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.214 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.214 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:51.215 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.891468+0000", 2023-11-05T08:33:51.215 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1409999999999996e-05 2023-11-05T08:33:51.215 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.215 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:51.215 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.216 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.216 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.216 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:292 3.2s0 3:41aa5db9:::smithi09039252-7:head [delete] snapc 8=[8,7,6,5] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e41)", 2023-11-05T08:33:51.216 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.121023+0000", 2023-11-05T08:33:51.216 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 56.867711436, 2023-11-05T08:33:51.217 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.061679772000000001, 2023-11-05T08:33:51.217 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:51.217 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:51.217 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:51.217 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:51.218 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:51.218 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 292 2023-11-05T08:33:51.218 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.218 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:51.218 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.219 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:51.219 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.121023+0000", 2023-11-05T08:33:51.219 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:51.219 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.219 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.219 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:51.220 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.121023+0000", 2023-11-05T08:33:51.220 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:51.220 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.220 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.221 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:51.221 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.121026+0000", 2023-11-05T08:33:51.221 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0369999999999999e-06 2023-11-05T08:33:51.221 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.221 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.221 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:51.222 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.121039+0000", 2023-11-05T08:33:51.222 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3103e-05 2023-11-05T08:33:51.222 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.222 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.222 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:51.223 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.121086+0000", 2023-11-05T08:33:51.223 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7095000000000002e-05 2023-11-05T08:33:51.223 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.223 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.223 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:51.224 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.121131+0000", 2023-11-05T08:33:51.224 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4455e-05 2023-11-05T08:33:51.224 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.224 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.224 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:51.225 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.129136+0000", 2023-11-05T08:33:51.225 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0080047439999999994 2023-11-05T08:33:51.225 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.225 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.225 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2023-11-05T08:33:51.225 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.129142+0000", 2023-11-05T08:33:51.226 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3759999999999996e-06 2023-11-05T08:33:51.226 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.226 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.226 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:51.226 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.158095+0000", 2023-11-05T08:33:51.227 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.028953402 2023-11-05T08:33:51.227 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.227 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.227 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:51.227 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.158534+0000", 2023-11-05T08:33:51.228 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00043911800000000001 2023-11-05T08:33:51.228 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.228 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.228 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:51.228 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.160380+0000", 2023-11-05T08:33:51.229 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001845923 2023-11-05T08:33:51.229 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.229 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.229 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:51.229 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.177604+0000", 2023-11-05T08:33:51.229 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017223405000000001 2023-11-05T08:33:51.230 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.230 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.230 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:51.230 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.182556+0000", 2023-11-05T08:33:51.230 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0049522740000000004 2023-11-05T08:33:51.231 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.231 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.231 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:51.231 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.182703+0000", 2023-11-05T08:33:51.231 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014684 2023-11-05T08:33:51.231 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.232 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:51.232 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.232 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.232 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.232 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:335 3.18s0 3:1aa38538:::smithi09039252-11:head [stat,delete] snapc 10=[10,7] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2023-11-05T08:33:51.233 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:04.225103+0000", 2023-11-05T08:33:51.233 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 46.763631562, 2023-11-05T08:33:51.233 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.028941109999999999, 2023-11-05T08:33:51.233 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:51.233 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:51.233 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:51.234 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:51.234 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:51.234 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 335 2023-11-05T08:33:51.234 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.234 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:51.235 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.235 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:51.235 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.225103+0000", 2023-11-05T08:33:51.235 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:51.235 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.235 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.236 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:51.236 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.225103+0000", 2023-11-05T08:33:51.236 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:51.236 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.236 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.237 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:51.237 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.225106+0000", 2023-11-05T08:33:51.237 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0929999999999999e-06 2023-11-05T08:33:51.237 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.237 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.238 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:51.238 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.225118+0000", 2023-11-05T08:33:51.238 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2017e-05 2023-11-05T08:33:51.238 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.238 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.239 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:51.239 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.225155+0000", 2023-11-05T08:33:51.239 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.6711999999999997e-05 2023-11-05T08:33:51.239 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.239 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.239 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:51.240 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.225172+0000", 2023-11-05T08:33:51.240 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7367000000000002e-05 2023-11-05T08:33:51.240 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.240 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.240 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:51.241 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.225312+0000", 2023-11-05T08:33:51.241 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000140323 2023-11-05T08:33:51.241 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.241 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.241 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op must wait for map", 2023-11-05T08:33:51.241 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.225355+0000", 2023-11-05T08:33:51.242 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.2746e-05 2023-11-05T08:33:51.242 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.242 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.242 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:51.242 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.237049+0000", 2023-11-05T08:33:51.243 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011693360999999999 2023-11-05T08:33:51.243 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.243 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.243 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:51.243 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.237652+0000", 2023-11-05T08:33:51.244 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000603749 2023-11-05T08:33:51.244 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.244 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.244 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:51.244 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.245013+0000", 2023-11-05T08:33:51.245 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0073607000000000004 2023-11-05T08:33:51.245 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.245 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.245 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:51.245 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.249533+0000", 2023-11-05T08:33:51.245 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0045203939999999996 2023-11-05T08:33:51.246 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.246 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.246 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:51.246 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.253639+0000", 2023-11-05T08:33:51.246 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0041051730000000002 2023-11-05T08:33:51.247 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.247 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.247 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:51.247 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.254044+0000", 2023-11-05T08:33:51.247 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00040547499999999998 2023-11-05T08:33:51.248 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.248 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:51.248 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.248 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.248 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.248 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(osd.5.22:1 3.10s0 3:084c43b8:::smithi09039252-5:head [assert-version v10,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:51.249 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:05.258544+0000", 2023-11-05T08:33:51.249 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 45.730190876000002, 2023-11-05T08:33:51.249 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.024845459, 2023-11-05T08:33:51.249 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:51.249 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2023-11-05T08:33:51.250 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:51.250 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.250 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:51.250 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258544+0000", 2023-11-05T08:33:51.250 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:51.251 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.251 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.251 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:51.251 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258544+0000", 2023-11-05T08:33:51.251 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:51.251 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.252 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.252 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:51.252 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258547+0000", 2023-11-05T08:33:51.252 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0740000000000001e-06 2023-11-05T08:33:51.252 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.253 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.253 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:51.253 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258561+0000", 2023-11-05T08:33:51.253 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4525999999999999e-05 2023-11-05T08:33:51.253 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.253 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.254 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:51.254 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258601+0000", 2023-11-05T08:33:51.254 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.9433e-05 2023-11-05T08:33:51.254 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.254 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.255 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:51.255 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258625+0000", 2023-11-05T08:33:51.255 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3819e-05 2023-11-05T08:33:51.255 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.255 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.256 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:51.256 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258788+0000", 2023-11-05T08:33:51.256 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00016302600000000001 2023-11-05T08:33:51.256 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.256 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.256 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op must wait for map", 2023-11-05T08:33:51.257 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258812+0000", 2023-11-05T08:33:51.257 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4043e-05 2023-11-05T08:33:51.257 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.257 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.258 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:51.258 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.275660+0000", 2023-11-05T08:33:51.258 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.016848823999999998 2023-11-05T08:33:51.258 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.258 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.258 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:51.259 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.276035+0000", 2023-11-05T08:33:51.259 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00037483200000000002 2023-11-05T08:33:51.259 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.259 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.259 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:51.260 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.283389+0000", 2023-11-05T08:33:51.260 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.007353882 2023-11-05T08:33:51.260 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.260 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:51.260 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.261 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.261 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.261 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:351 3.18s0 3:1aa38538:::smithi09039252-11:head [append 0~491520] snapc 10=[7] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:51.261 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:05.264529+0000", 2023-11-05T08:33:51.261 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 45.724205648000002, 2023-11-05T08:33:51.262 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.029015691, 2023-11-05T08:33:51.262 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:51.262 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:51.262 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:51.262 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:51.263 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:51.263 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 351 2023-11-05T08:33:51.263 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.263 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:51.263 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.263 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:51.264 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.264529+0000", 2023-11-05T08:33:51.264 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:51.264 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.264 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.264 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:51.265 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.264529+0000", 2023-11-05T08:33:51.265 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:51.265 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.265 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.265 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:51.266 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.264532+0000", 2023-11-05T08:33:51.266 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3730000000000001e-06 2023-11-05T08:33:51.266 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.266 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.266 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:51.267 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.266714+0000", 2023-11-05T08:33:51.267 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0021819880000000002 2023-11-05T08:33:51.267 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.267 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.267 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:51.267 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.266743+0000", 2023-11-05T08:33:51.268 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9218999999999999e-05 2023-11-05T08:33:51.268 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.268 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.268 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:51.268 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.266759+0000", 2023-11-05T08:33:51.269 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5102999999999999e-05 2023-11-05T08:33:51.269 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.269 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.269 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:51.269 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.266928+0000", 2023-11-05T08:33:51.270 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00016909299999999999 2023-11-05T08:33:51.270 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.270 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.270 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op must wait for map", 2023-11-05T08:33:51.271 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.266953+0000", 2023-11-05T08:33:51.271 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5344999999999999e-05 2023-11-05T08:33:51.271 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.271 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.271 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:51.271 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.276760+0000", 2023-11-05T08:33:51.272 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0098067109999999992 2023-11-05T08:33:51.272 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.272 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.272 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:51.272 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.277064+0000", 2023-11-05T08:33:51.273 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00030468299999999998 2023-11-05T08:33:51.273 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.273 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.273 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:51.273 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.279923+0000", 2023-11-05T08:33:51.274 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0028589119999999999 2023-11-05T08:33:51.274 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.274 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.274 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:51.274 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.292166+0000", 2023-11-05T08:33:51.275 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012242723 2023-11-05T08:33:51.275 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.275 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.275 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:51.275 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.292949+0000", 2023-11-05T08:33:51.276 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00078316699999999998 2023-11-05T08:33:51.276 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.276 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.276 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:51.276 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.293545+0000", 2023-11-05T08:33:51.276 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00059537399999999997 2023-11-05T08:33:51.277 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.277 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:51.277 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.277 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.277 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.278 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:354 3.18s0 3:1aa38538:::smithi09039252-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:51.278 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:05.281130+0000", 2023-11-05T08:33:51.278 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 45.707604054999997, 2023-11-05T08:33:51.278 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.021364054, 2023-11-05T08:33:51.278 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:51.279 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2023-11-05T08:33:51.279 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:51.279 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:51.279 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:51.279 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 354 2023-11-05T08:33:51.280 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.280 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:51.280 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.280 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:51.280 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.281130+0000", 2023-11-05T08:33:51.281 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:51.281 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.281 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.281 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:51.281 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.281130+0000", 2023-11-05T08:33:51.281 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:51.282 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.282 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.282 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:51.282 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.281131+0000", 2023-11-05T08:33:51.282 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.0299999999999998e-07 2023-11-05T08:33:51.283 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.283 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.283 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:51.283 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.281135+0000", 2023-11-05T08:33:51.283 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2799999999999999e-06 2023-11-05T08:33:51.284 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.284 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.284 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:51.284 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.281146+0000", 2023-11-05T08:33:51.284 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.119e-05 2023-11-05T08:33:51.285 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.285 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.285 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:51.285 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.281154+0000", 2023-11-05T08:33:51.285 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.7719999999999994e-06 2023-11-05T08:33:51.285 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.286 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.286 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:51.286 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.298629+0000", 2023-11-05T08:33:51.286 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017475195999999998 2023-11-05T08:33:51.286 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.287 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.287 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:51.287 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.298839+0000", 2023-11-05T08:33:51.287 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00020991799999999999 2023-11-05T08:33:51.287 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.288 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.288 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:51.288 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.302495+0000", 2023-11-05T08:33:51.288 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0036558950000000002 2023-11-05T08:33:51.288 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.289 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:51.289 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.289 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.289 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.289 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:638 3.10s0 3:0a3c6106:::smithi09039252-48:head [create,setxattr TahZbq^WXaahVeQfQsXqNXgcd (1963) in=25b,setxattr `Pj`q[kZqXoPYYj_`hNV^bohgce[Ull^lRXTctln (982) in=40b,setxattr ^gfWYcnTRmaWZQkrf`asZ\\XVPcRmRdRckrhUheci (10312) in=40b,setxattr hVm[Q^c]jmYfcedO_eoe[N_fohZ^nghUPgNQYY`h (15648) in=40b,setxattr \\jlYtrYeh_rthhUaqacc]`Uger\\]rPngbjPhbaWV (454) in=40b,setxattr nOgORnqOheegXnS]tSkTQTreXqqt`ojnVPsQbabf (771) in=40b,setxattr QR]c[o[]SToV\\goc\\pjtRRf`WYSngR[eQ`taQgcc (12488) in=40b,setxattr pY\\tNq\\a[n\\fajnSVYNng[\\kcqZ^VhqNmqRXUTlO (1121) in=40b,setxattr [laaiUa\\cY`kdXP\\SNRq[ig]rcanlRgnTtdfZi_W (1403) in=40b,setxattr aaf[PNUURnfSbT^fR_ge[lfbq]keQjYgrbO\\SbTa (1226) in=40b,setxattr SfaWR`YpWpbnn]sR]^eO[Wbi\\Se_TkrcQp[pcomR (4768) in=40b,setxattr pS^PVQeeOqs_S^YVlb^n\\h``m_Q\\prek^_qjcmgR (1121) in=40b,setxattr [aWfmed^jZheSjc^Xmb]TrYaU[[c]jb\\[UdNQngi (593) in=40b,setxattr kPO^emkeWSaQtnWaRVXk[hbN[TP^]ThcqpOfordV (2805) in=40b,setxattr PkhW`mV]l_O[nUSprX^tRPoRakfUNbOiRedpZb^n (421) in=40b,setxattr [`esP (1963) in=5b,setxattr SlhkjUdadks\\hNrmRohr_fl`itUjb_Vjia\\Vdqf` (10033) in=40b,setxattr TUWTXRdZcQc\\lOTbYmrWiirilafTQQrNdPoj_^sZ (3084) in=40b,setxattr ddbfaqshSg (1963) in=10b,setxattr nSgahmseaQO^hNUcamf\\m[SsQam^Uc``at``i`kU (1435) in=40b] snapc 34=[2f,7] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2023-11-05T08:33:51.290 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:40.606065+0000", 2023-11-05T08:33:51.290 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 10.382669502000001, 2023-11-05T08:33:51.290 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022517651, 2023-11-05T08:33:51.290 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:51.290 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:51.291 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:51.291 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:51.291 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:51.291 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 638 2023-11-05T08:33:51.291 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.292 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:51.292 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.292 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:51.292 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:40.606065+0000", 2023-11-05T08:33:51.292 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:51.293 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.293 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.293 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:51.293 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:40.606065+0000", 2023-11-05T08:33:51.293 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:51.294 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.294 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.294 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:51.294 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:40.606068+0000", 2023-11-05T08:33:51.294 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.01e-06 2023-11-05T08:33:51.295 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.295 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.295 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:51.295 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:40.606338+0000", 2023-11-05T08:33:51.295 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00026997799999999998 2023-11-05T08:33:51.296 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.296 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.296 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:51.296 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:40.606371+0000", 2023-11-05T08:33:51.296 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3278999999999998e-05 2023-11-05T08:33:51.296 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.297 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.297 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:51.297 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:40.606388+0000", 2023-11-05T08:33:51.297 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7073000000000001e-05 2023-11-05T08:33:51.297 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.298 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.298 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:51.298 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:40.606499+0000", 2023-11-05T08:33:51.298 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000110962 2023-11-05T08:33:51.298 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.299 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.299 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op must wait for map", 2023-11-05T08:33:51.299 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:40.606515+0000", 2023-11-05T08:33:51.299 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5352000000000001e-05 2023-11-05T08:33:51.299 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.300 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.300 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:51.300 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:40.614281+0000", 2023-11-05T08:33:51.300 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0077663560000000003 2023-11-05T08:33:51.300 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.301 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.301 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:51.301 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:40.614375+0000", 2023-11-05T08:33:51.301 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.4346999999999996e-05 2023-11-05T08:33:51.301 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.301 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.302 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:51.302 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:40.615206+0000", 2023-11-05T08:33:51.302 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00083111000000000003 2023-11-05T08:33:51.302 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.302 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.303 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:51.303 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:40.619735+0000", 2023-11-05T08:33:51.303 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0045284009999999996 2023-11-05T08:33:51.303 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.303 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.304 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:51.304 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:40.628377+0000", 2023-11-05T08:33:51.304 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0086421710000000006 2023-11-05T08:33:51.304 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:51.304 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:51.304 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:51.305 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:40.628583+0000", 2023-11-05T08:33:51.305 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00020561200000000001 2023-11-05T08:33:51.305 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.305 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:51.305 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.306 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:51.306 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:51.306 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:33:51.307 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-11-05T08:33:51.308 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1: {} 2023-11-05T08:33:51.308 INFO:teuthology.orchestra.run.smithi002.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-11-05T08:33:51.309 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2: {} 2023-11-05T08:33:51.309 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-11-05T08:33:51.310 INFO:teuthology.orchestra.run.smithi002.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-11-05T08:33:51.310 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3: {} 2023-11-05T08:33:51.311 INFO:teuthology.orchestra.run.smithi002.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-11-05T08:33:51.311 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4: {} 2023-11-05T08:33:51.312 INFO:teuthology.orchestra.run.smithi002.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-11-05T08:33:51.312 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5: {} 2023-11-05T08:33:51.312 INFO:teuthology.orchestra.run.smithi002.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-11-05T08:33:51.313 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6: {} 2023-11-05T08:33:51.314 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:33:51.306+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:33:51.314 INFO:teuthology.orchestra.run.smithi002.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-11-05T08:33:51.315 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7: {} 2023-11-05T08:33:51.315 INFO:teuthology.orchestra.run.smithi002.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-11-05T08:33:51.315 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8: {} 2023-11-05T08:33:51.316 INFO:teuthology.orchestra.run.smithi002.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-11-05T08:33:51.316 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9: {} 2023-11-05T08:33:51.317 INFO:teuthology.orchestra.run.smithi002.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-11-05T08:33:51.331 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10: {} 2023-11-05T08:33:51.331 INFO:teuthology.orchestra.run.smithi002.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-11-05T08:33:51.353 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11: {} 2023-11-05T08:33:51.354 INFO:teuthology.orchestra.run.smithi002.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-11-05T08:33:51.375 INFO:teuthology.orchestra.run.smithi002.stdout:osd.12: {} 2023-11-05T08:33:51.375 INFO:teuthology.orchestra.run.smithi002.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-11-05T08:33:51.397 INFO:teuthology.orchestra.run.smithi002.stdout:osd.13: {} 2023-11-05T08:33:51.397 INFO:teuthology.orchestra.run.smithi002.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-11-05T08:33:51.408 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:33:51.405+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:33:51.417 INFO:teuthology.orchestra.run.smithi002.stdout:osd.14: {} 2023-11-05T08:33:51.418 INFO:teuthology.orchestra.run.smithi002.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-11-05T08:33:51.441 INFO:teuthology.orchestra.run.smithi002.stdout:osd.15: {} 2023-11-05T08:33:51.442 INFO:teuthology.orchestra.run.smithi002.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-11-05T08:33:51.445 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:51.445 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:51.445 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-05T08:33:51.446 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-05T08:33:51.458 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-11-05T08:33:51.544 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:33:51.541+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:33:51.610 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:51.606+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:51.620 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:51.621 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:51.634 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-11-05T08:33:51.656 INFO:teuthology.orchestra.run.smithi002.stderr:reweighted osd.4 to 1 (10000) 2023-11-05T08:33:51.663 INFO:tasks.rados.rados.0.smithi090.stdout:308: done (0 left) 2023-11-05T08:33:51.663 INFO:tasks.rados.rados.0.smithi090.stdout:309: rollback oid 29 current snap is 32 2023-11-05T08:33:51.663 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 29 to 30 2023-11-05T08:33:51.663 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 13 current snap is 32 2023-11-05T08:33:51.663 INFO:tasks.rados.rados.0.smithi090.stdout:310: seq_num 119 ranges {2932736=458752,3391488=507904,3899392=507904,4407296=458752,4866048=81920} 2023-11-05T08:33:51.687 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2023-11-05T08:33:51.710 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:33:51.706+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:33:51.717 INFO:tasks.rados.rados.0.smithi090.stdout:310: writing smithi09039252-13 from 2932736 to 3391488 tid 1 2023-11-05T08:33:51.725 INFO:tasks.rados.rados.0.smithi090.stdout:310: writing smithi09039252-13 from 3391488 to 3899392 tid 2 2023-11-05T08:33:51.731 INFO:tasks.rados.rados.0.smithi090.stdout:310: writing smithi09039252-13 from 3899392 to 4407296 tid 3 2023-11-05T08:33:51.736 INFO:tasks.rados.rados.0.smithi090.stdout:310: writing smithi09039252-13 from 4407296 to 4866048 tid 4 2023-11-05T08:33:51.737 INFO:tasks.rados.rados.0.smithi090.stdout:310: writing smithi09039252-13 from 4866048 to 4947968 tid 5 2023-11-05T08:33:51.738 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 49 current snap is 32 2023-11-05T08:33:51.738 INFO:tasks.rados.rados.0.smithi090.stdout:309: finishing rollback tid 0 to smithi09039252-29 2023-11-05T08:33:51.738 INFO:tasks.rados.rados.0.smithi090.stdout:311: seq_num 120 ranges {4112384=475136,4587520=655360,5242880=622592,5865472=278528} 2023-11-05T08:33:51.778 INFO:tasks.rados.rados.0.smithi090.stdout:311: writing smithi09039252-49 from 4112384 to 4587520 tid 1 2023-11-05T08:33:51.784 INFO:tasks.rados.rados.0.smithi090.stdout:311: writing smithi09039252-49 from 4587520 to 5242880 tid 2 2023-11-05T08:33:51.789 INFO:tasks.rados.rados.0.smithi090.stdout:311: writing smithi09039252-49 from 5242880 to 5865472 tid 3 2023-11-05T08:33:51.791 INFO:tasks.rados.rados.0.smithi090.stdout:311: writing smithi09039252-49 from 5865472 to 6144000 tid 4 2023-11-05T08:33:51.792 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 8 current snap is 32 2023-11-05T08:33:51.792 INFO:tasks.rados.rados.0.smithi090.stdout:312: seq_num 121 ranges {1245184=491520,1736704=802816,2539520=720896,3260416=32768} 2023-11-05T08:33:51.803 INFO:tasks.rados.rados.0.smithi090.stdout:312: writing smithi09039252-8 from 1245184 to 1736704 tid 1 2023-11-05T08:33:51.808 INFO:tasks.rados.rados.0.smithi090.stdout:312: writing smithi09039252-8 from 1736704 to 2539520 tid 2 2023-11-05T08:33:51.811 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:33:51.805+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:33:51.814 INFO:tasks.rados.rados.0.smithi090.stdout:312: writing smithi09039252-8 from 2539520 to 3260416 tid 3 2023-11-05T08:33:51.814 INFO:tasks.rados.rados.0.smithi090.stdout:312: writing smithi09039252-8 from 3260416 to 3293184 tid 4 2023-11-05T08:33:51.815 INFO:tasks.rados.rados.0.smithi090.stdout:313: read oid 15 snap -1 2023-11-05T08:33:51.815 INFO:tasks.rados.rados.0.smithi090.stdout:313: expect (ObjNum 17 snap 0 seq_num 17) 2023-11-05T08:33:51.815 INFO:tasks.rados.rados.0.smithi090.stdout:314: rollback oid 40 current snap is 32 2023-11-05T08:33:51.816 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 40 to 30 2023-11-05T08:33:51.816 INFO:tasks.rados.rados.0.smithi090.stdout:309: finishing rollback tid 1 to smithi09039252-29 2023-11-05T08:33:51.816 INFO:tasks.rados.rados.0.smithi090.stdout:309: finishing rollback tid 2 to smithi09039252-29 2023-11-05T08:33:51.816 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 29 v 41 (ObjNum 114 snap 29 seq_num 114) dirty exists 2023-11-05T08:33:51.817 INFO:tasks.rados.rados.0.smithi090.stdout:310: finishing write tid 1 to smithi09039252-13 2023-11-05T08:33:51.817 INFO:tasks.rados.rados.0.smithi090.stdout:310: finishing write tid 2 to smithi09039252-13 2023-11-05T08:33:51.817 INFO:tasks.rados.rados.0.smithi090.stdout:310: finishing write tid 3 to smithi09039252-13 2023-11-05T08:33:51.818 INFO:tasks.rados.rados.0.smithi090.stdout:310: finishing write tid 4 to smithi09039252-13 2023-11-05T08:33:51.818 INFO:tasks.rados.rados.0.smithi090.stdout:310: finishing write tid 5 to smithi09039252-13 2023-11-05T08:33:51.818 INFO:tasks.rados.rados.0.smithi090.stdout:310: finishing write tid 6 to smithi09039252-13 2023-11-05T08:33:51.819 INFO:tasks.rados.rados.0.smithi090.stdout:310: finishing write tid 7 to smithi09039252-13 2023-11-05T08:33:51.819 INFO:tasks.rados.rados.0.smithi090.stdout:310: oid 13 updating version 0 to 46 2023-11-05T08:33:51.819 INFO:tasks.rados.rados.0.smithi090.stdout:310: oid 13 updating version 46 to 47 2023-11-05T08:33:51.819 INFO:tasks.rados.rados.0.smithi090.stdout:310: oid 13 version 47 is already newer than 43 2023-11-05T08:33:51.820 INFO:tasks.rados.rados.0.smithi090.stdout:310: oid 13 version 47 is already newer than 45 2023-11-05T08:33:51.820 INFO:tasks.rados.rados.0.smithi090.stdout:310: oid 13 updating version 47 to 48 2023-11-05T08:33:51.820 INFO:tasks.rados.rados.0.smithi090.stdout:310: oid 13 version 48 is already newer than 44 2023-11-05T08:33:51.820 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 13 v 48 (ObjNum 119 snap 32 seq_num 119) dirty exists 2023-11-05T08:33:51.820 INFO:tasks.rados.rados.0.smithi090.stdout:310: left oid 13 (ObjNum 119 snap 32 seq_num 119) 2023-11-05T08:33:51.821 INFO:tasks.rados.rados.0.smithi090.stdout:311: finishing write tid 1 to smithi09039252-49 2023-11-05T08:33:51.821 INFO:tasks.rados.rados.0.smithi090.stdout:311: finishing write tid 2 to smithi09039252-49 2023-11-05T08:33:51.821 INFO:tasks.rados.rados.0.smithi090.stdout:311: finishing write tid 3 to smithi09039252-49 2023-11-05T08:33:51.821 INFO:tasks.rados.rados.0.smithi090.stdout:311: finishing write tid 4 to smithi09039252-49 2023-11-05T08:33:51.821 INFO:tasks.rados.rados.0.smithi090.stdout:311: finishing write tid 5 to smithi09039252-49 2023-11-05T08:33:51.822 INFO:tasks.rados.rados.0.smithi090.stdout:311: finishing write tid 6 to smithi09039252-49 2023-11-05T08:33:51.822 INFO:tasks.rados.rados.0.smithi090.stdout:311: oid 49 updating version 0 to 52 2023-11-05T08:33:51.822 INFO:tasks.rados.rados.0.smithi090.stdout:311: oid 49 updating version 52 to 54 2023-11-05T08:33:51.822 INFO:tasks.rados.rados.0.smithi090.stdout:311: oid 49 version 54 is already newer than 53 2023-11-05T08:33:51.822 INFO:tasks.rados.rados.0.smithi090.stdout:311: oid 49 version 54 is already newer than 50 2023-11-05T08:33:51.823 INFO:tasks.rados.rados.0.smithi090.stdout:311: oid 49 version 54 is already newer than 51 2023-11-05T08:33:51.823 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 49 v 54 (ObjNum 120 snap 32 seq_num 120) dirty exists 2023-11-05T08:33:51.823 INFO:tasks.rados.rados.0.smithi090.stdout:311: left oid 49 (ObjNum 120 snap 32 seq_num 120) 2023-11-05T08:33:51.823 INFO:tasks.rados.rados.0.smithi090.stdout:309: done (5 left) 2023-11-05T08:33:51.823 INFO:tasks.rados.rados.0.smithi090.stdout:310: done (4 left) 2023-11-05T08:33:51.824 INFO:tasks.rados.rados.0.smithi090.stdout:311: done (3 left) 2023-11-05T08:33:51.824 INFO:tasks.rados.rados.0.smithi090.stdout:315: snap_create 2023-11-05T08:33:51.825 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:51.825 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:51.825 INFO:tasks.rados.rados.0.smithi090.stdout:312: finishing write tid 1 to smithi09039252-8 2023-11-05T08:33:51.826 INFO:tasks.rados.rados.0.smithi090.stdout:312: finishing write tid 2 to smithi09039252-8 2023-11-05T08:33:51.826 INFO:tasks.rados.rados.0.smithi090.stdout:313: expect (ObjNum 17 snap 0 seq_num 17) 2023-11-05T08:33:51.839 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-11-05T08:33:51.842 INFO:tasks.rados.rados.0.smithi090.stdout:312: finishing write tid 3 to smithi09039252-8 2023-11-05T08:33:51.843 INFO:tasks.rados.rados.0.smithi090.stdout:312: finishing write tid 4 to smithi09039252-8 2023-11-05T08:33:51.843 INFO:tasks.rados.rados.0.smithi090.stdout:314: finishing rollback tid 0 to smithi09039252-40 2023-11-05T08:33:51.843 INFO:tasks.rados.rados.0.smithi090.stdout:312: finishing write tid 5 to smithi09039252-8 2023-11-05T08:33:51.843 INFO:tasks.rados.rados.0.smithi090.stdout:314: finishing rollback tid 1 to smithi09039252-40 2023-11-05T08:33:51.844 INFO:tasks.rados.rados.0.smithi090.stdout:314: finishing rollback tid 2 to smithi09039252-40 2023-11-05T08:33:51.844 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 40 v 25 (ObjNum 17 snap 0 seq_num 17) dirty exists 2023-11-05T08:33:51.844 INFO:tasks.rados.rados.0.smithi090.stdout:312: finishing write tid 6 to smithi09039252-8 2023-11-05T08:33:51.844 INFO:tasks.rados.rados.0.smithi090.stdout:312: oid 8 updating version 0 to 112 2023-11-05T08:33:51.845 INFO:tasks.rados.rados.0.smithi090.stdout:312: oid 8 updating version 112 to 113 2023-11-05T08:33:51.845 INFO:tasks.rados.rados.0.smithi090.stdout:312: oid 8 updating version 113 to 114 2023-11-05T08:33:51.845 INFO:tasks.rados.rados.0.smithi090.stdout:312: oid 8 version 114 is already newer than 111 2023-11-05T08:33:51.845 INFO:tasks.rados.rados.0.smithi090.stdout:312: oid 8 version 114 is already newer than 110 2023-11-05T08:33:51.845 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 8 v 114 (ObjNum 121 snap 32 seq_num 121) dirty exists 2023-11-05T08:33:51.846 INFO:tasks.rados.rados.0.smithi090.stdout:312: left oid 8 (ObjNum 121 snap 32 seq_num 121) 2023-11-05T08:33:51.912 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:33:51.909+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:33:52.000 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:52.000 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:52.013 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:33:52.009+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:33:52.015 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-11-05T08:33:52.114 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:33:52.109+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:33:52.140 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:52.140 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:52.154 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-11-05T08:33:52.240 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:33:52.237+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:33:52.312 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:52.313 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:52.314 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-05T08:33:52.326 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2023-11-05T08:33:52.417 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:33:52.416+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:33:52.490 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:52.491 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:52.504 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2023-11-05T08:33:52.517 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:33:52.514+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:33:52.619 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:33:52.617+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:33:52.627 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:52.627 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:52.642 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2023-11-05T08:33:52.686 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-11-05T08:33:52.704 INFO:teuthology.orchestra.run.smithi002.stderr:reweighted osd.5 to 1 (10000) 2023-11-05T08:33:52.710 INFO:tasks.rados.rados.0.smithi090.stdout:312: done (3 left) 2023-11-05T08:33:52.710 INFO:tasks.rados.rados.0.smithi090.stdout:313: done (2 left) 2023-11-05T08:33:52.711 INFO:tasks.rados.rados.0.smithi090.stdout:314: done (1 left) 2023-11-05T08:33:52.711 INFO:tasks.rados.rados.0.smithi090.stdout:315: done (0 left) 2023-11-05T08:33:52.711 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 10 current snap is 33 2023-11-05T08:33:52.711 INFO:tasks.rados.rados.0.smithi090.stdout:316: seq_num 122 ranges {163840=507904,671744=425984,1097728=442368,1540096=540672,2080768=147456} 2023-11-05T08:33:52.715 INFO:tasks.rados.rados.0.smithi090.stdout:316: writing smithi09039252-10 from 163840 to 671744 tid 1 2023-11-05T08:33:52.718 INFO:tasks.rados.rados.0.smithi090.stdout:316: writing smithi09039252-10 from 671744 to 1097728 tid 2 2023-11-05T08:33:52.719 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:33:52.716+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:33:52.721 INFO:tasks.rados.rados.0.smithi090.stdout:316: writing smithi09039252-10 from 1097728 to 1540096 tid 3 2023-11-05T08:33:52.725 INFO:tasks.rados.rados.0.smithi090.stdout:316: writing smithi09039252-10 from 1540096 to 2080768 tid 4 2023-11-05T08:33:52.726 INFO:tasks.rados.rados.0.smithi090.stdout:316: writing smithi09039252-10 from 2080768 to 2228224 tid 5 2023-11-05T08:33:52.726 INFO:tasks.rados.rados.0.smithi090.stdout:317: read oid 27 snap -1 2023-11-05T08:33:52.727 INFO:tasks.rados.rados.0.smithi090.stdout:317: expect deleted 2023-11-05T08:33:52.727 INFO:tasks.rados.rados.0.smithi090.stdout:318: setattr oid 22 current snap is 33 2023-11-05T08:33:52.730 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2023-11-05T08:33:52.731 INFO:tasks.rados.rados.0.smithi090.stdout:319: setattr oid 3 current snap is 33 2023-11-05T08:33:52.736 INFO:tasks.rados.rados.0.smithi090.stdout:316: finishing write tid 1 to smithi09039252-10 2023-11-05T08:33:52.736 INFO:tasks.rados.rados.0.smithi090.stdout:317: done (3 left) 2023-11-05T08:33:52.737 INFO:tasks.rados.rados.0.smithi090.stdout:320: read oid 34 snap -1 2023-11-05T08:33:52.737 INFO:tasks.rados.rados.0.smithi090.stdout:320: expect (ObjNum 60 snap 1 seq_num 60) 2023-11-05T08:33:52.737 INFO:tasks.rados.rados.0.smithi090.stdout:321: snap_create 2023-11-05T08:33:52.737 INFO:tasks.rados.rados.0.smithi090.stdout:316: finishing write tid 2 to smithi09039252-10 2023-11-05T08:33:52.748 INFO:tasks.rados.rados.0.smithi090.stdout:316: finishing write tid 3 to smithi09039252-10 2023-11-05T08:33:52.749 INFO:tasks.rados.rados.0.smithi090.stdout:316: finishing write tid 4 to smithi09039252-10 2023-11-05T08:33:52.749 INFO:tasks.rados.rados.0.smithi090.stdout:316: finishing write tid 5 to smithi09039252-10 2023-11-05T08:33:52.751 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 3 v 56 (ObjNum 32 snap 0 seq_num 32) dirty exists 2023-11-05T08:33:52.752 INFO:tasks.rados.rados.0.smithi090.stdout:316: finishing write tid 6 to smithi09039252-10 2023-11-05T08:33:52.752 INFO:tasks.rados.rados.0.smithi090.stdout:320: expect (ObjNum 60 snap 1 seq_num 60) 2023-11-05T08:33:52.774 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:52.775 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:52.788 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2023-11-05T08:33:52.820 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:33:52.817+0000 7fdc5cd58640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:33:52.835 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 22 v 27 (ObjNum 21 snap 0 seq_num 21) dirty exists 2023-11-05T08:33:52.836 INFO:tasks.rados.rados.0.smithi090.stdout:316: finishing write tid 7 to smithi09039252-10 2023-11-05T08:33:52.836 INFO:tasks.rados.rados.0.smithi090.stdout:316: oid 10 updating version 0 to 31 2023-11-05T08:33:52.836 INFO:tasks.rados.rados.0.smithi090.stdout:316: oid 10 version 31 is already newer than 30 2023-11-05T08:33:52.836 INFO:tasks.rados.rados.0.smithi090.stdout:316: oid 10 updating version 31 to 32 2023-11-05T08:33:52.837 INFO:tasks.rados.rados.0.smithi090.stdout:316: oid 10 version 32 is already newer than 29 2023-11-05T08:33:52.837 INFO:tasks.rados.rados.0.smithi090.stdout:316: oid 10 version 32 is already newer than 27 2023-11-05T08:33:52.837 INFO:tasks.rados.rados.0.smithi090.stdout:316: oid 10 version 32 is already newer than 28 2023-11-05T08:33:52.837 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 10 v 32 (ObjNum 122 snap 33 seq_num 122) dirty exists 2023-11-05T08:33:52.837 INFO:tasks.rados.rados.0.smithi090.stdout:316: left oid 10 (ObjNum 122 snap 33 seq_num 122) 2023-11-05T08:33:52.909 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:52.910 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:52.920 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:52.918+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:52.924 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2023-11-05T08:33:53.021 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:33:53.016+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:33:53.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-05T08:33:53.029+0000 7f6042d9d640 0 --2- 172.21.15.2:0/1793042726 >> v2:172.21.15.69:3300/0 conn(0x7f6044150b90 0x7f6044151040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-11-05T08:33:53.041 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:53.042 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:53.055 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2023-11-05T08:33:53.122 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:33:53.117+0000 7fdc5cd58640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:33:53.173 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:53.173 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:53.187 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-11-05T08:33:53.223 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:33:53.216+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:33:53.316 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:53.317 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:53.324 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:33:53.321+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:33:53.330 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-11-05T08:33:53.425 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:33:53.421+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:33:53.446 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:53.446 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:53.447 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-05T08:33:53.459 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-11-05T08:33:53.527 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-11-05T08:33:53.522+0000 7f72b2c5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 42525) UID: 0 2023-11-05T08:33:53.577 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:53.578 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:53.588 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-11-05T08:33:53.591 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2023-11-05T08:33:53.610 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1: {} 2023-11-05T08:33:53.610 INFO:teuthology.orchestra.run.smithi002.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-11-05T08:33:53.627 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-05T08:33:53.630 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2: {} 2023-11-05T08:33:53.631 INFO:teuthology.orchestra.run.smithi002.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-11-05T08:33:53.651 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3: {} 2023-11-05T08:33:53.651 INFO:teuthology.orchestra.run.smithi002.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-11-05T08:33:53.669 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4: {} 2023-11-05T08:33:53.669 INFO:teuthology.orchestra.run.smithi002.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-11-05T08:33:53.691 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5: {} 2023-11-05T08:33:53.692 INFO:teuthology.orchestra.run.smithi002.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-11-05T08:33:53.714 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6: {} 2023-11-05T08:33:53.714 INFO:teuthology.orchestra.run.smithi002.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-11-05T08:33:53.722 INFO:teuthology.orchestra.run.smithi002.stderr:reweighted osd.6 to 1 (10000) 2023-11-05T08:33:53.722 INFO:teuthology.orchestra.run.smithi002.stderr:noscrub is set 2023-11-05T08:33:53.729 INFO:tasks.rados.rados.0.smithi090.stdout:316: done (4 left) 2023-11-05T08:33:53.729 INFO:tasks.rados.rados.0.smithi090.stdout:318: done (3 left) 2023-11-05T08:33:53.729 INFO:tasks.rados.rados.0.smithi090.stdout:319: done (2 left) 2023-11-05T08:33:53.729 INFO:tasks.rados.rados.0.smithi090.stdout:320: done (1 left) 2023-11-05T08:33:53.730 INFO:tasks.rados.rados.0.smithi090.stdout:321: done (0 left) 2023-11-05T08:33:53.730 INFO:tasks.rados.rados.0.smithi090.stdout:322: copy_from oid 6 from oid 48 current snap is 34 2023-11-05T08:33:53.730 INFO:tasks.rados.rados.0.smithi090.stdout:323: read oid 2 snap -1 2023-11-05T08:33:53.730 INFO:tasks.rados.rados.0.smithi090.stdout:323: expect (ObjNum 117 snap 30 seq_num 117) 2023-11-05T08:33:53.730 INFO:tasks.rados.rados.0.smithi090.stdout:324: snap_remove snap 30 2023-11-05T08:33:53.731 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:53.726+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:53.742 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7: {} 2023-11-05T08:33:53.742 INFO:teuthology.orchestra.run.smithi002.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-11-05T08:33:53.751 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2023-11-05T08:33:53.760 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8: {} 2023-11-05T08:33:53.761 INFO:teuthology.orchestra.run.smithi002.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-11-05T08:33:53.763 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:53.763 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:53.778 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2023-11-05T08:33:53.781 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9: {} 2023-11-05T08:33:53.781 INFO:teuthology.orchestra.run.smithi002.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-11-05T08:33:53.801 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10: {} 2023-11-05T08:33:53.802 INFO:teuthology.orchestra.run.smithi002.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-11-05T08:33:53.821 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11: {} 2023-11-05T08:33:53.822 INFO:teuthology.orchestra.run.smithi002.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-11-05T08:33:53.829 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:33:53.828+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:33:53.838 INFO:teuthology.orchestra.run.smithi002.stdout:osd.12: {} 2023-11-05T08:33:53.839 INFO:teuthology.orchestra.run.smithi002.stderr:osd.12: osd_enable_op_tracker = 'true' 2023-11-05T08:33:53.859 INFO:teuthology.orchestra.run.smithi002.stdout:osd.13: {} 2023-11-05T08:33:53.859 INFO:teuthology.orchestra.run.smithi002.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-11-05T08:33:53.880 INFO:teuthology.orchestra.run.smithi002.stdout:osd.14: {} 2023-11-05T08:33:53.880 INFO:teuthology.orchestra.run.smithi002.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-11-05T08:33:53.881 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-05T08:33:53.877+0000 7f78f6ffd640 0 -- 172.21.15.2:0/552868274 >> [v2:172.21.15.90:6824/739151808,v1:172.21.15.90:6825/739151808] conn(0x7f78f8221d00 msgr2=0x7f78f82240f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-11-05T08:33:53.931 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:33:53.924+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:33:53.938 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:33:53.938 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-11-05T08:33:53.938 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-11-05T08:33:53.938 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-11-05T08:33:53.939 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:33:53.951 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2023-11-05T08:33:54.031 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:33:54.029+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:33:54.071 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:33:54.072 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-11-05T08:33:54.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-11-05T08:33:54.072 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-11-05T08:33:54.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.073 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.as6 42/33 ECSubWrite(tid=38, reqid=osd.7.0:38, at_version=42'16, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:33:54.073 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:55.122849+0000", 2023-11-05T08:33:54.073 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 58.945017104000001, 2023-11-05T08:33:54.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015960385000000001, 2023-11-05T08:33:54.073 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:54.073 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:54.074 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:54.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:54.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.122849+0000", 2023-11-05T08:33:54.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:54.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.122849+0000", 2023-11-05T08:33:54.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:54.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.122850+0000", 2023-11-05T08:33:54.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1699999999999997e-07 2023-11-05T08:33:54.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:54.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.122895+0000", 2023-11-05T08:33:54.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5717999999999998e-05 2023-11-05T08:33:54.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:54.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.122919+0000", 2023-11-05T08:33:54.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3932000000000001e-05 2023-11-05T08:33:54.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:54.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.122941+0000", 2023-11-05T08:33:54.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1708999999999998e-05 2023-11-05T08:33:54.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.125137+0000", 2023-11-05T08:33:54.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021962219999999998 2023-11-05T08:33:54.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:54.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.125161+0000", 2023-11-05T08:33:54.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3436000000000001e-05 2023-11-05T08:33:54.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:54.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.138719+0000", 2023-11-05T08:33:54.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013558115000000001 2023-11-05T08:33:54.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:54.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:55.138809+0000", 2023-11-05T08:33:54.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0536000000000002e-05 2023-11-05T08:33:54.083 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.084 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:54.084 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.084 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:346 3.fs0 3:f2348bf8:::smithi09039252-39:head [read 0~638976,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:54.084 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:05.254913+0000", 2023-11-05T08:33:54.085 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 48.812952727999999, 2023-11-05T08:33:54.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019954645999999999, 2023-11-05T08:33:54.085 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:54.085 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:54.085 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:54.085 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:54.086 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:54.086 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 346 2023-11-05T08:33:54.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.086 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:54.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:54.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.254913+0000", 2023-11-05T08:33:54.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:54.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.254913+0000", 2023-11-05T08:33:54.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:54.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.254917+0000", 2023-11-05T08:33:54.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1330000000000001e-06 2023-11-05T08:33:54.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:54.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.254934+0000", 2023-11-05T08:33:54.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6897000000000001e-05 2023-11-05T08:33:54.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:54.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.254965+0000", 2023-11-05T08:33:54.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0769000000000003e-05 2023-11-05T08:33:54.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:54.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.254981+0000", 2023-11-05T08:33:54.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6015000000000002e-05 2023-11-05T08:33:54.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255140+0000", 2023-11-05T08:33:54.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015884499999999999 2023-11-05T08:33:54.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-11-05T08:33:54.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255161+0000", 2023-11-05T08:33:54.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1033e-05 2023-11-05T08:33:54.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.261568+0000", 2023-11-05T08:33:54.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0064071639999999999 2023-11-05T08:33:54.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:54.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.261728+0000", 2023-11-05T08:33:54.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015980500000000001 2023-11-05T08:33:54.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:54.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.274868+0000", 2023-11-05T08:33:54.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013139985 2023-11-05T08:33:54.096 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.096 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:54.096 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.097 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:347 3.fs0 3:f2348bf8:::smithi09039252-39:head [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:54.097 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:05.255019+0000", 2023-11-05T08:33:54.097 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 48.812846511000004, 2023-11-05T08:33:54.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021017214999999999, 2023-11-05T08:33:54.098 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:54.098 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:54.098 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:54.098 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:54.098 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:54.099 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 347 2023-11-05T08:33:54.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.099 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:54.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:54.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255019+0000", 2023-11-05T08:33:54.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:54.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255019+0000", 2023-11-05T08:33:54.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:54.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255021+0000", 2023-11-05T08:33:54.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1260000000000001e-06 2023-11-05T08:33:54.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:54.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255026+0000", 2023-11-05T08:33:54.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8640000000000001e-06 2023-11-05T08:33:54.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:54.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255039+0000", 2023-11-05T08:33:54.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2238999999999999e-05 2023-11-05T08:33:54.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:54.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255047+0000", 2023-11-05T08:33:54.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7159999999999992e-06 2023-11-05T08:33:54.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255354+0000", 2023-11-05T08:33:54.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030627999999999997 2023-11-05T08:33:54.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:33:54.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255382+0000", 2023-11-05T08:33:54.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8439e-05 2023-11-05T08:33:54.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.266020+0000", 2023-11-05T08:33:54.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010637807000000001 2023-11-05T08:33:54.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:54.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.266295+0000", 2023-11-05T08:33:54.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027543400000000002 2023-11-05T08:33:54.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:54.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.276037+0000", 2023-11-05T08:33:54.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0097413099999999996 2023-11-05T08:33:54.108 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.108 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:54.109 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.109 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:348 3.fs0 3:f2348bf8:::smithi09039252-39:head [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:54.109 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:05.255103+0000", 2023-11-05T08:33:54.110 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 48.812762554000003, 2023-11-05T08:33:54.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022537980999999999, 2023-11-05T08:33:54.110 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:54.110 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:54.110 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:54.110 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:54.111 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:54.111 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 348 2023-11-05T08:33:54.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.111 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:54.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:54.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255103+0000", 2023-11-05T08:33:54.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:54.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255103+0000", 2023-11-05T08:33:54.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:54.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255105+0000", 2023-11-05T08:33:54.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.24e-06 2023-11-05T08:33:54.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:54.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255136+0000", 2023-11-05T08:33:54.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0979000000000003e-05 2023-11-05T08:33:54.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:54.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255164+0000", 2023-11-05T08:33:54.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8662000000000001e-05 2023-11-05T08:33:54.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:54.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255181+0000", 2023-11-05T08:33:54.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6935999999999998e-05 2023-11-05T08:33:54.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255509+0000", 2023-11-05T08:33:54.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032754300000000002 2023-11-05T08:33:54.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:33:54.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255536+0000", 2023-11-05T08:33:54.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6889e-05 2023-11-05T08:33:54.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.267115+0000", 2023-11-05T08:33:54.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011579104 2023-11-05T08:33:54.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:54.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.267222+0000", 2023-11-05T08:33:54.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106864 2023-11-05T08:33:54.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:54.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.277641+0000", 2023-11-05T08:33:54.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010419764 2023-11-05T08:33:54.121 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.121 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:54.121 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.121 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:369 3.16s0 3:6df625fb:::smithi09039252-31:head [append 0~524288] snapc 15=[7] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2023-11-05T08:33:54.122 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:09.281473+0000", 2023-11-05T08:33:54.122 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 44.786392827, 2023-11-05T08:33:54.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020569061999999999, 2023-11-05T08:33:54.122 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:54.122 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:54.122 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:54.123 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:54.123 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:54.123 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 369 2023-11-05T08:33:54.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.123 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:54.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:54.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.281473+0000", 2023-11-05T08:33:54.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:54.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.281473+0000", 2023-11-05T08:33:54.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:54.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.281477+0000", 2023-11-05T08:33:54.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.98e-06 2023-11-05T08:33:54.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:54.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.282265+0000", 2023-11-05T08:33:54.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00078817999999999998 2023-11-05T08:33:54.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:54.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.282308+0000", 2023-11-05T08:33:54.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2775999999999998e-05 2023-11-05T08:33:54.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:54.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.282331+0000", 2023-11-05T08:33:54.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3332999999999998e-05 2023-11-05T08:33:54.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.282525+0000", 2023-11-05T08:33:54.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000193671 2023-11-05T08:33:54.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-11-05T08:33:54.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.282555+0000", 2023-11-05T08:33:54.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9669e-05 2023-11-05T08:33:54.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.287913+0000", 2023-11-05T08:33:54.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053581360000000003 2023-11-05T08:33:54.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:54.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.288217+0000", 2023-11-05T08:33:54.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030429900000000001 2023-11-05T08:33:54.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:54.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.291625+0000", 2023-11-05T08:33:54.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034076779999999999 2023-11-05T08:33:54.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:54.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.301060+0000", 2023-11-05T08:33:54.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0094351580000000008 2023-11-05T08:33:54.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:54.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.301904+0000", 2023-11-05T08:33:54.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00084351399999999996 2023-11-05T08:33:54.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:54.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.302042+0000", 2023-11-05T08:33:54.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013866799999999999 2023-11-05T08:33:54.136 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.136 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:54.136 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.137 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:370 3.16s0 3:6df625fb:::smithi09039252-31:head [append 0~524288] snapc 15=[7] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2023-11-05T08:33:54.137 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:09.290980+0000", 2023-11-05T08:33:54.137 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 44.776885509000003, 2023-11-05T08:33:54.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019531185999999999, 2023-11-05T08:33:54.137 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:54.138 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:54.138 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:54.138 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:54.138 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:54.138 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 370 2023-11-05T08:33:54.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.139 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:54.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:54.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.290980+0000", 2023-11-05T08:33:54.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:54.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.290980+0000", 2023-11-05T08:33:54.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:54.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.290983+0000", 2023-11-05T08:33:54.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9560000000000002e-06 2023-11-05T08:33:54.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:54.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.291841+0000", 2023-11-05T08:33:54.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00085721599999999995 2023-11-05T08:33:54.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:54.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.291879+0000", 2023-11-05T08:33:54.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7882000000000001e-05 2023-11-05T08:33:54.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:54.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.291904+0000", 2023-11-05T08:33:54.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5658999999999999e-05 2023-11-05T08:33:54.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.294143+0000", 2023-11-05T08:33:54.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022388830000000001 2023-11-05T08:33:54.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:54.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.294499+0000", 2023-11-05T08:33:54.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035557600000000003 2023-11-05T08:33:54.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:54.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.296802+0000", 2023-11-05T08:33:54.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023036509999999999 2023-11-05T08:33:54.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:54.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.302059+0000", 2023-11-05T08:33:54.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052565379999999998 2023-11-05T08:33:54.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:54.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.310264+0000", 2023-11-05T08:33:54.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0082055919999999994 2023-11-05T08:33:54.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.149 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:54.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.310512+0000", 2023-11-05T08:33:54.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024723300000000002 2023-11-05T08:33:54.149 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.149 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:54.149 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.150 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:371 3.16s0 3:6df625fb:::smithi09039252-31:head [append 0~245760] snapc 15=[7] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2023-11-05T08:33:54.150 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:09.296540+0000", 2023-11-05T08:33:54.150 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 44.771325599000001, 2023-11-05T08:33:54.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015979499000000001, 2023-11-05T08:33:54.151 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:54.151 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:54.151 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:54.151 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:54.151 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:54.152 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 371 2023-11-05T08:33:54.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.152 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:54.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:54.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.296540+0000", 2023-11-05T08:33:54.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:54.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.296540+0000", 2023-11-05T08:33:54.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:54.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.296543+0000", 2023-11-05T08:33:54.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8329999999999998e-06 2023-11-05T08:33:54.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:54.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.296864+0000", 2023-11-05T08:33:54.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032048100000000001 2023-11-05T08:33:54.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.156 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:54.156 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.296899+0000", 2023-11-05T08:33:54.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5735999999999998e-05 2023-11-05T08:33:54.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.157 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:54.157 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.296920+0000", 2023-11-05T08:33:54.157 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0268999999999999e-05 2023-11-05T08:33:54.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.158 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.302604+0000", 2023-11-05T08:33:54.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0056841879999999997 2023-11-05T08:33:54.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:54.159 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.302966+0000", 2023-11-05T08:33:54.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036163200000000002 2023-11-05T08:33:54.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:54.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.304745+0000", 2023-11-05T08:33:54.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001779596 2023-11-05T08:33:54.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:54.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.309738+0000", 2023-11-05T08:33:54.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049929500000000003 2023-11-05T08:33:54.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.161 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:54.161 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.312278+0000", 2023-11-05T08:33:54.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002540231 2023-11-05T08:33:54.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.162 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:54.162 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.312520+0000", 2023-11-05T08:33:54.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024158299999999999 2023-11-05T08:33:54.163 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.163 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:54.163 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.164 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:372 3.16s0 3:6df625fb:::smithi09039252-31:head [setxattr _header (57) in=7b] snapc 15=[7] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2023-11-05T08:33:54.164 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:09.296991+0000", 2023-11-05T08:33:54.164 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 44.770875394999997, 2023-11-05T08:33:54.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016671208, 2023-11-05T08:33:54.164 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:54.164 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:54.164 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:54.165 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:54.165 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:54.165 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 372 2023-11-05T08:33:54.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.165 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:54.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:54.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.296991+0000", 2023-11-05T08:33:54.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:54.167 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.296991+0000", 2023-11-05T08:33:54.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:54.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.296992+0000", 2023-11-05T08:33:54.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5230000000000001e-06 2023-11-05T08:33:54.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:54.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297001+0000", 2023-11-05T08:33:54.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1570000000000006e-06 2023-11-05T08:33:54.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:54.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297023+0000", 2023-11-05T08:33:54.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1965999999999999e-05 2023-11-05T08:33:54.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:54.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297074+0000", 2023-11-05T08:33:54.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0875000000000003e-05 2023-11-05T08:33:54.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.306255+0000", 2023-11-05T08:33:54.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0091812700000000001 2023-11-05T08:33:54.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:54.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.306557+0000", 2023-11-05T08:33:54.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030117799999999998 2023-11-05T08:33:54.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:54.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.308901+0000", 2023-11-05T08:33:54.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023442990000000002 2023-11-05T08:33:54.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:54.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.312840+0000", 2023-11-05T08:33:54.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039388670000000004 2023-11-05T08:33:54.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:54.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.313428+0000", 2023-11-05T08:33:54.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058814000000000002 2023-11-05T08:33:54.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:54.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.313662+0000", 2023-11-05T08:33:54.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000233933 2023-11-05T08:33:54.176 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.176 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:54.177 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.177 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:373 3.16s0 3:6df625fb:::smithi09039252-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e56)", 2023-11-05T08:33:54.177 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:09.297100+0000", 2023-11-05T08:33:54.177 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 44.770765752000003, 2023-11-05T08:33:54.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020900037999999999, 2023-11-05T08:33:54.178 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:54.178 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:54.178 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:54.178 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:54.179 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:54.179 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 373 2023-11-05T08:33:54.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.179 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:54.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:54.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297100+0000", 2023-11-05T08:33:54.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:54.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297100+0000", 2023-11-05T08:33:54.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:54.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297102+0000", 2023-11-05T08:33:54.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.454e-06 2023-11-05T08:33:54.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:54.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297108+0000", 2023-11-05T08:33:54.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8260000000000004e-06 2023-11-05T08:33:54.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:54.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297131+0000", 2023-11-05T08:33:54.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2042999999999999e-05 2023-11-05T08:33:54.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:54.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297145+0000", 2023-11-05T08:33:54.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4783000000000001e-05 2023-11-05T08:33:54.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.314858+0000", 2023-11-05T08:33:54.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017712525 2023-11-05T08:33:54.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:54.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.315292+0000", 2023-11-05T08:33:54.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043406100000000002 2023-11-05T08:33:54.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:54.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.318000+0000", 2023-11-05T08:33:54.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0027083459999999999 2023-11-05T08:33:54.187 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.187 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:54.187 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.188 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:501 3.fs0 3:f91b0618:::smithi09039252-26:7 [read 0~425984,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2023-11-05T08:33:54.188 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:19.393991+0000", 2023-11-05T08:33:54.188 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.673875002999999, 2023-11-05T08:33:54.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016310758000000002, 2023-11-05T08:33:54.188 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:54.189 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:54.189 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:54.189 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:54.189 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:54.189 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 501 2023-11-05T08:33:54.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.190 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:54.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:54.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.393991+0000", 2023-11-05T08:33:54.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:54.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.393991+0000", 2023-11-05T08:33:54.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:54.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.393995+0000", 2023-11-05T08:33:54.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6210000000000001e-06 2023-11-05T08:33:54.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:54.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394010+0000", 2023-11-05T08:33:54.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5065000000000001e-05 2023-11-05T08:33:54.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:54.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394045+0000", 2023-11-05T08:33:54.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5095000000000002e-05 2023-11-05T08:33:54.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:54.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394063+0000", 2023-11-05T08:33:54.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8091e-05 2023-11-05T08:33:54.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394163+0000", 2023-11-05T08:33:54.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100367 2023-11-05T08:33:54.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-11-05T08:33:54.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394195+0000", 2023-11-05T08:33:54.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1349000000000001e-05 2023-11-05T08:33:54.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.398429+0000", 2023-11-05T08:33:54.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0042346190000000002 2023-11-05T08:33:54.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:54.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.398635+0000", 2023-11-05T08:33:54.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020568600000000001 2023-11-05T08:33:54.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:54.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.410302+0000", 2023-11-05T08:33:54.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011666865 2023-11-05T08:33:54.199 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.200 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:54.200 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.200 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:502 3.fs0 3:f91b0618:::smithi09039252-26:7 [read 0~425984,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2023-11-05T08:33:54.200 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:19.394105+0000", 2023-11-05T08:33:54.201 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.673761202000001, 2023-11-05T08:33:54.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017035277000000001, 2023-11-05T08:33:54.201 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:54.201 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:54.201 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:54.201 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:54.202 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:54.202 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 502 2023-11-05T08:33:54.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.202 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:54.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:54.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394105+0000", 2023-11-05T08:33:54.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:54.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394105+0000", 2023-11-05T08:33:54.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:54.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394106+0000", 2023-11-05T08:33:54.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9999999999999996e-07 2023-11-05T08:33:54.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:54.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394111+0000", 2023-11-05T08:33:54.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9620000000000001e-06 2023-11-05T08:33:54.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:54.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394122+0000", 2023-11-05T08:33:54.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1837e-05 2023-11-05T08:33:54.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:54.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394135+0000", 2023-11-05T08:33:54.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3103e-05 2023-11-05T08:33:54.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394300+0000", 2023-11-05T08:33:54.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016444800000000001 2023-11-05T08:33:54.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:33:54.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394334+0000", 2023-11-05T08:33:54.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4518999999999999e-05 2023-11-05T08:33:54.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.399381+0000", 2023-11-05T08:33:54.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.005046315 2023-11-05T08:33:54.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:54.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.399560+0000", 2023-11-05T08:33:54.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179457 2023-11-05T08:33:54.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:54.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.411140+0000", 2023-11-05T08:33:54.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011579836 2023-11-05T08:33:54.212 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.212 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:54.212 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.212 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:503 3.fs0 3:f91b0618:::smithi09039252-26:7 [read 0~425984,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2023-11-05T08:33:54.213 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:19.394176+0000", 2023-11-05T08:33:54.213 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.673689793000001, 2023-11-05T08:33:54.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018069346999999999, 2023-11-05T08:33:54.213 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:54.213 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:54.213 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:54.214 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:54.214 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:54.214 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 503 2023-11-05T08:33:54.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.214 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:54.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:54.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394176+0000", 2023-11-05T08:33:54.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:54.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394176+0000", 2023-11-05T08:33:54.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:54.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394177+0000", 2023-11-05T08:33:54.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9800000000000003e-07 2023-11-05T08:33:54.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:54.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394183+0000", 2023-11-05T08:33:54.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4489999999999996e-06 2023-11-05T08:33:54.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:54.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394195+0000", 2023-11-05T08:33:54.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1572e-05 2023-11-05T08:33:54.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:54.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394203+0000", 2023-11-05T08:33:54.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0399999999999993e-06 2023-11-05T08:33:54.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394375+0000", 2023-11-05T08:33:54.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017232799999999999 2023-11-05T08:33:54.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:33:54.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.394382+0000", 2023-11-05T08:33:54.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5180000000000001e-06 2023-11-05T08:33:54.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.401433+0000", 2023-11-05T08:33:54.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0070509659999999997 2023-11-05T08:33:54.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:54.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.401608+0000", 2023-11-05T08:33:54.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017502199999999999 2023-11-05T08:33:54.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:54.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:19.412246+0000", 2023-11-05T08:33:54.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010637654 2023-11-05T08:33:54.224 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.224 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:54.224 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.225 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:579 3.fs0 3:f2f7b156:::smithi09039252-30:head [append 0~622592] snapc 2b=[2b,7] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2023-11-05T08:33:54.225 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:30.527658+0000", 2023-11-05T08:33:54.225 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.540208058000001, 2023-11-05T08:33:54.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.023980108, 2023-11-05T08:33:54.226 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:54.226 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:54.226 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:54.226 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:54.226 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:54.226 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 579 2023-11-05T08:33:54.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.227 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:54.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:54.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.527658+0000", 2023-11-05T08:33:54.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:54.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.527658+0000", 2023-11-05T08:33:54.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:54.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.527662+0000", 2023-11-05T08:33:54.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8530000000000002e-06 2023-11-05T08:33:54.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:54.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.528482+0000", 2023-11-05T08:33:54.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00081984099999999997 2023-11-05T08:33:54.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:54.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.528516+0000", 2023-11-05T08:33:54.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4604999999999998e-05 2023-11-05T08:33:54.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:54.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.528537+0000", 2023-11-05T08:33:54.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0933000000000001e-05 2023-11-05T08:33:54.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.528729+0000", 2023-11-05T08:33:54.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019200900000000001 2023-11-05T08:33:54.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-11-05T08:33:54.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.528762+0000", 2023-11-05T08:33:54.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3324999999999997e-05 2023-11-05T08:33:54.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.535307+0000", 2023-11-05T08:33:54.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0065444600000000002 2023-11-05T08:33:54.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:54.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.535736+0000", 2023-11-05T08:33:54.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042863200000000003 2023-11-05T08:33:54.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:54.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.539268+0000", 2023-11-05T08:33:54.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003532931 2023-11-05T08:33:54.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:54.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.547548+0000", 2023-11-05T08:33:54.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0082790180000000008 2023-11-05T08:33:54.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:54.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.551199+0000", 2023-11-05T08:33:54.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036517780000000001 2023-11-05T08:33:54.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:54.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.551638+0000", 2023-11-05T08:33:54.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043872300000000002 2023-11-05T08:33:54.239 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.239 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:54.239 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.240 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:580 3.fs0 3:f2f7b156:::smithi09039252-30:head [append 0~442368] snapc 2b=[2b,7] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2023-11-05T08:33:54.240 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:30.535237+0000", 2023-11-05T08:33:54.240 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.532628829, 2023-11-05T08:33:54.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017577461999999999, 2023-11-05T08:33:54.241 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:54.241 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:54.241 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:54.241 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:54.241 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:54.241 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 580 2023-11-05T08:33:54.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.242 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:54.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:54.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.535237+0000", 2023-11-05T08:33:54.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:54.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.535237+0000", 2023-11-05T08:33:54.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:54.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.535241+0000", 2023-11-05T08:33:54.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.963e-06 2023-11-05T08:33:54.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:54.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.535853+0000", 2023-11-05T08:33:54.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00061147299999999996 2023-11-05T08:33:54.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:54.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.535888+0000", 2023-11-05T08:33:54.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5355000000000003e-05 2023-11-05T08:33:54.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:54.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.535904+0000", 2023-11-05T08:33:54.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5699e-05 2023-11-05T08:33:54.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.543209+0000", 2023-11-05T08:33:54.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0073058610000000003 2023-11-05T08:33:54.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:54.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.543547+0000", 2023-11-05T08:33:54.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033735500000000001 2023-11-05T08:33:54.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:54.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.545547+0000", 2023-11-05T08:33:54.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020002169999999999 2023-11-05T08:33:54.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:54.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.551911+0000", 2023-11-05T08:33:54.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0063640119999999996 2023-11-05T08:33:54.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:54.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.552375+0000", 2023-11-05T08:33:54.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046346499999999998 2023-11-05T08:33:54.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:54.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.552815+0000", 2023-11-05T08:33:54.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044006200000000002 2023-11-05T08:33:54.252 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.252 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:54.252 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.253 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:584 3.fs0 3:f2f7b156:::smithi09039252-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e77)", 2023-11-05T08:33:54.253 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:30.548638+0000", 2023-11-05T08:33:54.253 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.519227561000001, 2023-11-05T08:33:54.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020181330000000001, 2023-11-05T08:33:54.253 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:54.254 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:54.254 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:54.254 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:54.254 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:54.254 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 584 2023-11-05T08:33:54.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.255 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:54.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:54.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.548638+0000", 2023-11-05T08:33:54.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:54.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.548638+0000", 2023-11-05T08:33:54.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:54.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.548639+0000", 2023-11-05T08:33:54.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7000000000000003e-07 2023-11-05T08:33:54.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:54.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.548644+0000", 2023-11-05T08:33:54.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1500000000000001e-06 2023-11-05T08:33:54.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:54.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.548655+0000", 2023-11-05T08:33:54.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1046000000000001e-05 2023-11-05T08:33:54.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:54.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.548663+0000", 2023-11-05T08:33:54.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9799999999999998e-06 2023-11-05T08:33:54.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.562111+0000", 2023-11-05T08:33:54.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013448402999999999 2023-11-05T08:33:54.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-11-05T08:33:54.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.562442+0000", 2023-11-05T08:33:54.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033102700000000001 2023-11-05T08:33:54.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.565260+0000", 2023-11-05T08:33:54.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028184479999999999 2023-11-05T08:33:54.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:54.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.565441+0000", 2023-11-05T08:33:54.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018098999999999999 2023-11-05T08:33:54.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:54.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:30.568820+0000", 2023-11-05T08:33:54.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033783160000000001 2023-11-05T08:33:54.264 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.264 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:54.265 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.265 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:658 3.13s0 3:ce8c844b:::smithi09039252-21:head [stat,delete] snapc 38=[7] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2023-11-05T08:33:54.265 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:45.661195+0000", 2023-11-05T08:33:54.265 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.4066714089999994, 2023-11-05T08:33:54.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018640074, 2023-11-05T08:33:54.266 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:54.266 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:54.266 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:33:54.266 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:33:54.267 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:54.267 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 658 2023-11-05T08:33:54.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.267 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:54.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:54.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:45.661195+0000", 2023-11-05T08:33:54.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:54.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:45.661195+0000", 2023-11-05T08:33:54.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:54.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:45.661197+0000", 2023-11-05T08:33:54.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7099999999999999e-06 2023-11-05T08:33:54.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:54.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:45.661208+0000", 2023-11-05T08:33:54.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0957e-05 2023-11-05T08:33:54.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:54.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:45.661242+0000", 2023-11-05T08:33:54.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4005999999999999e-05 2023-11-05T08:33:54.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:54.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:45.661260+0000", 2023-11-05T08:33:54.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7484999999999999e-05 2023-11-05T08:33:54.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:45.661399+0000", 2023-11-05T08:33:54.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013945299999999999 2023-11-05T08:33:54.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-11-05T08:33:54.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:45.661412+0000", 2023-11-05T08:33:54.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3006999999999999e-05 2023-11-05T08:33:54.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:45.668992+0000", 2023-11-05T08:33:54.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0075800030000000001 2023-11-05T08:33:54.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:54.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:45.669272+0000", 2023-11-05T08:33:54.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028012499999999999 2023-11-05T08:33:54.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:54.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:45.671159+0000", 2023-11-05T08:33:54.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0018865749999999999 2023-11-05T08:33:54.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:54.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:45.678095+0000", 2023-11-05T08:33:54.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0069362900000000003 2023-11-05T08:33:54.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:33:54.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:45.678268+0000", 2023-11-05T08:33:54.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017290099999999999 2023-11-05T08:33:54.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:54.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:45.679835+0000", 2023-11-05T08:33:54.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015665620000000001 2023-11-05T08:33:54.280 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.280 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:54.280 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.280 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.11s1 97/45 ECSubWrite(tid=19, reqid=osd.10.0:19, at_version=97'29, trim_to=0'0, roll_forward_to=94'27))", 2023-11-05T08:33:54.281 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:50.670523+0000", 2023-11-05T08:33:54.281 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.39734267, 2023-11-05T08:33:54.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020069789000000001, 2023-11-05T08:33:54.281 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:54.281 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:54.281 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:54.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:54.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.670523+0000", 2023-11-05T08:33:54.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:54.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.670523+0000", 2023-11-05T08:33:54.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:54.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.670524+0000", 2023-11-05T08:33:54.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1399999999999996e-07 2023-11-05T08:33:54.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:54.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.670577+0000", 2023-11-05T08:33:54.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3264999999999999e-05 2023-11-05T08:33:54.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:54.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.670607+0000", 2023-11-05T08:33:54.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9428999999999999e-05 2023-11-05T08:33:54.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:54.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.670628+0000", 2023-11-05T08:33:54.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.141e-05 2023-11-05T08:33:54.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.670755+0000", 2023-11-05T08:33:54.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012662099999999999 2023-11-05T08:33:54.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:54.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.672213+0000", 2023-11-05T08:33:54.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001458528 2023-11-05T08:33:54.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:54.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.690482+0000", 2023-11-05T08:33:54.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018268850999999999 2023-11-05T08:33:54.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:54.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.690593+0000", 2023-11-05T08:33:54.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011097100000000001 2023-11-05T08:33:54.290 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.290 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:54.291 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.291 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.es3 97/30 ECSubWrite(tid=127, reqid=osd.2.0:127, at_version=97'98, trim_to=0'0, roll_forward_to=94'96))", 2023-11-05T08:33:54.291 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:50.671293+0000", 2023-11-05T08:33:54.291 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.3965727640000001, 2023-11-05T08:33:54.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027785358, 2023-11-05T08:33:54.292 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:54.292 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:54.292 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:54.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:54.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.671293+0000", 2023-11-05T08:33:54.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:54.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.671293+0000", 2023-11-05T08:33:54.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:54.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.671294+0000", 2023-11-05T08:33:54.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.23e-07 2023-11-05T08:33:54.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:54.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.671347+0000", 2023-11-05T08:33:54.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2945000000000002e-05 2023-11-05T08:33:54.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:54.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.671379+0000", 2023-11-05T08:33:54.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2026000000000001e-05 2023-11-05T08:33:54.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:54.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.671473+0000", 2023-11-05T08:33:54.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4043999999999996e-05 2023-11-05T08:33:54.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.671776+0000", 2023-11-05T08:33:54.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000303073 2023-11-05T08:33:54.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:54.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.672490+0000", 2023-11-05T08:33:54.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071373699999999999 2023-11-05T08:33:54.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:54.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.699029+0000", 2023-11-05T08:33:54.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026539584000000001 2023-11-05T08:33:54.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:54.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.699079+0000", 2023-11-05T08:33:54.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9125999999999997e-05 2023-11-05T08:33:54.301 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.301 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:54.301 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.302 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.es3 97/30 ECSubWrite(tid=128, reqid=osd.2.0:128, at_version=97'100, trim_to=0'0, roll_forward_to=94'96))", 2023-11-05T08:33:54.302 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:50.674824+0000", 2023-11-05T08:33:54.302 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.393042355, 2023-11-05T08:33:54.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027511565000000002, 2023-11-05T08:33:54.302 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:54.303 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:54.303 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:54.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:54.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.674824+0000", 2023-11-05T08:33:54.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:54.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.674824+0000", 2023-11-05T08:33:54.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:54.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.674825+0000", 2023-11-05T08:33:54.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8500000000000001e-06 2023-11-05T08:33:54.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:54.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.674882+0000", 2023-11-05T08:33:54.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6125999999999998e-05 2023-11-05T08:33:54.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:54.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.674914+0000", 2023-11-05T08:33:54.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2722000000000003e-05 2023-11-05T08:33:54.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:54.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.674937+0000", 2023-11-05T08:33:54.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2385999999999999e-05 2023-11-05T08:33:54.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.688784+0000", 2023-11-05T08:33:54.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013846984999999999 2023-11-05T08:33:54.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:54.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.688799+0000", 2023-11-05T08:33:54.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4854999999999999e-05 2023-11-05T08:33:54.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:54.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.702253+0000", 2023-11-05T08:33:54.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01345492 2023-11-05T08:33:54.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:54.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.702335+0000", 2023-11-05T08:33:54.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1720999999999999e-05 2023-11-05T08:33:54.311 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.311 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:54.311 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.312 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.11s1 97/45 ECSubWrite(tid=20, reqid=osd.10.0:20, at_version=97'31, trim_to=0'0, roll_forward_to=94'27))", 2023-11-05T08:33:54.312 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:50.677830+0000", 2023-11-05T08:33:54.312 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.3900359130000002, 2023-11-05T08:33:54.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019062816, 2023-11-05T08:33:54.313 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:54.313 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:54.313 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:54.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:54.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.677830+0000", 2023-11-05T08:33:54.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:54.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.677830+0000", 2023-11-05T08:33:54.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:54.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.677832+0000", 2023-11-05T08:33:54.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.897e-06 2023-11-05T08:33:54.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:54.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.677897+0000", 2023-11-05T08:33:54.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5182e-05 2023-11-05T08:33:54.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:54.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.677927+0000", 2023-11-05T08:33:54.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0209000000000002e-05 2023-11-05T08:33:54.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:54.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.677948+0000", 2023-11-05T08:33:54.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0613000000000001e-05 2023-11-05T08:33:54.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.678133+0000", 2023-11-05T08:33:54.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000185539 2023-11-05T08:33:54.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:54.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.678157+0000", 2023-11-05T08:33:54.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3822999999999999e-05 2023-11-05T08:33:54.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:54.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.696828+0000", 2023-11-05T08:33:54.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018670237999999999 2023-11-05T08:33:54.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:54.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.696893+0000", 2023-11-05T08:33:54.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5314999999999994e-05 2023-11-05T08:33:54.322 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.322 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:54.322 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.322 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.322 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:54.323 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:33:54.323 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~58s 2023-11-05T08:33:54.324 INFO:teuthology.orchestra.run.smithi002.stdout:osd.15: {} 2023-11-05T08:33:54.325 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-05T08:33:54.325 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2023-11-05T08:33:54.326 INFO:teuthology.orchestra.run.smithi002.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-11-05T08:33:54.327 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-05T08:33:54.327 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:33:54.321+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:33:54.425 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:33:54.422+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:33:54.457 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:33:54.458 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-11-05T08:33:54.458 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-11-05T08:33:54.458 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:33:54.471 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2023-11-05T08:33:54.526 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:33:54.524+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:33:54.627 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:33:54.625+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:33:54.635 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:33:54.636 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-11-05T08:33:54.636 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-11-05T08:33:54.636 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-11-05T08:33:54.636 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:33:54.649 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2023-11-05T08:33:54.727 INFO:teuthology.orchestra.run.smithi002.stderr:reweighted osd.7 to 1 (10000) 2023-11-05T08:33:54.728 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:33:54.724+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:33:54.734 INFO:tasks.rados.rados.0.smithi090.stdout:324: done (2 left) 2023-11-05T08:33:54.734 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 26 current snap is 34 2023-11-05T08:33:54.734 INFO:tasks.rados.rados.0.smithi090.stdout:325: seq_num 123 ranges {1933312=524288,2457600=573440,3031040=688128,3719168=294912} 2023-11-05T08:33:54.755 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2023-11-05T08:33:54.758 INFO:tasks.rados.rados.0.smithi090.stdout:325: writing smithi09039252-26 from 1933312 to 2457600 tid 1 2023-11-05T08:33:54.764 INFO:tasks.rados.rados.0.smithi090.stdout:325: writing smithi09039252-26 from 2457600 to 3031040 tid 2 2023-11-05T08:33:54.771 INFO:tasks.rados.rados.0.smithi090.stdout:325: writing smithi09039252-26 from 3031040 to 3719168 tid 3 2023-11-05T08:33:54.775 INFO:tasks.rados.rados.0.smithi090.stdout:325: writing smithi09039252-26 from 3719168 to 4014080 tid 4 2023-11-05T08:33:54.775 INFO:tasks.rados.rados.0.smithi090.stdout:326: rollback oid 20 current snap is 34 2023-11-05T08:33:54.775 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 20 to 32 2023-11-05T08:33:54.775 INFO:tasks.rados.rados.0.smithi090.stdout:322: finishing copy_from racing read to smithi09039252-6 2023-11-05T08:33:54.775 INFO:tasks.rados.rados.0.smithi090.stdout:327: read oid 30 snap 32 2023-11-05T08:33:54.776 INFO:tasks.rados.rados.0.smithi090.stdout:327: expect (ObjNum 109 snap 29 seq_num 109) 2023-11-05T08:33:54.776 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 47 current snap is 34 2023-11-05T08:33:54.776 INFO:tasks.rados.rados.0.smithi090.stdout:322: finishing copy_from to smithi09039252-6 2023-11-05T08:33:54.776 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 6 v 49 (ObjNum 47 snap 0 seq_num 47) dirty exists 2023-11-05T08:33:54.776 INFO:tasks.rados.rados.0.smithi090.stdout:323: expect (ObjNum 117 snap 30 seq_num 117) 2023-11-05T08:33:54.784 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:33:54.785 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-11-05T08:33:54.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-11-05T08:33:54.785 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-11-05T08:33:54.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.785 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.3s3 36/30 ECSubWrite(tid=183, reqid=osd.13.0:183, at_version=36'12, trim_to=0'0, roll_forward_to=35'10))", 2023-11-05T08:33:54.786 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.072455+0000", 2023-11-05T08:33:54.786 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 65.708595387000003, 2023-11-05T08:33:54.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0063400499999999999, 2023-11-05T08:33:54.786 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:54.786 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:54.787 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:54.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:54.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072455+0000", 2023-11-05T08:33:54.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:54.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072455+0000", 2023-11-05T08:33:54.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:54.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072456+0000", 2023-11-05T08:33:54.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9999999999999998e-07 2023-11-05T08:33:54.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:54.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072512+0000", 2023-11-05T08:33:54.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5667000000000001e-05 2023-11-05T08:33:54.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:54.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072537+0000", 2023-11-05T08:33:54.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5833000000000002e-05 2023-11-05T08:33:54.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:54.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072581+0000", 2023-11-05T08:33:54.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3470999999999998e-05 2023-11-05T08:33:54.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072752+0000", 2023-11-05T08:33:54.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017130100000000001 2023-11-05T08:33:54.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:54.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.072775+0000", 2023-11-05T08:33:54.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3379000000000001e-05 2023-11-05T08:33:54.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:54.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.078646+0000", 2023-11-05T08:33:54.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0058709419999999997 2023-11-05T08:33:54.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:54.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.078795+0000", 2023-11-05T08:33:54.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014895699999999999 2023-11-05T08:33:54.796 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.796 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:54.796 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.797 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.8s5 36/30 ECSubWrite(tid=23, reqid=osd.5.0:23, at_version=36'15, trim_to=0'0, roll_forward_to=34'13))", 2023-11-05T08:33:54.797 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.079609+0000", 2023-11-05T08:33:54.797 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 65.701441685000006, 2023-11-05T08:33:54.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049579109999999997, 2023-11-05T08:33:54.798 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:54.798 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:54.798 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:54.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:54.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079609+0000", 2023-11-05T08:33:54.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:54.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079609+0000", 2023-11-05T08:33:54.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:54.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079609+0000", 2023-11-05T08:33:54.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.27e-07 2023-11-05T08:33:54.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:54.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079645+0000", 2023-11-05T08:33:54.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5522000000000003e-05 2023-11-05T08:33:54.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:54.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079672+0000", 2023-11-05T08:33:54.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6648e-05 2023-11-05T08:33:54.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:54.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079693+0000", 2023-11-05T08:33:54.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1206e-05 2023-11-05T08:33:54.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079849+0000", 2023-11-05T08:33:54.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156112 2023-11-05T08:33:54.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:54.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.079873+0000", 2023-11-05T08:33:54.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3589000000000001e-05 2023-11-05T08:33:54.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:54.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.084477+0000", 2023-11-05T08:33:54.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0046040969999999997 2023-11-05T08:33:54.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:54.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.084567+0000", 2023-11-05T08:33:54.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0309999999999994e-05 2023-11-05T08:33:54.807 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.808 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:54.808 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.808 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.8s5 36/30 ECSubWrite(tid=24, reqid=osd.5.0:23, at_version=0'0, trim_to=0'0, roll_forward_to=36'15))", 2023-11-05T08:33:54.808 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.088806+0000", 2023-11-05T08:33:54.809 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 65.692244415000005, 2023-11-05T08:33:54.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055977739999999998, 2023-11-05T08:33:54.809 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:54.809 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:54.809 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:54.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:54.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.088806+0000", 2023-11-05T08:33:54.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:54.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.088806+0000", 2023-11-05T08:33:54.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:54.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.092263+0000", 2023-11-05T08:33:54.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034565210000000002 2023-11-05T08:33:54.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:54.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.092287+0000", 2023-11-05T08:33:54.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4505999999999999e-05 2023-11-05T08:33:54.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:54.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.092314+0000", 2023-11-05T08:33:54.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6335000000000001e-05 2023-11-05T08:33:54.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:54.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.092329+0000", 2023-11-05T08:33:54.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5486e-05 2023-11-05T08:33:54.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.092519+0000", 2023-11-05T08:33:54.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019026700000000001 2023-11-05T08:33:54.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:54.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.092545+0000", 2023-11-05T08:33:54.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5644999999999999e-05 2023-11-05T08:33:54.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:54.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.094327+0000", 2023-11-05T08:33:54.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001781543 2023-11-05T08:33:54.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:54.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.094404+0000", 2023-11-05T08:33:54.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7471000000000004e-05 2023-11-05T08:33:54.819 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.819 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:54.819 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.820 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.2s4 41/40 ECSubWrite(tid=187, reqid=client.4691.0:292, at_version=41'9, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:54.820 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:54.162954+0000", 2023-11-05T08:33:54.820 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 60.618096291000001, 2023-11-05T08:33:54.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0045082990000000003, 2023-11-05T08:33:54.821 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:54.821 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:54.821 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:54.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:54.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.162954+0000", 2023-11-05T08:33:54.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:54.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.162954+0000", 2023-11-05T08:33:54.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:54.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.162956+0000", 2023-11-05T08:33:54.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.86e-06 2023-11-05T08:33:54.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:54.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.163012+0000", 2023-11-05T08:33:54.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5744e-05 2023-11-05T08:33:54.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:54.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.163048+0000", 2023-11-05T08:33:54.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6346000000000001e-05 2023-11-05T08:33:54.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:54.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.163075+0000", 2023-11-05T08:33:54.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6622e-05 2023-11-05T08:33:54.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.163236+0000", 2023-11-05T08:33:54.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016064599999999999 2023-11-05T08:33:54.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:54.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.163262+0000", 2023-11-05T08:33:54.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6356e-05 2023-11-05T08:33:54.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:54.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.167302+0000", 2023-11-05T08:33:54.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040403000000000001 2023-11-05T08:33:54.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:54.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:54.167463+0000", 2023-11-05T08:33:54.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016042500000000001 2023-11-05T08:33:54.830 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.830 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:54.831 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.831 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.16s1 46/45 ECSubWrite(tid=45, reqid=osd.11.0:45, at_version=46'11, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:54.831 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:59.219281+0000", 2023-11-05T08:33:54.831 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 55.561769276, 2023-11-05T08:33:54.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.005540456, 2023-11-05T08:33:54.832 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:54.832 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:54.832 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:54.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:54.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219281+0000", 2023-11-05T08:33:54.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:54.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219281+0000", 2023-11-05T08:33:54.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:54.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219283+0000", 2023-11-05T08:33:54.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.637e-06 2023-11-05T08:33:54.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:54.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219344+0000", 2023-11-05T08:33:54.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0831000000000001e-05 2023-11-05T08:33:54.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:54.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219376+0000", 2023-11-05T08:33:54.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1945e-05 2023-11-05T08:33:54.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:54.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219417+0000", 2023-11-05T08:33:54.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1335000000000003e-05 2023-11-05T08:33:54.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219601+0000", 2023-11-05T08:33:54.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018345699999999999 2023-11-05T08:33:54.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:54.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.219627+0000", 2023-11-05T08:33:54.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6732000000000001e-05 2023-11-05T08:33:54.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:54.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.224749+0000", 2023-11-05T08:33:54.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0051213129999999997 2023-11-05T08:33:54.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:54.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.224822+0000", 2023-11-05T08:33:54.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3206000000000003e-05 2023-11-05T08:33:54.842 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.842 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:54.842 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.843 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.11s2 46/45 ECSubWrite(tid=0, reqid=osd.10.0:0, at_version=46'17, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:54.843 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:59.222006+0000", 2023-11-05T08:33:54.843 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 55.559044583999999, 2023-11-05T08:33:54.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053541609999999996, 2023-11-05T08:33:54.844 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:54.844 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:54.844 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:54.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:54.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222006+0000", 2023-11-05T08:33:54.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:54.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222006+0000", 2023-11-05T08:33:54.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:54.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222008+0000", 2023-11-05T08:33:54.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7430000000000001e-06 2023-11-05T08:33:54.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:54.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222062+0000", 2023-11-05T08:33:54.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3891000000000003e-05 2023-11-05T08:33:54.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:54.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222095+0000", 2023-11-05T08:33:54.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3148999999999997e-05 2023-11-05T08:33:54.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:54.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222118+0000", 2023-11-05T08:33:54.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3258999999999999e-05 2023-11-05T08:33:54.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222279+0000", 2023-11-05T08:33:54.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016113 2023-11-05T08:33:54.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:54.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.222303+0000", 2023-11-05T08:33:54.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4088999999999999e-05 2023-11-05T08:33:54.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:54.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.227275+0000", 2023-11-05T08:33:54.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049715599999999999 2023-11-05T08:33:54.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:54.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.227360+0000", 2023-11-05T08:33:54.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5340000000000006e-05 2023-11-05T08:33:54.853 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.853 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:54.854 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.854 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.16s1 46/45 ECSubWrite(tid=47, reqid=osd.11.0:47, at_version=46'13, trim_to=0'0, roll_forward_to=46'11))", 2023-11-05T08:33:54.854 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:59.233948+0000", 2023-11-05T08:33:54.855 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 55.547102355, 2023-11-05T08:33:54.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048084130000000001, 2023-11-05T08:33:54.855 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:54.855 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:54.855 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:54.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:54.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.233948+0000", 2023-11-05T08:33:54.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:54.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.233948+0000", 2023-11-05T08:33:54.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:54.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.233950+0000", 2023-11-05T08:33:54.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5969999999999999e-06 2023-11-05T08:33:54.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:54.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.234005+0000", 2023-11-05T08:33:54.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5414999999999997e-05 2023-11-05T08:33:54.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:54.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.234040+0000", 2023-11-05T08:33:54.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.455e-05 2023-11-05T08:33:54.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:54.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.234061+0000", 2023-11-05T08:33:54.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1036e-05 2023-11-05T08:33:54.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.234178+0000", 2023-11-05T08:33:54.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000116822 2023-11-05T08:33:54.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:54.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.234195+0000", 2023-11-05T08:33:54.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6801999999999999e-05 2023-11-05T08:33:54.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:54.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.238676+0000", 2023-11-05T08:33:54.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004481044 2023-11-05T08:33:54.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:54.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.238757+0000", 2023-11-05T08:33:54.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1147000000000001e-05 2023-11-05T08:33:54.865 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.865 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:54.865 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.866 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.16s1 46/45 ECSubWrite(tid=48, reqid=osd.11.0:47, at_version=0'0, trim_to=0'0, roll_forward_to=46'13))", 2023-11-05T08:33:54.866 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:59.240171+0000", 2023-11-05T08:33:54.866 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 55.540879748000002, 2023-11-05T08:33:54.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0045632529999999998, 2023-11-05T08:33:54.866 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:54.867 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:54.867 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:54.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:54.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240171+0000", 2023-11-05T08:33:54.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:54.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240171+0000", 2023-11-05T08:33:54.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:54.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240172+0000", 2023-11-05T08:33:54.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.282e-06 2023-11-05T08:33:54.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:54.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240193+0000", 2023-11-05T08:33:54.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0601000000000001e-05 2023-11-05T08:33:54.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:54.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240214+0000", 2023-11-05T08:33:54.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1574999999999999e-05 2023-11-05T08:33:54.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:54.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240230+0000", 2023-11-05T08:33:54.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5656999999999999e-05 2023-11-05T08:33:54.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240361+0000", 2023-11-05T08:33:54.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000130488 2023-11-05T08:33:54.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:54.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.240372+0000", 2023-11-05T08:33:54.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1158999999999999e-05 2023-11-05T08:33:54.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:54.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.244663+0000", 2023-11-05T08:33:54.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004291563 2023-11-05T08:33:54.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:54.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:59.244734+0000", 2023-11-05T08:33:54.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0927999999999996e-05 2023-11-05T08:33:54.876 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.876 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:54.876 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.877 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.16s1 51/45 ECSubWrite(tid=58, reqid=osd.11.0:58, at_version=51'19, trim_to=0'0, roll_forward_to=49'17))", 2023-11-05T08:33:54.877 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:04.226221+0000", 2023-11-05T08:33:54.877 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.554829976000001, 2023-11-05T08:33:54.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.005713414, 2023-11-05T08:33:54.877 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:54.878 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:54.878 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:54.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:54.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226221+0000", 2023-11-05T08:33:54.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:54.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226221+0000", 2023-11-05T08:33:54.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:54.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226221+0000", 2023-11-05T08:33:54.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.53e-07 2023-11-05T08:33:54.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:54.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226253+0000", 2023-11-05T08:33:54.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1671999999999998e-05 2023-11-05T08:33:54.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:54.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226269+0000", 2023-11-05T08:33:54.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6150999999999998e-05 2023-11-05T08:33:54.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:54.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226282+0000", 2023-11-05T08:33:54.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3465000000000001e-05 2023-11-05T08:33:54.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226474+0000", 2023-11-05T08:33:54.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019140499999999999 2023-11-05T08:33:54.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:54.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.226501+0000", 2023-11-05T08:33:54.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7523000000000001e-05 2023-11-05T08:33:54.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:54.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.231902+0000", 2023-11-05T08:33:54.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0054011019999999996 2023-11-05T08:33:54.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:54.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:04.231934+0000", 2023-11-05T08:33:54.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1742999999999997e-05 2023-11-05T08:33:54.887 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.887 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:54.887 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.888 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.9s4 52/45 ECSubWrite(tid=75, reqid=client.4691.0:341, at_version=52'24, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:54.888 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:05.257804+0000", 2023-11-05T08:33:54.888 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 49.523246788000002, 2023-11-05T08:33:54.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010353123000000001, 2023-11-05T08:33:54.889 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:54.889 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:54.889 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:54.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:54.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.257804+0000", 2023-11-05T08:33:54.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:54.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.257804+0000", 2023-11-05T08:33:54.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:54.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.257805+0000", 2023-11-05T08:33:54.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3399999999999999e-07 2023-11-05T08:33:54.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:54.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.258142+0000", 2023-11-05T08:33:54.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033764100000000002 2023-11-05T08:33:54.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:54.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.258157+0000", 2023-11-05T08:33:54.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4625e-05 2023-11-05T08:33:54.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:54.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.258168+0000", 2023-11-05T08:33:54.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0992000000000001e-05 2023-11-05T08:33:54.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.258371+0000", 2023-11-05T08:33:54.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020296299999999999 2023-11-05T08:33:54.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:54.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.258404+0000", 2023-11-05T08:33:54.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2765999999999998e-05 2023-11-05T08:33:54.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:54.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.268067+0000", 2023-11-05T08:33:54.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0096629479999999993 2023-11-05T08:33:54.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:54.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.268157+0000", 2023-11-05T08:33:54.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0554000000000001e-05 2023-11-05T08:33:54.898 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.899 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:54.899 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.899 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(osd.14.22:3 3.5s0 3:a287d13d:::smithi09039252-33:head [assert-version v4,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:54.900 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:05.264331+0000", 2023-11-05T08:33:54.900 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 49.516720184, 2023-11-05T08:33:54.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010947936, 2023-11-05T08:33:54.900 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:54.900 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:33:54.901 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:54.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:54.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.264331+0000", 2023-11-05T08:33:54.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:54.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.264331+0000", 2023-11-05T08:33:54.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:54.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.264334+0000", 2023-11-05T08:33:54.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.467e-06 2023-11-05T08:33:54.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:54.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.264350+0000", 2023-11-05T08:33:54.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6200000000000001e-05 2023-11-05T08:33:54.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:54.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.264387+0000", 2023-11-05T08:33:54.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6684999999999999e-05 2023-11-05T08:33:54.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:54.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.264489+0000", 2023-11-05T08:33:54.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010218 2023-11-05T08:33:54.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.265518+0000", 2023-11-05T08:33:54.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0010291899999999999 2023-11-05T08:33:54.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-11-05T08:33:54.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.265535+0000", 2023-11-05T08:33:54.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6697e-05 2023-11-05T08:33:54.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.269530+0000", 2023-11-05T08:33:54.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039951769999999999 2023-11-05T08:33:54.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:33:54.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.269646+0000", 2023-11-05T08:33:54.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011579299999999999 2023-11-05T08:33:54.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:54.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.275278+0000", 2023-11-05T08:33:54.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0056325469999999999 2023-11-05T08:33:54.911 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.911 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:54.911 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.912 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.16s1 56/45 ECSubWrite(tid=64, reqid=client.4691.0:370, at_version=56'21, trim_to=0'0, roll_forward_to=51'19))", 2023-11-05T08:33:54.912 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:09.297268+0000", 2023-11-05T08:33:54.912 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.483782372, 2023-11-05T08:33:54.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053142609999999998, 2023-11-05T08:33:54.913 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:54.913 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:54.913 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:54.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:54.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297268+0000", 2023-11-05T08:33:54.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:54.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297268+0000", 2023-11-05T08:33:54.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:54.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297270+0000", 2023-11-05T08:33:54.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7570000000000001e-06 2023-11-05T08:33:54.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:54.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297426+0000", 2023-11-05T08:33:54.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015544199999999999 2023-11-05T08:33:54.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:54.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297457+0000", 2023-11-05T08:33:54.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1615999999999997e-05 2023-11-05T08:33:54.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:54.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.297480+0000", 2023-11-05T08:33:54.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3326e-05 2023-11-05T08:33:54.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.299414+0000", 2023-11-05T08:33:54.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001933518 2023-11-05T08:33:54.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:54.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.299445+0000", 2023-11-05T08:33:54.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0701999999999998e-05 2023-11-05T08:33:54.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:54.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.302505+0000", 2023-11-05T08:33:54.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030607590000000001 2023-11-05T08:33:54.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:54.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:09.302583+0000", 2023-11-05T08:33:54.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7140999999999999e-05 2023-11-05T08:33:54.923 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.923 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:54.923 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.923 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.9s4 92/45 ECSubWrite(tid=98, reqid=client.4691.0:662, at_version=92'38, trim_to=0'0, roll_forward_to=68'37))", 2023-11-05T08:33:54.924 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:45.749377+0000", 2023-11-05T08:33:54.924 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 9.0316736570000007, 2023-11-05T08:33:54.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0045228969999999997, 2023-11-05T08:33:54.924 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:54.924 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:54.925 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:54.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:54.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:45.749377+0000", 2023-11-05T08:33:54.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:54.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:45.749377+0000", 2023-11-05T08:33:54.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.927 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:54.927 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:45.749377+0000", 2023-11-05T08:33:54.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.48e-07 2023-11-05T08:33:54.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:54.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:45.749892+0000", 2023-11-05T08:33:54.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051456699999999995 2023-11-05T08:33:54.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:54.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:45.749905+0000", 2023-11-05T08:33:54.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2813000000000001e-05 2023-11-05T08:33:54.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:54.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:45.749913+0000", 2023-11-05T08:33:54.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7790000000000001e-06 2023-11-05T08:33:54.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:45.750008+0000", 2023-11-05T08:33:54.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4896999999999996e-05 2023-11-05T08:33:54.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:54.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:45.750025+0000", 2023-11-05T08:33:54.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6376e-05 2023-11-05T08:33:54.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:54.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:45.753817+0000", 2023-11-05T08:33:54.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037921589999999998 2023-11-05T08:33:54.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:54.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:45.753900+0000", 2023-11-05T08:33:54.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3058000000000003e-05 2023-11-05T08:33:54.934 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.934 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:54.935 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.935 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.11s2 97/45 ECSubWrite(tid=19, reqid=osd.10.0:19, at_version=97'29, trim_to=0'0, roll_forward_to=94'27))", 2023-11-05T08:33:54.935 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:50.670820+0000", 2023-11-05T08:33:54.935 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.1102306420000003, 2023-11-05T08:33:54.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048181229999999997, 2023-11-05T08:33:54.936 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:54.936 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:54.936 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:54.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:54.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.670820+0000", 2023-11-05T08:33:54.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:54.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.670820+0000", 2023-11-05T08:33:54.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:54.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.670821+0000", 2023-11-05T08:33:54.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6700000000000003e-07 2023-11-05T08:33:54.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:54.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.670905+0000", 2023-11-05T08:33:54.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4301000000000005e-05 2023-11-05T08:33:54.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:54.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.670938+0000", 2023-11-05T08:33:54.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2799000000000002e-05 2023-11-05T08:33:54.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:54.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.670963+0000", 2023-11-05T08:33:54.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4773000000000002e-05 2023-11-05T08:33:54.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.671128+0000", 2023-11-05T08:33:54.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016535899999999999 2023-11-05T08:33:54.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:54.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.671154+0000", 2023-11-05T08:33:54.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5863e-05 2023-11-05T08:33:54.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:54.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.675540+0000", 2023-11-05T08:33:54.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043857699999999998 2023-11-05T08:33:54.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:54.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.675638+0000", 2023-11-05T08:33:54.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8491000000000003e-05 2023-11-05T08:33:54.945 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.945 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:54.946 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.946 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.11s2 97/45 ECSubWrite(tid=20, reqid=osd.10.0:20, at_version=97'31, trim_to=0'0, roll_forward_to=94'27))", 2023-11-05T08:33:54.946 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:50.680220+0000", 2023-11-05T08:33:54.947 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.1008308839999996, 2023-11-05T08:33:54.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.009370527, 2023-11-05T08:33:54.947 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:54.947 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:54.947 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:54.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:54.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.680220+0000", 2023-11-05T08:33:54.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:54.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.680220+0000", 2023-11-05T08:33:54.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:54.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.680221+0000", 2023-11-05T08:33:54.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2759999999999999e-06 2023-11-05T08:33:54.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:54.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.680269+0000", 2023-11-05T08:33:54.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7506000000000002e-05 2023-11-05T08:33:54.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:54.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.680297+0000", 2023-11-05T08:33:54.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8843000000000002e-05 2023-11-05T08:33:54.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:54.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.680321+0000", 2023-11-05T08:33:54.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3149000000000001e-05 2023-11-05T08:33:54.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.680883+0000", 2023-11-05T08:33:54.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00056243000000000005 2023-11-05T08:33:54.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:54.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.680907+0000", 2023-11-05T08:33:54.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3566000000000001e-05 2023-11-05T08:33:54.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:54.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.689498+0000", 2023-11-05T08:33:54.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0085912869999999995 2023-11-05T08:33:54.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:54.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.689590+0000", 2023-11-05T08:33:54.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2470000000000001e-05 2023-11-05T08:33:54.957 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.957 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:54.957 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.958 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.14s2 97/96 ECSubWrite(tid=8, reqid=osd.9.0:7, at_version=0'0, trim_to=0'0, roll_forward_to=97'21))", 2023-11-05T08:33:54.958 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:50.691758+0000", 2023-11-05T08:33:54.958 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.0892928250000002, 2023-11-05T08:33:54.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011798856999999999, 2023-11-05T08:33:54.958 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:54.959 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:54.959 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:54.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:54.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.691758+0000", 2023-11-05T08:33:54.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:54.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.691758+0000", 2023-11-05T08:33:54.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:54.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.691760+0000", 2023-11-05T08:33:54.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7099999999999999e-06 2023-11-05T08:33:54.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:54.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.691787+0000", 2023-11-05T08:33:54.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7623e-05 2023-11-05T08:33:54.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:54.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.692664+0000", 2023-11-05T08:33:54.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00087682000000000005 2023-11-05T08:33:54.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:54.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.692694+0000", 2023-11-05T08:33:54.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9561999999999999e-05 2023-11-05T08:33:54.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.692842+0000", 2023-11-05T08:33:54.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014862 2023-11-05T08:33:54.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:54.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.692860+0000", 2023-11-05T08:33:54.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7785e-05 2023-11-05T08:33:54.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:54.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.703493+0000", 2023-11-05T08:33:54.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010633022000000001 2023-11-05T08:33:54.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:54.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.703557+0000", 2023-11-05T08:33:54.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3714999999999996e-05 2023-11-05T08:33:54.968 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.968 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:54.968 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.969 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.16s1 97/45 ECSubWrite(tid=170, reqid=osd.11.0:169, at_version=0'0, trim_to=0'0, roll_forward_to=97'48))", 2023-11-05T08:33:54.969 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:50.694630+0000", 2023-11-05T08:33:54.969 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.0864203730000002, 2023-11-05T08:33:54.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013227556, 2023-11-05T08:33:54.969 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:54.969 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:54.970 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:54.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:54.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.694630+0000", 2023-11-05T08:33:54.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:54.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.694630+0000", 2023-11-05T08:33:54.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:54.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.694632+0000", 2023-11-05T08:33:54.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4699999999999999e-06 2023-11-05T08:33:54.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:54.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.694653+0000", 2023-11-05T08:33:54.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1109999999999999e-05 2023-11-05T08:33:54.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:54.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.694676+0000", 2023-11-05T08:33:54.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2707000000000001e-05 2023-11-05T08:33:54.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:54.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.694692+0000", 2023-11-05T08:33:54.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5976000000000001e-05 2023-11-05T08:33:54.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.694828+0000", 2023-11-05T08:33:54.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013624000000000001 2023-11-05T08:33:54.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:54.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.694852+0000", 2023-11-05T08:33:54.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3958999999999999e-05 2023-11-05T08:33:54.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:54.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.707774+0000", 2023-11-05T08:33:54.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012921693 2023-11-05T08:33:54.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:54.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.707858+0000", 2023-11-05T08:33:54.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4400999999999994e-05 2023-11-05T08:33:54.979 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.979 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:54.979 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.980 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.11s2 97/45 ECSubWrite(tid=21, reqid=osd.10.0:20, at_version=0'0, trim_to=0'0, roll_forward_to=97'31))", 2023-11-05T08:33:54.980 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:50.698443+0000", 2023-11-05T08:33:54.980 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.0826073530000002, 2023-11-05T08:33:54.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0097025659999999993, 2023-11-05T08:33:54.980 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:54.981 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:54.981 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:54.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:54.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.698443+0000", 2023-11-05T08:33:54.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:54.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.698443+0000", 2023-11-05T08:33:54.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:54.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.698444+0000", 2023-11-05T08:33:54.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0699999999999999e-06 2023-11-05T08:33:54.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:54.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.698464+0000", 2023-11-05T08:33:54.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9740000000000001e-05 2023-11-05T08:33:54.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:54.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.698485+0000", 2023-11-05T08:33:54.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0919000000000001e-05 2023-11-05T08:33:54.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:54.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.698501+0000", 2023-11-05T08:33:54.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5874000000000001e-05 2023-11-05T08:33:54.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.698630+0000", 2023-11-05T08:33:54.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000128925 2023-11-05T08:33:54.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:54.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.698646+0000", 2023-11-05T08:33:54.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6418000000000001e-05 2023-11-05T08:33:54.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:54.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.708068+0000", 2023-11-05T08:33:54.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0094216560000000005 2023-11-05T08:33:54.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:54.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.708146+0000", 2023-11-05T08:33:54.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7964000000000001e-05 2023-11-05T08:33:54.990 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.990 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:54.991 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:54.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.991 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.2s4 97/96 ECSubWrite(tid=327, reqid=osd.13.0:327, at_version=97'25, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:54.991 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:50.720266+0000", 2023-11-05T08:33:54.991 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.0607847289999999, 2023-11-05T08:33:54.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0045741080000000003, 2023-11-05T08:33:54.992 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:54.992 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:54.992 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:54.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:54.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.720266+0000", 2023-11-05T08:33:54.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:54.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.720266+0000", 2023-11-05T08:33:54.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:54.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:54.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.720267+0000", 2023-11-05T08:33:54.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2399999999999997e-07 2023-11-05T08:33:54.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:54.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.720300+0000", 2023-11-05T08:33:54.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.307e-05 2023-11-05T08:33:54.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:54.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.720319+0000", 2023-11-05T08:33:54.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9055e-05 2023-11-05T08:33:54.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:54.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.720332+0000", 2023-11-05T08:33:54.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3023999999999999e-05 2023-11-05T08:33:54.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:54.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.720936+0000", 2023-11-05T08:33:54.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060406200000000005 2023-11-05T08:33:54.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:54.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:54.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:54.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.720952+0000", 2023-11-05T08:33:54.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5976000000000001e-05 2023-11-05T08:33:55.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:55.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.724790+0000", 2023-11-05T08:33:55.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038381219999999998 2023-11-05T08:33:55.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:55.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.724840+0000", 2023-11-05T08:33:55.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9975000000000002e-05 2023-11-05T08:33:55.001 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.002 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:55.002 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.002 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.3s3 97/96 ECSubWrite(tid=331, reqid=osd.13.0:329, at_version=0'0, trim_to=0'0, roll_forward_to=97'17))", 2023-11-05T08:33:55.003 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:50.732707+0000", 2023-11-05T08:33:55.003 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.0483440169999998, 2023-11-05T08:33:55.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052692909999999997, 2023-11-05T08:33:55.003 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:55.003 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:55.004 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:55.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:55.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.732707+0000", 2023-11-05T08:33:55.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:55.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.732707+0000", 2023-11-05T08:33:55.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:55.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.732708+0000", 2023-11-05T08:33:55.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5700000000000001e-07 2023-11-05T08:33:55.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:55.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.732722+0000", 2023-11-05T08:33:55.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4253999999999999e-05 2023-11-05T08:33:55.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:55.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.732741+0000", 2023-11-05T08:33:55.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9032e-05 2023-11-05T08:33:55.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:55.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.732753+0000", 2023-11-05T08:33:55.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2284e-05 2023-11-05T08:33:55.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:55.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.732851+0000", 2023-11-05T08:33:55.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7774000000000003e-05 2023-11-05T08:33:55.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:55.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.732868+0000", 2023-11-05T08:33:55.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7501000000000001e-05 2023-11-05T08:33:55.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:55.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.737922+0000", 2023-11-05T08:33:55.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050540369999999999 2023-11-05T08:33:55.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:55.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.737976+0000", 2023-11-05T08:33:55.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3551999999999998e-05 2023-11-05T08:33:55.013 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.013 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:55.013 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.014 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.014 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:55.014 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:33:55.016 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-11-05T08:33:55.016 INFO:tasks.rados.rados.0.smithi090.stdout:325: finishing write tid 1 to smithi09039252-26 2023-11-05T08:33:55.017 INFO:tasks.rados.rados.0.smithi090.stdout:326: finishing rollback tid 1 to smithi09039252-20 2023-11-05T08:33:55.017 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 20 v 28 (ObjNum 289 snap 0 seq_num 0) dirty dne 2023-11-05T08:33:55.017 INFO:tasks.rados.rados.0.smithi090.stdout:325: finishing write tid 2 to smithi09039252-26 2023-11-05T08:33:55.017 INFO:tasks.rados.rados.0.smithi090.stdout:325: finishing write tid 3 to smithi09039252-26 2023-11-05T08:33:55.017 INFO:tasks.rados.rados.0.smithi090.stdout:325: finishing write tid 4 to smithi09039252-26 2023-11-05T08:33:55.018 INFO:tasks.rados.rados.0.smithi090.stdout:325: finishing write tid 5 to smithi09039252-26 2023-11-05T08:33:55.018 INFO:tasks.rados.rados.0.smithi090.stdout:327: expect (ObjNum 109 snap 29 seq_num 109) 2023-11-05T08:33:55.019 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:33:55.013+0000 7fdc5cd58640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:33:55.099 INFO:tasks.rados.rados.0.smithi090.stdout:325: finishing write tid 6 to smithi09039252-26 2023-11-05T08:33:55.099 INFO:tasks.rados.rados.0.smithi090.stdout:325: oid 26 updating version 0 to 61 2023-11-05T08:33:55.099 INFO:tasks.rados.rados.0.smithi090.stdout:325: oid 26 version 61 is already newer than 60 2023-11-05T08:33:55.099 INFO:tasks.rados.rados.0.smithi090.stdout:325: oid 26 updating version 61 to 63 2023-11-05T08:33:55.100 INFO:tasks.rados.rados.0.smithi090.stdout:325: oid 26 version 63 is already newer than 62 2023-11-05T08:33:55.100 INFO:tasks.rados.rados.0.smithi090.stdout:325: oid 26 version 63 is already newer than 59 2023-11-05T08:33:55.100 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 26 v 63 (ObjNum 123 snap 34 seq_num 123) dirty exists 2023-11-05T08:33:55.100 INFO:tasks.rados.rados.0.smithi090.stdout:325: left oid 26 (ObjNum 123 snap 34 seq_num 123) 2023-11-05T08:33:55.100 INFO:tasks.rados.rados.0.smithi090.stdout:328: seq_num 124 ranges {2686976=540672,3227648=720896,3948544=786432,4734976=49152} 2023-11-05T08:33:55.116 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:33:55.114+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:33:55.118 INFO:tasks.rados.rados.0.smithi090.stdout:328: writing smithi09039252-47 from 2686976 to 3227648 tid 1 2023-11-05T08:33:55.124 INFO:tasks.rados.rados.0.smithi090.stdout:328: writing smithi09039252-47 from 3227648 to 3948544 tid 2 2023-11-05T08:33:55.129 INFO:tasks.rados.rados.0.smithi090.stdout:328: writing smithi09039252-47 from 3948544 to 4734976 tid 3 2023-11-05T08:33:55.130 INFO:tasks.rados.rados.0.smithi090.stdout:328: writing smithi09039252-47 from 4734976 to 4784128 tid 4 2023-11-05T08:33:55.130 INFO:tasks.rados.rados.0.smithi090.stdout:322: done (5 left) 2023-11-05T08:33:55.130 INFO:tasks.rados.rados.0.smithi090.stdout:323: done (4 left) 2023-11-05T08:33:55.131 INFO:tasks.rados.rados.0.smithi090.stdout:325: done (3 left) 2023-11-05T08:33:55.131 INFO:tasks.rados.rados.0.smithi090.stdout:326: done (2 left) 2023-11-05T08:33:55.131 INFO:tasks.rados.rados.0.smithi090.stdout:327: done (1 left) 2023-11-05T08:33:55.131 INFO:tasks.rados.rados.0.smithi090.stdout:329: delete oid 38 current snap is 34 2023-11-05T08:33:55.132 INFO:tasks.rados.rados.0.smithi090.stdout:328: finishing write tid 1 to smithi09039252-47 2023-11-05T08:33:55.135 INFO:tasks.rados.rados.0.smithi090.stdout:328: finishing write tid 2 to smithi09039252-47 2023-11-05T08:33:55.136 INFO:tasks.rados.rados.0.smithi090.stdout:329: done (1 left) 2023-11-05T08:33:55.137 INFO:tasks.rados.rados.0.smithi090.stdout:330: read oid 24 snap 32 2023-11-05T08:33:55.137 INFO:tasks.rados.rados.0.smithi090.stdout:330: expect deleted 2023-11-05T08:33:55.137 INFO:tasks.rados.rados.0.smithi090.stdout:331: snap_create 2023-11-05T08:33:55.139 INFO:tasks.rados.rados.0.smithi090.stdout:328: finishing write tid 3 to smithi09039252-47 2023-11-05T08:33:55.140 INFO:tasks.rados.rados.0.smithi090.stdout:328: finishing write tid 4 to smithi09039252-47 2023-11-05T08:33:55.141 INFO:tasks.rados.rados.0.smithi090.stdout:328: finishing write tid 5 to smithi09039252-47 2023-11-05T08:33:55.144 INFO:tasks.rados.rados.0.smithi090.stdout:328: finishing write tid 6 to smithi09039252-47 2023-11-05T08:33:55.144 INFO:tasks.rados.rados.0.smithi090.stdout:328: oid 47 updating version 0 to 21 2023-11-05T08:33:55.144 INFO:tasks.rados.rados.0.smithi090.stdout:328: oid 47 updating version 21 to 22 2023-11-05T08:33:55.145 INFO:tasks.rados.rados.0.smithi090.stdout:328: oid 47 updating version 22 to 23 2023-11-05T08:33:55.145 INFO:tasks.rados.rados.0.smithi090.stdout:328: oid 47 updating version 23 to 24 2023-11-05T08:33:55.145 INFO:tasks.rados.rados.0.smithi090.stdout:328: oid 47 updating version 24 to 25 2023-11-05T08:33:55.145 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 47 v 25 (ObjNum 124 snap 34 seq_num 124) dirty exists 2023-11-05T08:33:55.145 INFO:tasks.rados.rados.0.smithi090.stdout:328: left oid 47 (ObjNum 124 snap 34 seq_num 124) 2023-11-05T08:33:55.153 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:33:55.153 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-11-05T08:33:55.154 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-11-05T08:33:55.154 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:33:55.167 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-11-05T08:33:55.216 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:33:55.212+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:33:55.317 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:33:55.316+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:33:55.334 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:33:55.334 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-11-05T08:33:55.334 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-11-05T08:33:55.334 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-11-05T08:33:55.335 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:33:55.347 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-11-05T08:33:55.419 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:33:55.412+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:33:55.473 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:33:55.473 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-11-05T08:33:55.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-11-05T08:33:55.474 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-11-05T08:33:55.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.474 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.15s2 95/94 [PushOp(3:a88fa0fb:::smithi09039252-1:head, version: 32'2, data_included: [0~8192], data_size: 8192, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:a88fa0fb:::smithi09039252-1:head@32'2, size: 32768, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 32768, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2023-11-05T08:33:55.474 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:48.724761+0000", 2023-11-05T08:33:55.475 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 6.7446958519999995, 2023-11-05T08:33:55.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036152630000000001, 2023-11-05T08:33:55.475 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:55.475 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:55.475 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:55.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:55.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:48.724761+0000", 2023-11-05T08:33:55.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:55.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:48.724761+0000", 2023-11-05T08:33:55.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:55.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:48.724762+0000", 2023-11-05T08:33:55.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.576e-06 2023-11-05T08:33:55.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:55.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:48.724822+0000", 2023-11-05T08:33:55.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0041999999999999e-05 2023-11-05T08:33:55.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:55.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:48.724866+0000", 2023-11-05T08:33:55.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3979e-05 2023-11-05T08:33:55.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:55.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:48.724913+0000", 2023-11-05T08:33:55.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6529000000000001e-05 2023-11-05T08:33:55.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:55.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:48.725132+0000", 2023-11-05T08:33:55.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0002187 2023-11-05T08:33:55.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:55.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:48.728376+0000", 2023-11-05T08:33:55.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032444370000000002 2023-11-05T08:33:55.483 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.483 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:55.483 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.484 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.ds1 95/94 [PushOp(3:bf7e600d:::smithi09039252-23:10, version: 52'15, data_included: [262144~86016], data_size: 86016, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:bf7e600d:::smithi09039252-23:10@52'15, size: 1392640, copy_subset: [], clone_subset: {}, snapset: 27=[]:{3: [3], 10: [7]}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1392640, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2023-11-05T08:33:55.484 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:48.739116+0000", 2023-11-05T08:33:55.484 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 6.7303401730000001, 2023-11-05T08:33:55.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043537640000000004, 2023-11-05T08:33:55.484 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:55.485 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:55.485 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:55.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:55.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:48.739116+0000", 2023-11-05T08:33:55.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:55.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:48.739116+0000", 2023-11-05T08:33:55.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:55.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:48.739118+0000", 2023-11-05T08:33:55.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.544e-06 2023-11-05T08:33:55.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:55.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:48.739254+0000", 2023-11-05T08:33:55.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013563 2023-11-05T08:33:55.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:55.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:48.739292+0000", 2023-11-05T08:33:55.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7957000000000003e-05 2023-11-05T08:33:55.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:55.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:48.739325+0000", 2023-11-05T08:33:55.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3339e-05 2023-11-05T08:33:55.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:55.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:48.739534+0000", 2023-11-05T08:33:55.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020938700000000001 2023-11-05T08:33:55.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:55.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:48.743470+0000", 2023-11-05T08:33:55.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039359069999999998 2023-11-05T08:33:55.492 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.492 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:55.493 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.493 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.14s0 95/94 [PushOp(3:292b6c2e:::smithi09039252-40:7, version: 49'15, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:292b6c2e:::smithi09039252-40:7@49'15, size: 1671168, copy_subset: [], clone_subset: {}, snapset: 15=[]:{7: [7]}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2023-11-05T08:33:55.493 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:48.742791+0000", 2023-11-05T08:33:55.494 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 6.7266651020000001, 2023-11-05T08:33:55.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036366419999999998, 2023-11-05T08:33:55.494 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:55.494 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:55.494 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:55.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:55.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:48.742791+0000", 2023-11-05T08:33:55.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:55.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:48.742791+0000", 2023-11-05T08:33:55.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:55.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:48.742794+0000", 2023-11-05T08:33:55.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0660000000000002e-06 2023-11-05T08:33:55.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:55.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:48.743348+0000", 2023-11-05T08:33:55.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00055418299999999998 2023-11-05T08:33:55.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:55.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:48.743416+0000", 2023-11-05T08:33:55.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8538e-05 2023-11-05T08:33:55.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:55.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:48.743463+0000", 2023-11-05T08:33:55.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6381999999999997e-05 2023-11-05T08:33:55.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:55.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:48.743651+0000", 2023-11-05T08:33:55.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000188102 2023-11-05T08:33:55.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:55.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:48.746428+0000", 2023-11-05T08:33:55.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0027773709999999998 2023-11-05T08:33:55.502 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.502 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:55.502 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.503 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.14s0 97/96 ECSubWrite(tid=7, reqid=osd.9.0:7, at_version=97'21, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:55.503 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:50.680773+0000", 2023-11-05T08:33:55.503 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.7886839139999999, 2023-11-05T08:33:55.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.005781614, 2023-11-05T08:33:55.504 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:55.504 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:55.504 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:55.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:55.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.680773+0000", 2023-11-05T08:33:55.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:55.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.680773+0000", 2023-11-05T08:33:55.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:55.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.680774+0000", 2023-11-05T08:33:55.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4100000000000001e-06 2023-11-05T08:33:55.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:55.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.680826+0000", 2023-11-05T08:33:55.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1635000000000003e-05 2023-11-05T08:33:55.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:55.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.680851+0000", 2023-11-05T08:33:55.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4853000000000001e-05 2023-11-05T08:33:55.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:55.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.680872+0000", 2023-11-05T08:33:55.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0965999999999999e-05 2023-11-05T08:33:55.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:55.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.681022+0000", 2023-11-05T08:33:55.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000150042 2023-11-05T08:33:55.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:55.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.681040+0000", 2023-11-05T08:33:55.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8182999999999999e-05 2023-11-05T08:33:55.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:55.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.686494+0000", 2023-11-05T08:33:55.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0054546109999999998 2023-11-05T08:33:55.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:55.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.686554+0000", 2023-11-05T08:33:55.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9914000000000003e-05 2023-11-05T08:33:55.514 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.514 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:55.514 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.515 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.17s1 97/96 ECSubWrite(tid=56, reqid=osd.5.0:56, at_version=97'23, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:55.515 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:50.682777+0000", 2023-11-05T08:33:55.515 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.7866792890000003, 2023-11-05T08:33:55.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0082614190000000007, 2023-11-05T08:33:55.515 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:55.516 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:55.516 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:55.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:55.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.682777+0000", 2023-11-05T08:33:55.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:55.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.682777+0000", 2023-11-05T08:33:55.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:55.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.682778+0000", 2023-11-05T08:33:55.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1969999999999999e-06 2023-11-05T08:33:55.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:55.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.682873+0000", 2023-11-05T08:33:55.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4953999999999999e-05 2023-11-05T08:33:55.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:55.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.682902+0000", 2023-11-05T08:33:55.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.862e-05 2023-11-05T08:33:55.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:55.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.682925+0000", 2023-11-05T08:33:55.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2449000000000001e-05 2023-11-05T08:33:55.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:55.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.683077+0000", 2023-11-05T08:33:55.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015288000000000001 2023-11-05T08:33:55.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:55.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.683100+0000", 2023-11-05T08:33:55.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2515999999999999e-05 2023-11-05T08:33:55.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:55.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.690978+0000", 2023-11-05T08:33:55.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0078779750000000006 2023-11-05T08:33:55.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:55.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.691039+0000", 2023-11-05T08:33:55.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0828000000000001e-05 2023-11-05T08:33:55.525 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.525 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:55.526 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.526 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.17s1 97/96 ECSubWrite(tid=57, reqid=osd.5.0:57, at_version=97'25, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:55.526 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:50.687497+0000", 2023-11-05T08:33:55.527 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.7819598240000003, 2023-11-05T08:33:55.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0078308790000000007, 2023-11-05T08:33:55.527 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:55.527 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:55.527 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:55.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:55.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.687497+0000", 2023-11-05T08:33:55.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:55.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.687497+0000", 2023-11-05T08:33:55.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:55.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.687498+0000", 2023-11-05T08:33:55.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.234e-06 2023-11-05T08:33:55.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:55.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.687582+0000", 2023-11-05T08:33:55.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4433999999999998e-05 2023-11-05T08:33:55.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:55.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.687609+0000", 2023-11-05T08:33:55.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6526e-05 2023-11-05T08:33:55.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:55.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.687631+0000", 2023-11-05T08:33:55.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2487e-05 2023-11-05T08:33:55.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:55.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.687766+0000", 2023-11-05T08:33:55.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000134423 2023-11-05T08:33:55.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:55.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.687782+0000", 2023-11-05T08:33:55.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6492999999999999e-05 2023-11-05T08:33:55.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:55.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.695260+0000", 2023-11-05T08:33:55.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0074778359999999999 2023-11-05T08:33:55.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:55.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.695328+0000", 2023-11-05T08:33:55.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7446000000000004e-05 2023-11-05T08:33:55.537 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.537 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:55.537 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.537 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.14s0 97/96 ECSubWrite(tid=8, reqid=osd.9.0:7, at_version=0'0, trim_to=0'0, roll_forward_to=97'21))", 2023-11-05T08:33:55.538 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:50.690765+0000", 2023-11-05T08:33:55.538 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.7786917039999999, 2023-11-05T08:33:55.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0059671919999999996, 2023-11-05T08:33:55.538 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:55.538 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:55.539 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:55.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:55.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.690765+0000", 2023-11-05T08:33:55.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:55.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.690765+0000", 2023-11-05T08:33:55.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:55.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.690766+0000", 2023-11-05T08:33:55.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2500000000000001e-06 2023-11-05T08:33:55.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:55.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.690791+0000", 2023-11-05T08:33:55.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4672e-05 2023-11-05T08:33:55.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:55.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.690821+0000", 2023-11-05T08:33:55.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9963000000000001e-05 2023-11-05T08:33:55.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:55.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.690844+0000", 2023-11-05T08:33:55.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3672999999999998e-05 2023-11-05T08:33:55.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:55.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.691011+0000", 2023-11-05T08:33:55.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000166603 2023-11-05T08:33:55.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:55.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.691034+0000", 2023-11-05T08:33:55.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2762e-05 2023-11-05T08:33:55.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:55.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.696660+0000", 2023-11-05T08:33:55.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.005625817 2023-11-05T08:33:55.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:55.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.696732+0000", 2023-11-05T08:33:55.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2452000000000004e-05 2023-11-05T08:33:55.548 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.548 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:55.548 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.549 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.as0 97/96 ECSubWrite(tid=106, reqid=osd.7.0:106, at_version=97'32, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:55.549 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:50.700380+0000", 2023-11-05T08:33:55.549 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.7690763369999996, 2023-11-05T08:33:55.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010907586, 2023-11-05T08:33:55.550 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:55.550 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:55.550 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:55.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:55.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.700380+0000", 2023-11-05T08:33:55.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:55.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.700380+0000", 2023-11-05T08:33:55.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:55.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.700381+0000", 2023-11-05T08:33:55.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9699999999999995e-07 2023-11-05T08:33:55.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:55.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.700468+0000", 2023-11-05T08:33:55.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6952000000000004e-05 2023-11-05T08:33:55.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:55.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.700492+0000", 2023-11-05T08:33:55.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4051000000000001e-05 2023-11-05T08:33:55.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:55.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.700510+0000", 2023-11-05T08:33:55.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8187999999999999e-05 2023-11-05T08:33:55.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:55.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.701459+0000", 2023-11-05T08:33:55.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00094915799999999997 2023-11-05T08:33:55.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:55.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.701479+0000", 2023-11-05T08:33:55.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0078999999999999e-05 2023-11-05T08:33:55.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:55.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.711221+0000", 2023-11-05T08:33:55.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0097411570000000003 2023-11-05T08:33:55.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:55.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.711288+0000", 2023-11-05T08:33:55.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7304000000000005e-05 2023-11-05T08:33:55.559 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.559 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:55.560 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.560 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.as0 97/96 ECSubWrite(tid=107, reqid=osd.7.0:107, at_version=97'34, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:55.560 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:50.704209+0000", 2023-11-05T08:33:55.561 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.7652473300000002, 2023-11-05T08:33:55.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0084617830000000005, 2023-11-05T08:33:55.561 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:55.561 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:55.561 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:55.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:55.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.704209+0000", 2023-11-05T08:33:55.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:55.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.704209+0000", 2023-11-05T08:33:55.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:55.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.704210+0000", 2023-11-05T08:33:55.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1939999999999999e-06 2023-11-05T08:33:55.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:55.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.704677+0000", 2023-11-05T08:33:55.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046684100000000001 2023-11-05T08:33:55.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:55.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.707527+0000", 2023-11-05T08:33:55.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028498389999999998 2023-11-05T08:33:55.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:55.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.707545+0000", 2023-11-05T08:33:55.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7935e-05 2023-11-05T08:33:55.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:55.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.707629+0000", 2023-11-05T08:33:55.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4325000000000003e-05 2023-11-05T08:33:55.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:55.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.707644+0000", 2023-11-05T08:33:55.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4617000000000001e-05 2023-11-05T08:33:55.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:55.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.712627+0000", 2023-11-05T08:33:55.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049833630000000002 2023-11-05T08:33:55.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:55.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.712671+0000", 2023-11-05T08:33:55.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3668999999999998e-05 2023-11-05T08:33:55.570 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.570 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:55.571 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.571 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.17s1 97/96 ECSubWrite(tid=58, reqid=osd.5.0:57, at_version=0'0, trim_to=0'0, roll_forward_to=97'25))", 2023-11-05T08:33:55.571 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:50.707986+0000", 2023-11-05T08:33:55.572 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.7614710129999995, 2023-11-05T08:33:55.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0046376669999999998, 2023-11-05T08:33:55.572 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:55.572 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:55.572 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:55.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:55.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.707986+0000", 2023-11-05T08:33:55.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:55.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.707986+0000", 2023-11-05T08:33:55.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:55.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.707987+0000", 2023-11-05T08:33:55.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5000000000000001e-07 2023-11-05T08:33:55.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:55.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.708002+0000", 2023-11-05T08:33:55.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5491000000000001e-05 2023-11-05T08:33:55.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:55.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.708023+0000", 2023-11-05T08:33:55.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0494999999999999e-05 2023-11-05T08:33:55.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:55.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.708046+0000", 2023-11-05T08:33:55.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3407000000000001e-05 2023-11-05T08:33:55.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:55.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.708238+0000", 2023-11-05T08:33:55.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019166800000000001 2023-11-05T08:33:55.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:55.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.708253+0000", 2023-11-05T08:33:55.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5136999999999999e-05 2023-11-05T08:33:55.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:55.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.712575+0000", 2023-11-05T08:33:55.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043220719999999997 2023-11-05T08:33:55.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:55.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.712623+0000", 2023-11-05T08:33:55.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8446999999999999e-05 2023-11-05T08:33:55.581 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.581 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:55.582 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.582 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.as0 97/96 ECSubWrite(tid=108, reqid=osd.7.0:107, at_version=0'0, trim_to=0'0, roll_forward_to=97'34))", 2023-11-05T08:33:55.582 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:50.713623+0000", 2023-11-05T08:33:55.582 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.7558338510000002, 2023-11-05T08:33:55.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003057293, 2023-11-05T08:33:55.583 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:55.583 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:55.583 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:55.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:55.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.713623+0000", 2023-11-05T08:33:55.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:55.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.713623+0000", 2023-11-05T08:33:55.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:55.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.713624+0000", 2023-11-05T08:33:55.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.122e-06 2023-11-05T08:33:55.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:55.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.713642+0000", 2023-11-05T08:33:55.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7870999999999999e-05 2023-11-05T08:33:55.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:55.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.713661+0000", 2023-11-05T08:33:55.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9134999999999999e-05 2023-11-05T08:33:55.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:55.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.713673+0000", 2023-11-05T08:33:55.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2054e-05 2023-11-05T08:33:55.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:55.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.713847+0000", 2023-11-05T08:33:55.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017419700000000001 2023-11-05T08:33:55.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:55.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.713866+0000", 2023-11-05T08:33:55.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9273999999999999e-05 2023-11-05T08:33:55.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:55.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.716638+0000", 2023-11-05T08:33:55.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0027711210000000001 2023-11-05T08:33:55.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:55.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.716680+0000", 2023-11-05T08:33:55.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2518999999999997e-05 2023-11-05T08:33:55.592 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.592 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:55.593 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.593 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.2s2 97/96 ECSubWrite(tid=326, reqid=osd.13.0:326, at_version=97'23, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:55.593 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:50.720090+0000", 2023-11-05T08:33:55.594 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.7493665700000003, 2023-11-05T08:33:55.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0056196509999999998, 2023-11-05T08:33:55.594 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:55.594 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:55.594 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:55.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:55.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.720090+0000", 2023-11-05T08:33:55.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:55.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.720090+0000", 2023-11-05T08:33:55.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:55.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.720091+0000", 2023-11-05T08:33:55.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.361e-06 2023-11-05T08:33:55.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:55.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.720128+0000", 2023-11-05T08:33:55.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6396000000000002e-05 2023-11-05T08:33:55.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:55.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.720154+0000", 2023-11-05T08:33:55.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6078999999999999e-05 2023-11-05T08:33:55.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:55.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.720167+0000", 2023-11-05T08:33:55.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3580999999999999e-05 2023-11-05T08:33:55.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:55.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.720266+0000", 2023-11-05T08:33:55.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8684000000000006e-05 2023-11-05T08:33:55.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:55.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.720285+0000", 2023-11-05T08:33:55.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8799000000000001e-05 2023-11-05T08:33:55.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:55.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.725640+0000", 2023-11-05T08:33:55.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053554170000000003 2023-11-05T08:33:55.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:55.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.725710+0000", 2023-11-05T08:33:55.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9333999999999997e-05 2023-11-05T08:33:55.603 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.603 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:55.603 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.604 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.2s2 97/96 ECSubWrite(tid=327, reqid=osd.13.0:327, at_version=97'25, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:55.604 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:50.720200+0000", 2023-11-05T08:33:55.604 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.7492564220000002, 2023-11-05T08:33:55.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0071118170000000003, 2023-11-05T08:33:55.604 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:55.605 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:55.605 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:55.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:55.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.720200+0000", 2023-11-05T08:33:55.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:55.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.720200+0000", 2023-11-05T08:33:55.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:55.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.720201+0000", 2023-11-05T08:33:55.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9699999999999996e-07 2023-11-05T08:33:55.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:55.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.720257+0000", 2023-11-05T08:33:55.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6203999999999999e-05 2023-11-05T08:33:55.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:55.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.720273+0000", 2023-11-05T08:33:55.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6209999999999999e-05 2023-11-05T08:33:55.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:55.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.720286+0000", 2023-11-05T08:33:55.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3259e-05 2023-11-05T08:33:55.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:55.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.721100+0000", 2023-11-05T08:33:55.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00081335999999999995 2023-11-05T08:33:55.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:55.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.721114+0000", 2023-11-05T08:33:55.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4036000000000001e-05 2023-11-05T08:33:55.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:55.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.727256+0000", 2023-11-05T08:33:55.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061419539999999998 2023-11-05T08:33:55.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:55.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.727312+0000", 2023-11-05T08:33:55.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6196999999999997e-05 2023-11-05T08:33:55.614 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.614 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:55.615 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.615 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.as0 97/96 ECSubWrite(tid=109, reqid=osd.7.0:109, at_version=97'36, trim_to=0'0, roll_forward_to=97'34))", 2023-11-05T08:33:55.615 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:50.724982+0000", 2023-11-05T08:33:55.615 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.7444742670000002, 2023-11-05T08:33:55.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038473470000000001, 2023-11-05T08:33:55.616 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:55.616 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:55.616 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:55.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:55.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.724982+0000", 2023-11-05T08:33:55.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:55.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.724982+0000", 2023-11-05T08:33:55.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:55.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.724983+0000", 2023-11-05T08:33:55.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0750000000000001e-06 2023-11-05T08:33:55.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:55.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.725013+0000", 2023-11-05T08:33:55.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9407000000000002e-05 2023-11-05T08:33:55.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:55.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.725036+0000", 2023-11-05T08:33:55.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3119e-05 2023-11-05T08:33:55.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:55.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.725049+0000", 2023-11-05T08:33:55.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2753e-05 2023-11-05T08:33:55.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:55.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.725211+0000", 2023-11-05T08:33:55.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000162263 2023-11-05T08:33:55.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:55.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.725225+0000", 2023-11-05T08:33:55.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3732e-05 2023-11-05T08:33:55.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:55.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.728782+0000", 2023-11-05T08:33:55.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0035568679999999999 2023-11-05T08:33:55.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:55.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.728830+0000", 2023-11-05T08:33:55.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8130000000000002e-05 2023-11-05T08:33:55.626 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.626 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:55.626 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.626 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.3s6 97/96 ECSubWrite(tid=329, reqid=osd.13.0:329, at_version=97'17, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:33:55.627 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:50.725234+0000", 2023-11-05T08:33:55.627 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.7442225379999998, 2023-11-05T08:33:55.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036310679999999999, 2023-11-05T08:33:55.627 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:55.627 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:55.628 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:55.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:55.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.725234+0000", 2023-11-05T08:33:55.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:55.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.725234+0000", 2023-11-05T08:33:55.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:55.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.725235+0000", 2023-11-05T08:33:55.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.277e-06 2023-11-05T08:33:55.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:55.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.725272+0000", 2023-11-05T08:33:55.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6365999999999997e-05 2023-11-05T08:33:55.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:55.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.725289+0000", 2023-11-05T08:33:55.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7079e-05 2023-11-05T08:33:55.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:55.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.725303+0000", 2023-11-05T08:33:55.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4061e-05 2023-11-05T08:33:55.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:55.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.725400+0000", 2023-11-05T08:33:55.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7405000000000007e-05 2023-11-05T08:33:55.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:55.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.725414+0000", 2023-11-05T08:33:55.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3638000000000001e-05 2023-11-05T08:33:55.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:55.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.728814+0000", 2023-11-05T08:33:55.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033999329999999999 2023-11-05T08:33:55.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:55.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.728865+0000", 2023-11-05T08:33:55.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1309e-05 2023-11-05T08:33:55.637 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.637 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:55.637 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.638 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.as0 97/96 ECSubWrite(tid=110, reqid=osd.7.0:110, at_version=97'38, trim_to=0'0, roll_forward_to=97'34))", 2023-11-05T08:33:55.638 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:50.726874+0000", 2023-11-05T08:33:55.638 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.7425825980000003, 2023-11-05T08:33:55.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004620781, 2023-11-05T08:33:55.639 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:55.639 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:55.639 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:55.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:55.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.726874+0000", 2023-11-05T08:33:55.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:55.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.726874+0000", 2023-11-05T08:33:55.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:55.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.726875+0000", 2023-11-05T08:33:55.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.16e-07 2023-11-05T08:33:55.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:55.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.726904+0000", 2023-11-05T08:33:55.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9098000000000002e-05 2023-11-05T08:33:55.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:55.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.726918+0000", 2023-11-05T08:33:55.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4425e-05 2023-11-05T08:33:55.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:55.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.726929+0000", 2023-11-05T08:33:55.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0137e-05 2023-11-05T08:33:55.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:55.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.727013+0000", 2023-11-05T08:33:55.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4641000000000005e-05 2023-11-05T08:33:55.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:55.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.727027+0000", 2023-11-05T08:33:55.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4174e-05 2023-11-05T08:33:55.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:55.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.731429+0000", 2023-11-05T08:33:55.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0044011299999999996 2023-11-05T08:33:55.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:55.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.731495+0000", 2023-11-05T08:33:55.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6260000000000006e-05 2023-11-05T08:33:55.650 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.651 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:55.651 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.651 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.2s2 97/96 ECSubWrite(tid=330, reqid=osd.13.0:327, at_version=0'0, trim_to=0'0, roll_forward_to=97'25))", 2023-11-05T08:33:55.651 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:50.729461+0000", 2023-11-05T08:33:55.652 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.7399953440000004, 2023-11-05T08:33:55.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029486239999999999, 2023-11-05T08:33:55.652 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:55.652 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:55.652 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:55.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:55.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.729461+0000", 2023-11-05T08:33:55.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:55.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.729461+0000", 2023-11-05T08:33:55.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:55.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.729462+0000", 2023-11-05T08:33:55.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1209999999999999e-06 2023-11-05T08:33:55.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:55.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.729493+0000", 2023-11-05T08:33:55.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0587999999999999e-05 2023-11-05T08:33:55.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:55.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.729518+0000", 2023-11-05T08:33:55.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5333999999999998e-05 2023-11-05T08:33:55.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:55.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.729532+0000", 2023-11-05T08:33:55.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3767e-05 2023-11-05T08:33:55.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:55.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.729612+0000", 2023-11-05T08:33:55.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0175999999999999e-05 2023-11-05T08:33:55.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:55.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.729625+0000", 2023-11-05T08:33:55.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2274e-05 2023-11-05T08:33:55.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:55.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.732364+0000", 2023-11-05T08:33:55.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0027398819999999999 2023-11-05T08:33:55.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:55.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.732410+0000", 2023-11-05T08:33:55.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5482000000000003e-05 2023-11-05T08:33:55.661 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.661 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:55.662 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.662 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.3s6 97/96 ECSubWrite(tid=331, reqid=osd.13.0:329, at_version=0'0, trim_to=0'0, roll_forward_to=97'17))", 2023-11-05T08:33:55.662 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:50.732464+0000", 2023-11-05T08:33:55.663 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.736992893, 2023-11-05T08:33:55.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052222149999999997, 2023-11-05T08:33:55.663 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:55.663 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:55.663 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:55.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:55.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.732464+0000", 2023-11-05T08:33:55.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:55.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.732464+0000", 2023-11-05T08:33:55.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:55.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.732465+0000", 2023-11-05T08:33:55.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4799999999999997e-07 2023-11-05T08:33:55.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:55.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.732476+0000", 2023-11-05T08:33:55.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.114e-05 2023-11-05T08:33:55.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:55.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.732491+0000", 2023-11-05T08:33:55.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5165e-05 2023-11-05T08:33:55.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:55.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.732501+0000", 2023-11-05T08:33:55.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0034000000000001e-05 2023-11-05T08:33:55.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:55.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.732566+0000", 2023-11-05T08:33:55.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5610000000000004e-05 2023-11-05T08:33:55.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:55.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.732575+0000", 2023-11-05T08:33:55.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8380000000000008e-06 2023-11-05T08:33:55.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:55.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.737629+0000", 2023-11-05T08:33:55.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050540519999999999 2023-11-05T08:33:55.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:55.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.737686+0000", 2023-11-05T08:33:55.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6527999999999998e-05 2023-11-05T08:33:55.672 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.672 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:55.672 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.673 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.as0 97/96 ECSubWrite(tid=111, reqid=osd.7.0:110, at_version=0'0, trim_to=0'0, roll_forward_to=97'38))", 2023-11-05T08:33:55.673 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:50.735294+0000", 2023-11-05T08:33:55.673 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.7341624739999997, 2023-11-05T08:33:55.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053153330000000002, 2023-11-05T08:33:55.673 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:55.674 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:55.674 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:55.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:55.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.735294+0000", 2023-11-05T08:33:55.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:55.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.735294+0000", 2023-11-05T08:33:55.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:55.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.735295+0000", 2023-11-05T08:33:55.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1039999999999999e-06 2023-11-05T08:33:55.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:55.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.735313+0000", 2023-11-05T08:33:55.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7416999999999999e-05 2023-11-05T08:33:55.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:55.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.735336+0000", 2023-11-05T08:33:55.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3753000000000001e-05 2023-11-05T08:33:55.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:55.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.735350+0000", 2023-11-05T08:33:55.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3909e-05 2023-11-05T08:33:55.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:55.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.735477+0000", 2023-11-05T08:33:55.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000127176 2023-11-05T08:33:55.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:55.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.735494+0000", 2023-11-05T08:33:55.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6232999999999999e-05 2023-11-05T08:33:55.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:55.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.740564+0000", 2023-11-05T08:33:55.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050700720000000001 2023-11-05T08:33:55.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:55.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.740609+0000", 2023-11-05T08:33:55.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5668999999999999e-05 2023-11-05T08:33:55.683 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.683 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:55.683 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.684 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.17s1 101/96 ECSubWrite(tid=61, reqid=osd.5.0:61, at_version=101'29, trim_to=0'0, roll_forward_to=99'27))", 2023-11-05T08:33:55.684 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:54.752251+0000", 2023-11-05T08:33:55.684 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.71720605900000001, 2023-11-05T08:33:55.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029454239999999999, 2023-11-05T08:33:55.684 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:33:55.684 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:33:55.685 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:33:55.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:33:55.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:54.752251+0000", 2023-11-05T08:33:55.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:33:55.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:54.752251+0000", 2023-11-05T08:33:55.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:33:55.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:33:55.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:54.752251+0000", 2023-11-05T08:33:55.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0699999999999999e-07 2023-11-05T08:33:55.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:33:55.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:54.752273+0000", 2023-11-05T08:33:55.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2531000000000002e-05 2023-11-05T08:33:55.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:33:55.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:54.752295+0000", 2023-11-05T08:33:55.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2201999999999999e-05 2023-11-05T08:33:55.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:33:55.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:54.752305+0000", 2023-11-05T08:33:55.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4360000000000008e-06 2023-11-05T08:33:55.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:33:55.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:54.752370+0000", 2023-11-05T08:33:55.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4896999999999999e-05 2023-11-05T08:33:55.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:33:55.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:54.752378+0000", 2023-11-05T08:33:55.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1110000000000008e-06 2023-11-05T08:33:55.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:33:55.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:54.755143+0000", 2023-11-05T08:33:55.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0027654670000000002 2023-11-05T08:33:55.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:33:55.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:33:55.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:33:55.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:54.755196+0000", 2023-11-05T08:33:55.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2573e-05 2023-11-05T08:33:55.693 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.693 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:55.694 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.694 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:33:55.694 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:33:55.694 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:33:55.696 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-11-05T08:33:55.696 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:33:55.692+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:33:55.737 INFO:teuthology.orchestra.run.smithi002.stderr:reweighted osd.8 to 1 (10000) 2023-11-05T08:33:55.745 INFO:tasks.rados.rados.0.smithi090.stdout:328: done (2 left) 2023-11-05T08:33:55.745 INFO:tasks.rados.rados.0.smithi090.stdout:330: done (1 left) 2023-11-05T08:33:55.746 INFO:tasks.rados.rados.0.smithi090.stdout:331: done (0 left) 2023-11-05T08:33:55.746 INFO:tasks.rados.rados.0.smithi090.stdout:332: read oid 20 snap -1 2023-11-05T08:33:55.746 INFO:tasks.rados.rados.0.smithi090.stdout:332: expect deleted 2023-11-05T08:33:55.746 INFO:tasks.rados.rados.0.smithi090.stdout:333: read oid 38 snap -1 2023-11-05T08:33:55.746 INFO:tasks.rados.rados.0.smithi090.stdout:333: expect deleted 2023-11-05T08:33:55.747 INFO:tasks.rados.rados.0.smithi090.stdout:334: copy_from oid 36 from oid 15 current snap is 35 2023-11-05T08:33:55.747 INFO:tasks.rados.rados.0.smithi090.stdout:335: setattr oid 30 current snap is 35 2023-11-05T08:33:55.748 INFO:tasks.rados.rados.0.smithi090.stdout:336: rollback oid 27 current snap is 35 2023-11-05T08:33:55.748 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 27 to 34 2023-11-05T08:33:55.748 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 7 current snap is 35 2023-11-05T08:33:55.748 INFO:tasks.rados.rados.0.smithi090.stdout:337: seq_num 126 ranges {0=573440,573440=622592,1196032=622592,1818624=311296} 2023-11-05T08:33:55.753 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-11-05T08:33:55.754 INFO:tasks.rados.rados.0.smithi090.stdout:337: writing smithi09039252-7 from 0 to 573440 tid 1 2023-11-05T08:33:55.760 INFO:tasks.rados.rados.0.smithi090.stdout:337: writing smithi09039252-7 from 573440 to 1196032 tid 2 2023-11-05T08:33:55.767 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2023-11-05T08:33:55.767 INFO:tasks.rados.rados.0.smithi090.stdout:337: writing smithi09039252-7 from 1196032 to 1818624 tid 3 2023-11-05T08:33:55.771 INFO:tasks.rados.rados.0.smithi090.stdout:337: writing smithi09039252-7 from 1818624 to 2129920 tid 4 2023-11-05T08:33:55.771 INFO:tasks.rados.rados.0.smithi090.stdout:338: snap_create 2023-11-05T08:33:55.771 INFO:tasks.rados.rados.0.smithi090.stdout:336: finishing rollback tid 1 to smithi09039252-27 2023-11-05T08:33:55.771 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 27 v 0 (ObjNum 29969 snap 0 seq_num 2669769280) dirty dne 2023-11-05T08:33:55.772 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 30 v 65 (ObjNum 109 snap 29 seq_num 109) dirty exists 2023-11-05T08:33:55.772 INFO:tasks.rados.rados.0.smithi090.stdout:334: finishing copy_from racing read to smithi09039252-36 2023-11-05T08:33:55.773 INFO:tasks.rados.rados.0.smithi090.stdout:334: finishing copy_from to smithi09039252-36 2023-11-05T08:33:55.774 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 36 v 30 (ObjNum 17 snap 0 seq_num 17) dirty exists 2023-11-05T08:33:55.777 INFO:tasks.rados.rados.0.smithi090.stdout:337: finishing write tid 1 to smithi09039252-7 2023-11-05T08:33:55.778 INFO:tasks.rados.rados.0.smithi090.stdout:337: finishing write tid 2 to smithi09039252-7 2023-11-05T08:33:55.782 INFO:tasks.rados.rados.0.smithi090.stdout:337: finishing write tid 3 to smithi09039252-7 2023-11-05T08:33:55.785 INFO:tasks.rados.rados.0.smithi090.stdout:337: finishing write tid 4 to smithi09039252-7 2023-11-05T08:33:55.786 INFO:tasks.rados.rados.0.smithi090.stdout:337: finishing write tid 5 to smithi09039252-7 2023-11-05T08:33:55.789 INFO:tasks.rados.rados.0.smithi090.stdout:337: finishing write tid 6 to smithi09039252-7 2023-11-05T08:33:55.789 INFO:tasks.rados.rados.0.smithi090.stdout:337: oid 7 updating version 0 to 28 2023-11-05T08:33:55.790 INFO:tasks.rados.rados.0.smithi090.stdout:337: oid 7 updating version 28 to 29 2023-11-05T08:33:55.790 INFO:tasks.rados.rados.0.smithi090.stdout:337: oid 7 version 29 is already newer than 27 2023-11-05T08:33:55.790 INFO:tasks.rados.rados.0.smithi090.stdout:337: oid 7 version 29 is already newer than 26 2023-11-05T08:33:55.790 INFO:tasks.rados.rados.0.smithi090.stdout:337: oid 7 updating version 29 to 30 2023-11-05T08:33:55.790 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 7 v 30 (ObjNum 126 snap 35 seq_num 126) dirty exists 2023-11-05T08:33:55.791 INFO:tasks.rados.rados.0.smithi090.stdout:337: left oid 7 (ObjNum 126 snap 35 seq_num 126) 2023-11-05T08:33:55.796 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:33:55.792+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:33:55.826 INFO:teuthology.orchestra.run.smithi002.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-05T08:33:55.828 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-05T08:33:55.829 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-11-05T08:33:55.924 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:33:55.921+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:33:55.980 INFO:teuthology.orchestra.run.smithi002.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-05T08:33:55.983 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-05T08:33:55.983 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-11-05T08:33:56.028 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:33:56.025+0000 7fdc5cd58640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:33:56.099 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:33:56.097+0000 7fdc5cd58640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:33:56.131 INFO:teuthology.orchestra.run.smithi002.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-05T08:33:56.133 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-05T08:33:56.134 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-11-05T08:33:56.201 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:33:56.196+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:33:56.302 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:33:56.298+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:33:56.304 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:33:56.304 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [], 2023-11-05T08:33:56.304 INFO:teuthology.orchestra.run.smithi002.stdout: "num_ops": 0 2023-11-05T08:33:56.304 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:33:56.318 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-11-05T08:33:56.328 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-05T08:33:56.402 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:56.400+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:56.445 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:33:56.445 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [], 2023-11-05T08:33:56.446 INFO:teuthology.orchestra.run.smithi002.stdout: "complaint_time": 30, 2023-11-05T08:33:56.446 INFO:teuthology.orchestra.run.smithi002.stdout: "num_blocked_ops": 0 2023-11-05T08:33:56.446 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:33:56.460 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-11-05T08:33:56.500 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-11-05T08:33:56.503 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:33:56.496+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:33:56.522 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1: {} 2023-11-05T08:33:56.522 INFO:teuthology.orchestra.run.smithi002.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-11-05T08:33:56.542 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2: {} 2023-11-05T08:33:56.542 INFO:teuthology.orchestra.run.smithi002.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-11-05T08:33:56.563 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3: {} 2023-11-05T08:33:56.563 INFO:teuthology.orchestra.run.smithi002.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-11-05T08:33:56.580 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4: {} 2023-11-05T08:33:56.580 INFO:teuthology.orchestra.run.smithi002.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-11-05T08:33:56.585 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:56.586 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:56.600 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2023-11-05T08:33:56.602 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5: {} 2023-11-05T08:33:56.603 INFO:teuthology.orchestra.run.smithi002.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-11-05T08:33:56.605 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:33:56.601+0000 7fdc5cd58640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:33:56.625 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6: {} 2023-11-05T08:33:56.625 INFO:teuthology.orchestra.run.smithi002.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-11-05T08:33:56.647 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7: {} 2023-11-05T08:33:56.647 INFO:teuthology.orchestra.run.smithi002.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-11-05T08:33:56.666 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8: {} 2023-11-05T08:33:56.667 INFO:teuthology.orchestra.run.smithi002.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-11-05T08:33:56.687 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9: {} 2023-11-05T08:33:56.687 INFO:teuthology.orchestra.run.smithi002.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-11-05T08:33:56.705 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:33:56.700+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:33:56.707 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10: {} 2023-11-05T08:33:56.707 INFO:teuthology.orchestra.run.smithi002.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-11-05T08:33:56.718 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:56.718 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:56.727 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11: {} 2023-11-05T08:33:56.728 INFO:teuthology.orchestra.run.smithi002.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-11-05T08:33:56.733 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2023-11-05T08:33:56.746 INFO:teuthology.orchestra.run.smithi002.stdout:osd.12: {} 2023-11-05T08:33:56.746 INFO:teuthology.orchestra.run.smithi002.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-11-05T08:33:56.747 INFO:teuthology.orchestra.run.smithi002.stderr:nodeep-scrub is set 2023-11-05T08:33:56.747 INFO:teuthology.orchestra.run.smithi002.stderr:reweighted osd.9 to 1 (10000) 2023-11-05T08:33:56.754 INFO:tasks.rados.rados.0.smithi090.stdout:332: done (6 left) 2023-11-05T08:33:56.755 INFO:tasks.rados.rados.0.smithi090.stdout:333: done (5 left) 2023-11-05T08:33:56.755 INFO:tasks.rados.rados.0.smithi090.stdout:334: done (4 left) 2023-11-05T08:33:56.755 INFO:tasks.rados.rados.0.smithi090.stdout:335: done (3 left) 2023-11-05T08:33:56.755 INFO:tasks.rados.rados.0.smithi090.stdout:336: done (2 left) 2023-11-05T08:33:56.755 INFO:tasks.rados.rados.0.smithi090.stdout:337: done (1 left) 2023-11-05T08:33:56.755 INFO:tasks.rados.rados.0.smithi090.stdout:338: done (0 left) 2023-11-05T08:33:56.756 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 3 current snap is 36 2023-11-05T08:33:56.756 INFO:tasks.rados.rados.0.smithi090.stdout:339: seq_num 127 ranges {540672=589824,1130496=753664,1884160=475136,2359296=327680} 2023-11-05T08:33:56.765 INFO:tasks.rados.rados.0.smithi090.stdout:339: writing smithi09039252-3 from 540672 to 1130496 tid 1 2023-11-05T08:33:56.769 INFO:teuthology.orchestra.run.smithi002.stdout:osd.13: {} 2023-11-05T08:33:56.769 INFO:teuthology.orchestra.run.smithi002.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-11-05T08:33:56.773 INFO:tasks.rados.rados.0.smithi090.stdout:339: writing smithi09039252-3 from 1130496 to 1884160 tid 2 2023-11-05T08:33:56.778 INFO:tasks.rados.rados.0.smithi090.stdout:339: writing smithi09039252-3 from 1884160 to 2359296 tid 3 2023-11-05T08:33:56.781 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2023-11-05T08:33:56.782 INFO:tasks.rados.rados.0.smithi090.stdout:339: writing smithi09039252-3 from 2359296 to 2686976 tid 4 2023-11-05T08:33:56.782 INFO:tasks.rados.rados.0.smithi090.stdout:340: delete oid 43 current snap is 36 2023-11-05T08:33:56.789 INFO:tasks.rados.rados.0.smithi090.stdout:339: finishing write tid 1 to smithi09039252-3 2023-11-05T08:33:56.790 INFO:tasks.rados.rados.0.smithi090.stdout:339: finishing write tid 2 to smithi09039252-3 2023-11-05T08:33:56.790 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-05T08:33:56.785+0000 7f49d0d84640 0 -- 172.21.15.2:0/1795343779 >> [v2:172.21.15.69:6808/567567051,v1:172.21.15.69:6809/567567051] conn(0x7f49cc219500 msgr2=0x7f49cc21b8f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-11-05T08:33:56.791 INFO:teuthology.orchestra.run.smithi002.stdout:osd.14: {} 2023-11-05T08:33:56.791 INFO:teuthology.orchestra.run.smithi002.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-11-05T08:33:56.795 INFO:tasks.rados.rados.0.smithi090.stdout:340: done (1 left) 2023-11-05T08:33:56.795 INFO:tasks.rados.rados.0.smithi090.stdout:341: rmattr oid 46 current snap is 36 2023-11-05T08:33:56.795 INFO:tasks.rados.rados.0.smithi090.stdout:341: done (1 left) 2023-11-05T08:33:56.796 INFO:tasks.rados.rados.0.smithi090.stdout:342: read oid 16 snap -1 2023-11-05T08:33:56.796 INFO:tasks.rados.rados.0.smithi090.stdout:342: expect (ObjNum 68 snap 6 seq_num 68) 2023-11-05T08:33:56.796 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 26 current snap is 36 2023-11-05T08:33:56.796 INFO:tasks.rados.rados.0.smithi090.stdout:343: seq_num 129 ranges {4014080=622592,4636672=655360,5292032=688128,5980160=212992} 2023-11-05T08:33:56.806 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:33:56.800+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:33:56.812 INFO:teuthology.orchestra.run.smithi002.stdout:osd.15: {} 2023-11-05T08:33:56.812 INFO:teuthology.orchestra.run.smithi002.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-11-05T08:33:56.817 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-05T08:33:56.817 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-05T08:33:56.825 INFO:tasks.rados.rados.0.smithi090.stdout:343: writing smithi09039252-26 from 4014080 to 4636672 tid 1 2023-11-05T08:33:56.829 INFO:tasks.rados.rados.0.smithi090.stdout:343: writing smithi09039252-26 from 4636672 to 5292032 tid 2 2023-11-05T08:33:56.833 INFO:tasks.rados.rados.0.smithi090.stdout:343: writing smithi09039252-26 from 5292032 to 5980160 tid 3 2023-11-05T08:33:56.835 INFO:tasks.rados.rados.0.smithi090.stdout:343: writing smithi09039252-26 from 5980160 to 6193152 tid 4 2023-11-05T08:33:56.835 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 24 current snap is 36 2023-11-05T08:33:56.835 INFO:tasks.rados.rados.0.smithi090.stdout:339: finishing write tid 3 to smithi09039252-3 2023-11-05T08:33:56.835 INFO:tasks.rados.rados.0.smithi090.stdout:344: seq_num 130 ranges {0=638976,638976=671744,1310720=802816,2113536=81920} 2023-11-05T08:33:56.838 INFO:tasks.rados.rados.0.smithi090.stdout:344: writing smithi09039252-24 from 0 to 638976 tid 1 2023-11-05T08:33:56.842 INFO:tasks.rados.rados.0.smithi090.stdout:344: writing smithi09039252-24 from 638976 to 1310720 tid 2 2023-11-05T08:33:56.847 INFO:tasks.rados.rados.0.smithi090.stdout:344: writing smithi09039252-24 from 1310720 to 2113536 tid 3 2023-11-05T08:33:56.848 INFO:tasks.rados.rados.0.smithi090.stdout:344: writing smithi09039252-24 from 2113536 to 2195456 tid 4 2023-11-05T08:33:56.848 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 48 current snap is 36 2023-11-05T08:33:56.848 INFO:tasks.rados.rados.0.smithi090.stdout:345: seq_num 131 ranges {802816=655360,1458176=425984,1884160=638976,2523136=491520} 2023-11-05T08:33:56.857 INFO:tasks.rados.rados.0.smithi090.stdout:345: writing smithi09039252-48 from 802816 to 1458176 tid 1 2023-11-05T08:33:56.859 INFO:tasks.rados.rados.0.smithi090.stdout:345: writing smithi09039252-48 from 1458176 to 1884160 tid 2 2023-11-05T08:33:56.863 INFO:tasks.rados.rados.0.smithi090.stdout:345: writing smithi09039252-48 from 1884160 to 2523136 tid 3 2023-11-05T08:33:56.866 INFO:tasks.rados.rados.0.smithi090.stdout:345: writing smithi09039252-48 from 2523136 to 3014656 tid 4 2023-11-05T08:33:56.866 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 17 current snap is 36 2023-11-05T08:33:56.866 INFO:tasks.rados.rados.0.smithi090.stdout:346: seq_num 132 ranges {0=671744,671744=573440,1245184=753664,1998848=229376} 2023-11-05T08:33:56.869 INFO:tasks.rados.rados.0.smithi090.stdout:346: writing smithi09039252-17 from 0 to 671744 tid 1 2023-11-05T08:33:56.873 INFO:tasks.rados.rados.0.smithi090.stdout:346: writing smithi09039252-17 from 671744 to 1245184 tid 2 2023-11-05T08:33:56.875 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:56.875 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:56.877 INFO:tasks.rados.rados.0.smithi090.stdout:346: writing smithi09039252-17 from 1245184 to 1998848 tid 3 2023-11-05T08:33:56.879 INFO:tasks.rados.rados.0.smithi090.stdout:346: writing smithi09039252-17 from 1998848 to 2228224 tid 4 2023-11-05T08:33:56.880 INFO:tasks.rados.rados.0.smithi090.stdout:347: rollback oid 44 current snap is 36 2023-11-05T08:33:56.880 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 44 to 33 2023-11-05T08:33:56.880 INFO:tasks.rados.rados.0.smithi090.stdout:339: finishing write tid 4 to smithi09039252-3 2023-11-05T08:33:56.880 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 28 current snap is 36 2023-11-05T08:33:56.880 INFO:tasks.rados.rados.0.smithi090.stdout:339: finishing write tid 5 to smithi09039252-3 2023-11-05T08:33:56.880 INFO:tasks.rados.rados.0.smithi090.stdout:348: seq_num 133 ranges {458752=688128,1146880=720896,1867776=475136,2342912=360448} 2023-11-05T08:33:56.886 INFO:tasks.rados.rados.0.smithi090.stdout:348: writing smithi09039252-28 from 458752 to 1146880 tid 1 2023-11-05T08:33:56.890 INFO:tasks.rados.rados.0.smithi090.stdout:348: writing smithi09039252-28 from 1146880 to 1867776 tid 2 2023-11-05T08:33:56.894 INFO:tasks.rados.rados.0.smithi090.stdout:348: writing smithi09039252-28 from 1867776 to 2342912 tid 3 2023-11-05T08:33:56.895 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2023-11-05T08:33:56.896 INFO:tasks.rados.rados.0.smithi090.stdout:348: writing smithi09039252-28 from 2342912 to 2703360 tid 4 2023-11-05T08:33:56.896 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 33 current snap is 36 2023-11-05T08:33:56.897 INFO:tasks.rados.rados.0.smithi090.stdout:342: expect (ObjNum 68 snap 6 seq_num 68) 2023-11-05T08:33:56.939 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:33:56.937+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:33:56.981 INFO:tasks.rados.rados.0.smithi090.stdout:339: finishing write tid 6 to smithi09039252-3 2023-11-05T08:33:56.982 INFO:tasks.rados.rados.0.smithi090.stdout:339: oid 3 updating version 0 to 63 2023-11-05T08:33:56.982 INFO:tasks.rados.rados.0.smithi090.stdout:339: oid 3 version 63 is already newer than 62 2023-11-05T08:33:56.982 INFO:tasks.rados.rados.0.smithi090.stdout:339: oid 3 version 63 is already newer than 60 2023-11-05T08:33:56.982 INFO:tasks.rados.rados.0.smithi090.stdout:339: oid 3 updating version 63 to 64 2023-11-05T08:33:56.983 INFO:tasks.rados.rados.0.smithi090.stdout:339: oid 3 version 64 is already newer than 61 2023-11-05T08:33:56.983 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 3 v 64 (ObjNum 127 snap 36 seq_num 127) dirty exists 2023-11-05T08:33:56.983 INFO:tasks.rados.rados.0.smithi090.stdout:339: left oid 3 (ObjNum 127 snap 36 seq_num 127) 2023-11-05T08:33:56.983 INFO:tasks.rados.rados.0.smithi090.stdout:343: finishing write tid 1 to smithi09039252-26 2023-11-05T08:33:56.983 INFO:tasks.rados.rados.0.smithi090.stdout:343: finishing write tid 2 to smithi09039252-26 2023-11-05T08:33:56.984 INFO:tasks.rados.rados.0.smithi090.stdout:343: finishing write tid 3 to smithi09039252-26 2023-11-05T08:33:56.984 INFO:tasks.rados.rados.0.smithi090.stdout:343: finishing write tid 4 to smithi09039252-26 2023-11-05T08:33:56.984 INFO:tasks.rados.rados.0.smithi090.stdout:343: finishing write tid 5 to smithi09039252-26 2023-11-05T08:33:56.984 INFO:tasks.rados.rados.0.smithi090.stdout:343: finishing write tid 6 to smithi09039252-26 2023-11-05T08:33:56.984 INFO:tasks.rados.rados.0.smithi090.stdout:343: oid 26 updating version 0 to 70 2023-11-05T08:33:56.984 INFO:tasks.rados.rados.0.smithi090.stdout:343: oid 26 version 70 is already newer than 69 2023-11-05T08:33:56.985 INFO:tasks.rados.rados.0.smithi090.stdout:343: oid 26 version 70 is already newer than 68 2023-11-05T08:33:56.985 INFO:tasks.rados.rados.0.smithi090.stdout:343: oid 26 version 70 is already newer than 67 2023-11-05T08:33:56.985 INFO:tasks.rados.rados.0.smithi090.stdout:343: oid 26 updating version 70 to 71 2023-11-05T08:33:56.985 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 26 v 71 (ObjNum 129 snap 36 seq_num 129) dirty exists 2023-11-05T08:33:56.985 INFO:tasks.rados.rados.0.smithi090.stdout:343: left oid 26 (ObjNum 129 snap 36 seq_num 129) 2023-11-05T08:33:56.985 INFO:tasks.rados.rados.0.smithi090.stdout:344: finishing write tid 1 to smithi09039252-24 2023-11-05T08:33:56.986 INFO:tasks.rados.rados.0.smithi090.stdout:344: finishing write tid 2 to smithi09039252-24 2023-11-05T08:33:56.986 INFO:tasks.rados.rados.0.smithi090.stdout:349: seq_num 134 ranges {540672=704512,1245184=475136,1720320=573440,2293760=507904} 2023-11-05T08:33:56.988 INFO:tasks.rados.rados.0.smithi090.stdout:349: writing smithi09039252-33 from 540672 to 1245184 tid 1 2023-11-05T08:33:56.991 INFO:tasks.rados.rados.0.smithi090.stdout:349: writing smithi09039252-33 from 1245184 to 1720320 tid 2 2023-11-05T08:33:56.995 INFO:tasks.rados.rados.0.smithi090.stdout:349: writing smithi09039252-33 from 1720320 to 2293760 tid 3 2023-11-05T08:33:56.998 INFO:tasks.rados.rados.0.smithi090.stdout:349: writing smithi09039252-33 from 2293760 to 2801664 tid 4 2023-11-05T08:33:56.998 INFO:tasks.rados.rados.0.smithi090.stdout:339: done (8 left) 2023-11-05T08:33:56.999 INFO:tasks.rados.rados.0.smithi090.stdout:342: done (7 left) 2023-11-05T08:33:56.999 INFO:tasks.rados.rados.0.smithi090.stdout:343: done (6 left) 2023-11-05T08:33:56.999 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 8 current snap is 36 2023-11-05T08:33:56.999 INFO:tasks.rados.rados.0.smithi090.stdout:350: seq_num 135 ranges {3293184=720896,4014080=622592,4636672=688128,5324800=245760} 2023-11-05T08:33:57.009 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:33:57.005+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:33:57.022 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:57.022 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:57.023 INFO:tasks.rados.rados.0.smithi090.stdout:350: writing smithi09039252-8 from 3293184 to 4014080 tid 1 2023-11-05T08:33:57.026 INFO:tasks.rados.rados.0.smithi090.stdout:350: writing smithi09039252-8 from 4014080 to 4636672 tid 2 2023-11-05T08:33:57.031 INFO:tasks.rados.rados.0.smithi090.stdout:350: writing smithi09039252-8 from 4636672 to 5324800 tid 3 2023-11-05T08:33:57.034 INFO:tasks.rados.rados.0.smithi090.stdout:350: writing smithi09039252-8 from 5324800 to 5570560 tid 4 2023-11-05T08:33:57.034 INFO:tasks.rados.rados.0.smithi090.stdout:351: read oid 12 snap -1 2023-11-05T08:33:57.034 INFO:tasks.rados.rados.0.smithi090.stdout:351: expect deleted 2023-11-05T08:33:57.034 INFO:tasks.rados.rados.0.smithi090.stdout:352: setattr oid 46 current snap is 36 2023-11-05T08:33:57.035 INFO:tasks.rados.rados.0.smithi090.stdout:344: finishing write tid 3 to smithi09039252-24 2023-11-05T08:33:57.035 INFO:tasks.rados.rados.0.smithi090.stdout:344: finishing write tid 4 to smithi09039252-24 2023-11-05T08:33:57.035 INFO:tasks.rados.rados.0.smithi090.stdout:344: finishing write tid 5 to smithi09039252-24 2023-11-05T08:33:57.035 INFO:tasks.rados.rados.0.smithi090.stdout:344: finishing write tid 6 to smithi09039252-24 2023-11-05T08:33:57.035 INFO:tasks.rados.rados.0.smithi090.stdout:344: oid 24 updating version 0 to 47 2023-11-05T08:33:57.036 INFO:tasks.rados.rados.0.smithi090.stdout:344: oid 24 updating version 47 to 48 2023-11-05T08:33:57.036 INFO:tasks.rados.rados.0.smithi090.stdout:344: oid 24 updating version 48 to 50 2023-11-05T08:33:57.036 INFO:tasks.rados.rados.0.smithi090.stdout:344: oid 24 updating version 50 to 51 2023-11-05T08:33:57.036 INFO:tasks.rados.rados.0.smithi090.stdout:344: oid 24 version 51 is already newer than 49 2023-11-05T08:33:57.036 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 24 v 51 (ObjNum 130 snap 36 seq_num 130) dirty exists 2023-11-05T08:33:57.037 INFO:tasks.rados.rados.0.smithi090.stdout:344: left oid 24 (ObjNum 130 snap 36 seq_num 130) 2023-11-05T08:33:57.037 INFO:tasks.rados.rados.0.smithi090.stdout:347: finishing rollback tid 0 to smithi09039252-44 2023-11-05T08:33:57.037 INFO:tasks.rados.rados.0.smithi090.stdout:346: finishing write tid 1 to smithi09039252-17 2023-11-05T08:33:57.037 INFO:tasks.rados.rados.0.smithi090.stdout:345: finishing write tid 1 to smithi09039252-48 2023-11-05T08:33:57.037 INFO:tasks.rados.rados.0.smithi090.stdout:346: finishing write tid 2 to smithi09039252-17 2023-11-05T08:33:57.038 INFO:tasks.rados.rados.0.smithi090.stdout:345: finishing write tid 2 to smithi09039252-48 2023-11-05T08:33:57.038 INFO:tasks.rados.rados.0.smithi090.stdout:347: finishing rollback tid 1 to smithi09039252-44 2023-11-05T08:33:57.038 INFO:tasks.rados.rados.0.smithi090.stdout:347: finishing rollback tid 2 to smithi09039252-44 2023-11-05T08:33:57.038 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 44 v 47 (ObjNum 43 snap 0 seq_num 43) dirty exists 2023-11-05T08:33:57.038 INFO:tasks.rados.rados.0.smithi090.stdout:345: finishing write tid 3 to smithi09039252-48 2023-11-05T08:33:57.038 INFO:tasks.rados.rados.0.smithi090.stdout:345: finishing write tid 4 to smithi09039252-48 2023-11-05T08:33:57.039 INFO:tasks.rados.rados.0.smithi090.stdout:346: finishing write tid 3 to smithi09039252-17 2023-11-05T08:33:57.039 INFO:tasks.rados.rados.0.smithi090.stdout:345: finishing write tid 5 to smithi09039252-48 2023-11-05T08:33:57.039 INFO:tasks.rados.rados.0.smithi090.stdout:346: finishing write tid 4 to smithi09039252-17 2023-11-05T08:33:57.039 INFO:tasks.rados.rados.0.smithi090.stdout:348: finishing write tid 1 to smithi09039252-28 2023-11-05T08:33:57.039 INFO:tasks.rados.rados.0.smithi090.stdout:346: finishing write tid 5 to smithi09039252-17 2023-11-05T08:33:57.040 INFO:tasks.rados.rados.0.smithi090.stdout:348: finishing write tid 2 to smithi09039252-28 2023-11-05T08:33:57.040 INFO:tasks.rados.rados.0.smithi090.stdout:345: finishing write tid 6 to smithi09039252-48 2023-11-05T08:33:57.040 INFO:tasks.rados.rados.0.smithi090.stdout:345: oid 48 updating version 0 to 65 2023-11-05T08:33:57.040 INFO:tasks.rados.rados.0.smithi090.stdout:345: oid 48 updating version 65 to 66 2023-11-05T08:33:57.040 INFO:tasks.rados.rados.0.smithi090.stdout:345: oid 48 version 66 is already newer than 64 2023-11-05T08:33:57.040 INFO:tasks.rados.rados.0.smithi090.stdout:345: oid 48 updating version 66 to 67 2023-11-05T08:33:57.041 INFO:tasks.rados.rados.0.smithi090.stdout:345: oid 48 updating version 67 to 68 2023-11-05T08:33:57.041 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 48 v 68 (ObjNum 131 snap 36 seq_num 131) dirty exists 2023-11-05T08:33:57.041 INFO:tasks.rados.rados.0.smithi090.stdout:345: left oid 48 (ObjNum 131 snap 36 seq_num 131) 2023-11-05T08:33:57.041 INFO:tasks.rados.rados.0.smithi090.stdout:348: finishing write tid 3 to smithi09039252-28 2023-11-05T08:33:57.041 INFO:tasks.rados.rados.0.smithi090.stdout:346: finishing write tid 6 to smithi09039252-17 2023-11-05T08:33:57.042 INFO:tasks.rados.rados.0.smithi090.stdout:346: oid 17 updating version 0 to 32 2023-11-05T08:33:57.042 INFO:tasks.rados.rados.0.smithi090.stdout:346: oid 17 version 32 is already newer than 31 2023-11-05T08:33:57.042 INFO:tasks.rados.rados.0.smithi090.stdout:346: oid 17 version 32 is already newer than 30 2023-11-05T08:33:57.042 INFO:tasks.rados.rados.0.smithi090.stdout:346: oid 17 updating version 32 to 34 2023-11-05T08:33:57.042 INFO:tasks.rados.rados.0.smithi090.stdout:346: oid 17 version 34 is already newer than 33 2023-11-05T08:33:57.043 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 17 v 34 (ObjNum 132 snap 36 seq_num 132) dirty exists 2023-11-05T08:33:57.043 INFO:tasks.rados.rados.0.smithi090.stdout:346: left oid 17 (ObjNum 132 snap 36 seq_num 132) 2023-11-05T08:33:57.043 INFO:tasks.rados.rados.0.smithi090.stdout:348: finishing write tid 4 to smithi09039252-28 2023-11-05T08:33:57.043 INFO:tasks.rados.rados.0.smithi090.stdout:348: finishing write tid 5 to smithi09039252-28 2023-11-05T08:33:57.043 INFO:tasks.rados.rados.0.smithi090.stdout:348: finishing write tid 6 to smithi09039252-28 2023-11-05T08:33:57.044 INFO:tasks.rados.rados.0.smithi090.stdout:348: oid 28 updating version 0 to 56 2023-11-05T08:33:57.044 INFO:tasks.rados.rados.0.smithi090.stdout:348: oid 28 version 56 is already newer than 55 2023-11-05T08:33:57.044 INFO:tasks.rados.rados.0.smithi090.stdout:348: oid 28 updating version 56 to 57 2023-11-05T08:33:57.044 INFO:tasks.rados.rados.0.smithi090.stdout:348: oid 28 version 57 is already newer than 54 2023-11-05T08:33:57.044 INFO:tasks.rados.rados.0.smithi090.stdout:348: oid 28 version 57 is already newer than 53 2023-11-05T08:33:57.044 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 28 v 57 (ObjNum 133 snap 36 seq_num 133) dirty exists 2023-11-05T08:33:57.045 INFO:tasks.rados.rados.0.smithi090.stdout:348: left oid 28 (ObjNum 133 snap 36 seq_num 133) 2023-11-05T08:33:57.045 INFO:tasks.rados.rados.0.smithi090.stdout:349: finishing write tid 1 to smithi09039252-33 2023-11-05T08:33:57.045 INFO:tasks.rados.rados.0.smithi090.stdout:349: finishing write tid 2 to smithi09039252-33 2023-11-05T08:33:57.045 INFO:tasks.rados.rados.0.smithi090.stdout:349: finishing write tid 3 to smithi09039252-33 2023-11-05T08:33:57.045 INFO:tasks.rados.rados.0.smithi090.stdout:349: finishing write tid 4 to smithi09039252-33 2023-11-05T08:33:57.046 INFO:tasks.rados.rados.0.smithi090.stdout:349: finishing write tid 5 to smithi09039252-33 2023-11-05T08:33:57.046 INFO:tasks.rados.rados.0.smithi090.stdout:349: finishing write tid 6 to smithi09039252-33 2023-11-05T08:33:57.046 INFO:tasks.rados.rados.0.smithi090.stdout:349: oid 33 updating version 0 to 27 2023-11-05T08:33:57.046 INFO:tasks.rados.rados.0.smithi090.stdout:349: oid 33 updating version 27 to 31 2023-11-05T08:33:57.046 INFO:tasks.rados.rados.0.smithi090.stdout:349: oid 33 version 31 is already newer than 28 2023-11-05T08:33:57.046 INFO:tasks.rados.rados.0.smithi090.stdout:349: oid 33 version 31 is already newer than 30 2023-11-05T08:33:57.047 INFO:tasks.rados.rados.0.smithi090.stdout:349: oid 33 version 31 is already newer than 29 2023-11-05T08:33:57.047 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 33 v 31 (ObjNum 134 snap 36 seq_num 134) dirty exists 2023-11-05T08:33:57.047 INFO:tasks.rados.rados.0.smithi090.stdout:349: left oid 33 (ObjNum 134 snap 36 seq_num 134) 2023-11-05T08:33:57.049 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2023-11-05T08:33:57.049 INFO:tasks.rados.rados.0.smithi090.stdout:344: done (8 left) 2023-11-05T08:33:57.049 INFO:tasks.rados.rados.0.smithi090.stdout:345: done (7 left) 2023-11-05T08:33:57.050 INFO:tasks.rados.rados.0.smithi090.stdout:346: done (6 left) 2023-11-05T08:33:57.050 INFO:tasks.rados.rados.0.smithi090.stdout:347: done (5 left) 2023-11-05T08:33:57.050 INFO:tasks.rados.rados.0.smithi090.stdout:348: done (4 left) 2023-11-05T08:33:57.050 INFO:tasks.rados.rados.0.smithi090.stdout:349: done (3 left) 2023-11-05T08:33:57.051 INFO:tasks.rados.rados.0.smithi090.stdout:353: read oid 32 snap -1 2023-11-05T08:33:57.051 INFO:tasks.rados.rados.0.smithi090.stdout:353: expect (ObjNum 31 snap 0 seq_num 31) 2023-11-05T08:33:57.051 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 35 current snap is 36 2023-11-05T08:33:57.051 INFO:tasks.rados.rados.0.smithi090.stdout:354: seq_num 136 ranges {573440=737280,1310720=770048,2080768=540672,2621440=245760} 2023-11-05T08:33:57.051 INFO:tasks.rados.rados.0.smithi090.stdout:354: writing smithi09039252-35 from 573440 to 1310720 tid 1 2023-11-05T08:33:57.051 INFO:tasks.rados.rados.0.smithi090.stdout:354: writing smithi09039252-35 from 1310720 to 2080768 tid 2 2023-11-05T08:33:57.052 INFO:tasks.rados.rados.0.smithi090.stdout:354: writing smithi09039252-35 from 2080768 to 2621440 tid 3 2023-11-05T08:33:57.053 INFO:tasks.rados.rados.0.smithi090.stdout:354: writing smithi09039252-35 from 2621440 to 2867200 tid 4 2023-11-05T08:33:57.053 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 20 current snap is 36 2023-11-05T08:33:57.053 INFO:tasks.rados.rados.0.smithi090.stdout:350: finishing write tid 1 to smithi09039252-8 2023-11-05T08:33:57.053 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 46 v 73 (ObjNum 17 snap 0 seq_num 17) dirty exists 2023-11-05T08:33:57.053 INFO:tasks.rados.rados.0.smithi090.stdout:353: expect (ObjNum 31 snap 0 seq_num 31) 2023-11-05T08:33:57.092 INFO:tasks.rados.rados.0.smithi090.stdout:350: finishing write tid 2 to smithi09039252-8 2023-11-05T08:33:57.092 INFO:tasks.rados.rados.0.smithi090.stdout:355: seq_num 137 ranges {0=753664,753664=786432,1540096=655360,2195456=114688} 2023-11-05T08:33:57.096 INFO:tasks.rados.rados.0.smithi090.stdout:355: writing smithi09039252-20 from 0 to 753664 tid 1 2023-11-05T08:33:57.100 INFO:tasks.rados.rados.0.smithi090.stdout:355: writing smithi09039252-20 from 753664 to 1540096 tid 2 2023-11-05T08:33:57.104 INFO:tasks.rados.rados.0.smithi090.stdout:355: writing smithi09039252-20 from 1540096 to 2195456 tid 3 2023-11-05T08:33:57.106 INFO:tasks.rados.rados.0.smithi090.stdout:355: writing smithi09039252-20 from 2195456 to 2310144 tid 4 2023-11-05T08:33:57.106 INFO:tasks.rados.rados.0.smithi090.stdout:352: done (5 left) 2023-11-05T08:33:57.107 INFO:tasks.rados.rados.0.smithi090.stdout:353: done (4 left) 2023-11-05T08:33:57.107 INFO:tasks.rados.rados.0.smithi090.stdout:356: copy_from oid 44 from oid 17 current snap is 36 2023-11-05T08:33:57.108 INFO:tasks.rados.rados.0.smithi090.stdout:357: copy_from oid 41 from oid 15 current snap is 36 2023-11-05T08:33:57.108 INFO:tasks.rados.rados.0.smithi090.stdout:350: finishing write tid 3 to smithi09039252-8 2023-11-05T08:33:57.108 INFO:tasks.rados.rados.0.smithi090.stdout:350: finishing write tid 4 to smithi09039252-8 2023-11-05T08:33:57.108 INFO:tasks.rados.rados.0.smithi090.stdout:350: finishing write tid 5 to smithi09039252-8 2023-11-05T08:33:57.108 INFO:tasks.rados.rados.0.smithi090.stdout:350: finishing write tid 6 to smithi09039252-8 2023-11-05T08:33:57.109 INFO:tasks.rados.rados.0.smithi090.stdout:350: oid 8 updating version 0 to 120 2023-11-05T08:33:57.109 INFO:tasks.rados.rados.0.smithi090.stdout:350: oid 8 version 120 is already newer than 118 2023-11-05T08:33:57.109 INFO:tasks.rados.rados.0.smithi090.stdout:350: oid 8 updating version 120 to 121 2023-11-05T08:33:57.109 INFO:tasks.rados.rados.0.smithi090.stdout:350: oid 8 updating version 121 to 122 2023-11-05T08:33:57.110 INFO:tasks.rados.rados.0.smithi090.stdout:350: oid 8 version 122 is already newer than 119 2023-11-05T08:33:57.110 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 8 v 122 (ObjNum 135 snap 36 seq_num 135) dirty exists 2023-11-05T08:33:57.110 INFO:tasks.rados.rados.0.smithi090.stdout:350: left oid 8 (ObjNum 135 snap 36 seq_num 135) 2023-11-05T08:33:57.110 INFO:tasks.rados.rados.0.smithi090.stdout:354: finishing write tid 1 to smithi09039252-35 2023-11-05T08:33:57.110 INFO:tasks.rados.rados.0.smithi090.stdout:354: finishing write tid 2 to smithi09039252-35 2023-11-05T08:33:57.111 INFO:tasks.rados.rados.0.smithi090.stdout:350: done (5 left) 2023-11-05T08:33:57.111 INFO:tasks.rados.rados.0.smithi090.stdout:351: done (4 left) 2023-11-05T08:33:57.111 INFO:tasks.rados.rados.0.smithi090.stdout:358: rollback oid 23 current snap is 36 2023-11-05T08:33:57.111 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 23 to 33 2023-11-05T08:33:57.111 INFO:tasks.rados.rados.0.smithi090.stdout:354: finishing write tid 3 to smithi09039252-35 2023-11-05T08:33:57.112 INFO:tasks.rados.rados.0.smithi090.stdout:354: finishing write tid 4 to smithi09039252-35 2023-11-05T08:33:57.112 INFO:tasks.rados.rados.0.smithi090.stdout:354: finishing write tid 5 to smithi09039252-35 2023-11-05T08:33:57.112 INFO:tasks.rados.rados.0.smithi090.stdout:354: finishing write tid 6 to smithi09039252-35 2023-11-05T08:33:57.112 INFO:tasks.rados.rados.0.smithi090.stdout:354: oid 35 updating version 0 to 24 2023-11-05T08:33:57.112 INFO:tasks.rados.rados.0.smithi090.stdout:354: oid 35 updating version 24 to 25 2023-11-05T08:33:57.113 INFO:tasks.rados.rados.0.smithi090.stdout:354: oid 35 updating version 25 to 26 2023-11-05T08:33:57.113 INFO:tasks.rados.rados.0.smithi090.stdout:354: oid 35 updating version 26 to 27 2023-11-05T08:33:57.113 INFO:tasks.rados.rados.0.smithi090.stdout:354: oid 35 version 27 is already newer than 23 2023-11-05T08:33:57.113 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 35 v 27 (ObjNum 136 snap 36 seq_num 136) dirty exists 2023-11-05T08:33:57.113 INFO:tasks.rados.rados.0.smithi090.stdout:354: left oid 35 (ObjNum 136 snap 36 seq_num 136) 2023-11-05T08:33:57.113 INFO:tasks.rados.rados.0.smithi090.stdout:355: finishing write tid 1 to smithi09039252-20 2023-11-05T08:33:57.114 INFO:tasks.rados.rados.0.smithi090.stdout:354: done (4 left) 2023-11-05T08:33:57.114 INFO:tasks.rados.rados.0.smithi090.stdout:359: read oid 3 snap 33 2023-11-05T08:33:57.114 INFO:tasks.rados.rados.0.smithi090.stdout:359: expect (ObjNum 32 snap 0 seq_num 32) 2023-11-05T08:33:57.114 INFO:tasks.rados.rados.0.smithi090.stdout:360: rollback oid 18 current snap is 36 2023-11-05T08:33:57.114 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 18 to 32 2023-11-05T08:33:57.115 INFO:tasks.rados.rados.0.smithi090.stdout:361: snap_create 2023-11-05T08:33:57.116 INFO:tasks.rados.rados.0.smithi090.stdout:355: finishing write tid 2 to smithi09039252-20 2023-11-05T08:33:57.116 INFO:tasks.rados.rados.0.smithi090.stdout:355: finishing write tid 3 to smithi09039252-20 2023-11-05T08:33:57.116 INFO:tasks.rados.rados.0.smithi090.stdout:358: finishing rollback tid 0 to smithi09039252-23 2023-11-05T08:33:57.116 INFO:tasks.rados.rados.0.smithi090.stdout:358: finishing rollback tid 1 to smithi09039252-23 2023-11-05T08:33:57.117 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:33:57.112+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:33:57.117 INFO:tasks.rados.rados.0.smithi090.stdout:355: finishing write tid 4 to smithi09039252-20 2023-11-05T08:33:57.118 INFO:tasks.rados.rados.0.smithi090.stdout:357: finishing copy_from racing read to smithi09039252-41 2023-11-05T08:33:57.119 INFO:tasks.rados.rados.0.smithi090.stdout:359: expect (ObjNum 32 snap 0 seq_num 32) 2023-11-05T08:33:57.174 INFO:tasks.rados.rados.0.smithi090.stdout:358: finishing rollback tid 2 to smithi09039252-23 2023-11-05T08:33:57.175 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 23 v 36 (ObjNum 60 snap 1 seq_num 60) dirty exists 2023-11-05T08:33:57.175 INFO:tasks.rados.rados.0.smithi090.stdout:355: finishing write tid 5 to smithi09039252-20 2023-11-05T08:33:57.175 INFO:tasks.rados.rados.0.smithi090.stdout:360: finishing rollback tid 0 to smithi09039252-18 2023-11-05T08:33:57.175 INFO:tasks.rados.rados.0.smithi090.stdout:360: finishing rollback tid 1 to smithi09039252-18 2023-11-05T08:33:57.176 INFO:tasks.rados.rados.0.smithi090.stdout:360: finishing rollback tid 2 to smithi09039252-18 2023-11-05T08:33:57.176 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 18 v 51 (ObjNum 115 snap 30 seq_num 115) dirty exists 2023-11-05T08:33:57.176 INFO:tasks.rados.rados.0.smithi090.stdout:357: finishing copy_from to smithi09039252-41 2023-11-05T08:33:57.176 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 41 v 30 (ObjNum 17 snap 0 seq_num 17) dirty exists 2023-11-05T08:33:57.176 INFO:tasks.rados.rados.0.smithi090.stdout:356: finishing copy_from racing read to smithi09039252-44 2023-11-05T08:33:57.177 INFO:tasks.rados.rados.0.smithi090.stdout:355: finishing write tid 6 to smithi09039252-20 2023-11-05T08:33:57.177 INFO:tasks.rados.rados.0.smithi090.stdout:355: oid 20 updating version 0 to 47 2023-11-05T08:33:57.177 INFO:tasks.rados.rados.0.smithi090.stdout:355: oid 20 version 47 is already newer than 44 2023-11-05T08:33:57.177 INFO:tasks.rados.rados.0.smithi090.stdout:355: oid 20 version 47 is already newer than 46 2023-11-05T08:33:57.177 INFO:tasks.rados.rados.0.smithi090.stdout:355: oid 20 version 47 is already newer than 45 2023-11-05T08:33:57.178 INFO:tasks.rados.rados.0.smithi090.stdout:355: oid 20 version 47 is already newer than 43 2023-11-05T08:33:57.178 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 20 v 47 (ObjNum 137 snap 36 seq_num 137) dirty exists 2023-11-05T08:33:57.178 INFO:tasks.rados.rados.0.smithi090.stdout:355: left oid 20 (ObjNum 137 snap 36 seq_num 137) 2023-11-05T08:33:57.178 INFO:tasks.rados.rados.0.smithi090.stdout:356: finishing copy_from to smithi09039252-44 2023-11-05T08:33:57.178 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 44 v 34 (ObjNum 132 snap 36 seq_num 132) dirty exists 2023-11-05T08:33:57.217 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:33:57.213+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:33:57.219 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:57.219 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:57.234 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2023-11-05T08:33:57.318 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:33:57.316+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:33:57.400 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:57.481 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:57.483 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2023-11-05T08:33:57.484 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-11-05T08:33:57.478+0000 7f72b2c5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 42525) UID: 0 2023-11-05T08:33:57.585 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:57.582+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:57.615 INFO:teuthology.orchestra.run.smithi002.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:57.615 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:57.631 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-11-05T08:33:57.685 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:33:57.681+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:33:57.758 INFO:teuthology.orchestra.run.smithi002.stderr:reweighted osd.10 to 1 (10000) 2023-11-05T08:33:57.766 INFO:tasks.rados.rados.0.smithi090.stdout:355: done (6 left) 2023-11-05T08:33:57.766 INFO:tasks.rados.rados.0.smithi090.stdout:356: done (5 left) 2023-11-05T08:33:57.767 INFO:tasks.rados.rados.0.smithi090.stdout:357: done (4 left) 2023-11-05T08:33:57.767 INFO:tasks.rados.rados.0.smithi090.stdout:358: done (3 left) 2023-11-05T08:33:57.767 INFO:tasks.rados.rados.0.smithi090.stdout:359: done (2 left) 2023-11-05T08:33:57.767 INFO:tasks.rados.rados.0.smithi090.stdout:360: done (1 left) 2023-11-05T08:33:57.767 INFO:tasks.rados.rados.0.smithi090.stdout:361: done (0 left) 2023-11-05T08:33:57.768 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 39 current snap is 37 2023-11-05T08:33:57.768 INFO:tasks.rados.rados.0.smithi090.stdout:362: seq_num 138 ranges {1933312=770048,2703360=540672,3244032=753664,3997696=262144} 2023-11-05T08:33:57.769 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:57.769 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:57.782 INFO:tasks.rados.rados.0.smithi090.stdout:362: writing smithi09039252-39 from 1933312 to 2703360 tid 1 2023-11-05T08:33:57.783 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-11-05T08:33:57.785 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2023-11-05T08:33:57.786 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:33:57.782+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:33:57.786 INFO:tasks.rados.rados.0.smithi090.stdout:362: writing smithi09039252-39 from 2703360 to 3244032 tid 2 2023-11-05T08:33:57.790 INFO:tasks.rados.rados.0.smithi090.stdout:362: writing smithi09039252-39 from 3244032 to 3997696 tid 3 2023-11-05T08:33:57.792 INFO:tasks.rados.rados.0.smithi090.stdout:362: writing smithi09039252-39 from 3997696 to 4259840 tid 4 2023-11-05T08:33:57.793 INFO:tasks.rados.rados.0.smithi090.stdout:363: rollback oid 35 current snap is 37 2023-11-05T08:33:57.793 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 35 to 36 2023-11-05T08:33:57.793 INFO:tasks.rados.rados.0.smithi090.stdout:364: snap_remove snap 32 2023-11-05T08:33:57.885 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:33:57.881+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:33:57.906 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:57.907 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:57.920 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-11-05T08:33:57.987 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:33:57.980+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:33:58.044 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:58.044 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:58.058 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-11-05T08:33:58.088 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:33:58.085+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:33:58.179 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:58.180 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:58.188 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:33:58.184+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:33:58.194 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-11-05T08:33:58.290 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:58.286+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:58.321 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:58.322 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:58.336 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-11-05T08:33:58.391 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:33:58.388+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:33:58.462 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:58.463 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:58.477 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2023-11-05T08:33:58.492 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:33:58.488+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:33:58.592 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:33:58.588+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:33:58.646 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:58.646 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:58.660 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2023-11-05T08:33:58.693 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:58.690+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:58.770 INFO:teuthology.orchestra.run.smithi002.stderr:reweighted osd.11 to 1 (10000) 2023-11-05T08:33:58.778 INFO:tasks.rados.rados.0.smithi090.stdout:364: done (2 left) 2023-11-05T08:33:58.778 INFO:tasks.rados.rados.0.smithi090.stdout:365: snap_remove snap 35 2023-11-05T08:33:58.781 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-11-05T08:33:58.802 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 12 1 2023-11-05T08:33:58.803 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:33:58.797+0000 7fdc5cd58640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:33:58.818 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-05T08:33:58.850 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:58.851 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:58.865 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2023-11-05T08:33:58.911 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:58.908+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:58.977 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-11-05T08:33:58.989 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:58.989 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:58.995 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:33:58.989+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:33:58.999 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1: {} 2023-11-05T08:33:58.999 INFO:teuthology.orchestra.run.smithi002.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-11-05T08:33:59.004 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2023-11-05T08:33:59.020 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2: {} 2023-11-05T08:33:59.021 INFO:teuthology.orchestra.run.smithi002.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-11-05T08:33:59.042 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3: {} 2023-11-05T08:33:59.042 INFO:teuthology.orchestra.run.smithi002.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-11-05T08:33:59.060 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4: {} 2023-11-05T08:33:59.061 INFO:teuthology.orchestra.run.smithi002.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-11-05T08:33:59.079 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5: {} 2023-11-05T08:33:59.080 INFO:teuthology.orchestra.run.smithi002.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-11-05T08:33:59.095 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:33:59.092+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:33:59.100 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6: {} 2023-11-05T08:33:59.100 INFO:teuthology.orchestra.run.smithi002.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-11-05T08:33:59.122 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7: {} 2023-11-05T08:33:59.123 INFO:teuthology.orchestra.run.smithi002.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-11-05T08:33:59.140 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8: {} 2023-11-05T08:33:59.141 INFO:teuthology.orchestra.run.smithi002.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-11-05T08:33:59.160 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9: {} 2023-11-05T08:33:59.160 INFO:teuthology.orchestra.run.smithi002.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-11-05T08:33:59.170 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:33:59.171 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:33:59.182 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10: {} 2023-11-05T08:33:59.183 INFO:teuthology.orchestra.run.smithi002.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-11-05T08:33:59.185 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2023-11-05T08:33:59.197 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:33:59.192+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:33:59.205 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11: {} 2023-11-05T08:33:59.205 INFO:teuthology.orchestra.run.smithi002.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-11-05T08:33:59.224 INFO:teuthology.orchestra.run.smithi002.stdout:osd.12: {} 2023-11-05T08:33:59.224 INFO:teuthology.orchestra.run.smithi002.stderr:osd.12: osd_enable_op_tracker = 'true' 2023-11-05T08:33:59.245 INFO:teuthology.orchestra.run.smithi002.stdout:osd.13: {} 2023-11-05T08:33:59.245 INFO:teuthology.orchestra.run.smithi002.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-11-05T08:33:59.267 INFO:teuthology.orchestra.run.smithi002.stdout:osd.14: {} 2023-11-05T08:33:59.268 INFO:teuthology.orchestra.run.smithi002.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-11-05T08:33:59.289 INFO:teuthology.orchestra.run.smithi002.stdout:osd.15: {} 2023-11-05T08:33:59.290 INFO:teuthology.orchestra.run.smithi002.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-11-05T08:33:59.295 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-05T08:33:59.296 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-05T08:33:59.297 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:33:59.292+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:33:59.349 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:33:59.350 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2023-11-05T08:33:59.350 INFO:teuthology.orchestra.run.smithi053.stdout: "complaint_time": 30, 2023-11-05T08:33:59.350 INFO:teuthology.orchestra.run.smithi053.stdout: "num_blocked_ops": 0 2023-11-05T08:33:59.350 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:33:59.365 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2023-11-05T08:33:59.398 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:33:59.392+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:33:59.499 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:33:59.496+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:33:59.538 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:33:59.538 INFO:teuthology.orchestra.run.smithi053.stdout: "size": 20, 2023-11-05T08:33:59.539 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 600, 2023-11-05T08:33:59.539 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2023-11-05T08:33:59.539 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.539 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:59.539 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:28.710073+0000", 2023-11-05T08:33:59.540 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 90.824215613000007, 2023-11-05T08:33:59.540 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.052834791999999998, 2023-11-05T08:33:59.540 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:59.540 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:59.540 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:59.541 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:59.541 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:59.541 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 1 2023-11-05T08:33:59.541 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.541 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:59.542 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.542 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:59.542 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710073+0000", 2023-11-05T08:33:59.542 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.542 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.542 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.543 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:59.543 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710073+0000", 2023-11-05T08:33:59.543 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.543 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.543 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.544 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:59.544 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710076+0000", 2023-11-05T08:33:59.544 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1729999999999999e-06 2023-11-05T08:33:59.544 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.544 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.544 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:59.545 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710108+0000", 2023-11-05T08:33:59.545 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1458000000000003e-05 2023-11-05T08:33:59.545 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.545 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.545 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:59.546 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710140+0000", 2023-11-05T08:33:59.546 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2532e-05 2023-11-05T08:33:59.546 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.546 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.546 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:59.546 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710165+0000", 2023-11-05T08:33:59.547 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5205e-05 2023-11-05T08:33:59.547 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.547 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.547 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:59.547 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710311+0000", 2023-11-05T08:33:59.548 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014532799999999999 2023-11-05T08:33:59.548 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.548 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.548 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2023-11-05T08:33:59.548 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.710327+0000", 2023-11-05T08:33:59.549 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6059e-05 2023-11-05T08:33:59.549 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.549 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.549 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:59.549 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.759348+0000", 2023-11-05T08:33:59.550 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.049021481999999998 2023-11-05T08:33:59.550 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.550 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.550 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:59.550 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.759880+0000", 2023-11-05T08:33:59.551 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00053197400000000005 2023-11-05T08:33:59.551 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.551 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.551 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:59.551 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.760478+0000", 2023-11-05T08:33:59.552 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00059839300000000002 2023-11-05T08:33:59.552 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.552 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.552 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:59.552 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.761955+0000", 2023-11-05T08:33:59.553 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001476991 2023-11-05T08:33:59.553 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.553 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.553 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:59.553 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.762611+0000", 2023-11-05T08:33:59.554 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00065598700000000002 2023-11-05T08:33:59.554 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.554 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.554 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:59.554 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.762782+0000", 2023-11-05T08:33:59.554 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00017064000000000001 2023-11-05T08:33:59.555 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.555 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.555 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:59.555 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:28.762908+0000", 2023-11-05T08:33:59.555 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012557000000000001 2023-11-05T08:33:59.556 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.556 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:59.556 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.556 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.556 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.557 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:59.557 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.151583+0000", 2023-11-05T08:33:59.557 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 90.382705819999998, 2023-11-05T08:33:59.557 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.020286660000000001, 2023-11-05T08:33:59.557 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:59.558 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:59.558 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:59.558 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:59.558 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:59.558 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 114 2023-11-05T08:33:59.558 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.559 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:59.559 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.559 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:59.559 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151583+0000", 2023-11-05T08:33:59.559 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.560 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.560 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.560 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:59.560 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151583+0000", 2023-11-05T08:33:59.560 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.561 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.561 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.561 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:59.561 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151583+0000", 2023-11-05T08:33:59.561 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.2600000000000002e-07 2023-11-05T08:33:59.561 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.562 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.562 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:59.562 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151589+0000", 2023-11-05T08:33:59.562 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5099999999999998e-06 2023-11-05T08:33:59.562 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.563 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.563 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:59.563 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151598+0000", 2023-11-05T08:33:59.563 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1730000000000001e-06 2023-11-05T08:33:59.563 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.564 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.564 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:59.564 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151608+0000", 2023-11-05T08:33:59.564 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0059e-05 2023-11-05T08:33:59.564 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.565 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.565 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:59.565 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.165015+0000", 2023-11-05T08:33:59.565 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.013407207000000001 2023-11-05T08:33:59.565 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.566 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.566 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:59.566 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.165268+0000", 2023-11-05T08:33:59.566 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00025284999999999999 2023-11-05T08:33:59.566 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.566 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.567 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:59.567 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.165635+0000", 2023-11-05T08:33:59.567 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00036695899999999998 2023-11-05T08:33:59.567 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.567 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.568 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:59.568 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.170105+0000", 2023-11-05T08:33:59.568 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0044696960000000004 2023-11-05T08:33:59.568 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.568 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.569 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:59.569 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.171719+0000", 2023-11-05T08:33:59.569 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001613855 2023-11-05T08:33:59.569 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.569 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.570 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:59.570 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.171810+0000", 2023-11-05T08:33:59.570 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1625000000000005e-05 2023-11-05T08:33:59.570 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.570 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.571 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:59.571 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.171869+0000", 2023-11-05T08:33:59.571 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.91e-05 2023-11-05T08:33:59.571 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.571 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:59.572 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.572 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.572 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.572 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:59.572 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.151660+0000", 2023-11-05T08:33:59.572 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 90.382628021000002, 2023-11-05T08:33:59.573 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022553719, 2023-11-05T08:33:59.573 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:59.573 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:59.573 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:59.573 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:59.574 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:59.574 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 115 2023-11-05T08:33:59.574 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.574 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:59.574 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.575 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:59.575 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151660+0000", 2023-11-05T08:33:59.575 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.575 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.575 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.575 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:59.576 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151660+0000", 2023-11-05T08:33:59.576 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.576 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.576 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.576 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:59.576 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151661+0000", 2023-11-05T08:33:59.577 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.5600000000000004e-07 2023-11-05T08:33:59.577 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.577 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.577 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:59.577 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151799+0000", 2023-11-05T08:33:59.578 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000137207 2023-11-05T08:33:59.578 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.578 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.578 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:59.578 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151814+0000", 2023-11-05T08:33:59.579 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4885000000000001e-05 2023-11-05T08:33:59.579 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.579 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.579 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:59.579 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151827+0000", 2023-11-05T08:33:59.579 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3413e-05 2023-11-05T08:33:59.580 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.580 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.580 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:59.580 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.167513+0000", 2023-11-05T08:33:59.580 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.015686091999999999 2023-11-05T08:33:59.580 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.581 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.581 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:59.581 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.167652+0000", 2023-11-05T08:33:59.581 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013890500000000001 2023-11-05T08:33:59.581 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.582 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.582 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:59.582 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.167860+0000", 2023-11-05T08:33:59.582 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00020838599999999999 2023-11-05T08:33:59.582 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.583 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.583 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:59.583 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.171602+0000", 2023-11-05T08:33:59.583 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0037417449999999999 2023-11-05T08:33:59.583 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.583 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.584 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:59.584 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174068+0000", 2023-11-05T08:33:59.584 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0024658010000000001 2023-11-05T08:33:59.584 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.584 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.585 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:59.585 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174152+0000", 2023-11-05T08:33:59.585 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.3752999999999996e-05 2023-11-05T08:33:59.585 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.585 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.586 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:59.586 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174214+0000", 2023-11-05T08:33:59.586 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.2576000000000005e-05 2023-11-05T08:33:59.586 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.586 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:59.587 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.587 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.587 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.587 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:59.587 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.151849+0000", 2023-11-05T08:33:59.587 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 90.382439665000007, 2023-11-05T08:33:59.588 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022629176000000001, 2023-11-05T08:33:59.588 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:59.588 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:59.588 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:59.588 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:59.589 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:59.589 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 116 2023-11-05T08:33:59.589 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.589 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:59.589 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.590 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:59.590 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151849+0000", 2023-11-05T08:33:59.590 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.590 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.590 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.591 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:59.591 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151849+0000", 2023-11-05T08:33:59.591 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.591 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.591 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.592 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:59.592 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151850+0000", 2023-11-05T08:33:59.592 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.037e-06 2023-11-05T08:33:59.592 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.592 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.593 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:59.593 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151858+0000", 2023-11-05T08:33:59.593 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.3389999999999993e-06 2023-11-05T08:33:59.593 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.593 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.594 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:59.594 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151868+0000", 2023-11-05T08:33:59.594 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6670000000000008e-06 2023-11-05T08:33:59.594 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.594 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.595 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:59.595 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151875+0000", 2023-11-05T08:33:59.595 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9229999999999996e-06 2023-11-05T08:33:59.595 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.595 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.595 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:59.596 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.168777+0000", 2023-11-05T08:33:59.596 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.016901884999999998 2023-11-05T08:33:59.596 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.596 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.596 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:59.597 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.169009+0000", 2023-11-05T08:33:59.597 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00023236400000000001 2023-11-05T08:33:59.597 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.597 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.597 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:59.598 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.169365+0000", 2023-11-05T08:33:59.598 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00035625300000000001 2023-11-05T08:33:59.598 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.598 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.598 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:59.598 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.171624+0000", 2023-11-05T08:33:59.599 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0022587169999999999 2023-11-05T08:33:59.599 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.599 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.599 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:59.599 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174314+0000", 2023-11-05T08:33:59.600 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002690263 2023-11-05T08:33:59.600 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.600 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.600 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:59.600 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174405+0000", 2023-11-05T08:33:59.601 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1064999999999996e-05 2023-11-05T08:33:59.601 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.601 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.601 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:59.601 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174478+0000", 2023-11-05T08:33:59.602 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.2663000000000005e-05 2023-11-05T08:33:59.602 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.602 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:59.602 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.602 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.603 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.603 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:59.603 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.151886+0000", 2023-11-05T08:33:59.603 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 90.382402249999998, 2023-11-05T08:33:59.603 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.023174162000000002, 2023-11-05T08:33:59.603 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:59.604 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:59.604 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:59.604 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:59.604 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:59.604 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 117 2023-11-05T08:33:59.605 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.605 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:59.605 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.605 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:59.605 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151886+0000", 2023-11-05T08:33:59.606 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.606 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.606 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.606 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:59.606 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151886+0000", 2023-11-05T08:33:59.607 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.607 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.607 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.607 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:59.607 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151887+0000", 2023-11-05T08:33:59.608 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.2300000000000001e-07 2023-11-05T08:33:59.608 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.608 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.608 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:59.608 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151893+0000", 2023-11-05T08:33:59.609 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.19e-06 2023-11-05T08:33:59.609 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.609 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.609 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:59.609 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151905+0000", 2023-11-05T08:33:59.610 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1883e-05 2023-11-05T08:33:59.610 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.610 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.610 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:59.610 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.151912+0000", 2023-11-05T08:33:59.610 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.0690000000000004e-06 2023-11-05T08:33:59.611 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.611 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.611 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:59.611 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.170509+0000", 2023-11-05T08:33:59.611 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.018596686000000001 2023-11-05T08:33:59.612 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.612 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.612 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:59.612 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.170791+0000", 2023-11-05T08:33:59.612 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00028221799999999999 2023-11-05T08:33:59.613 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.613 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.613 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:59.613 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.171137+0000", 2023-11-05T08:33:59.613 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000346454 2023-11-05T08:33:59.614 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.614 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.614 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:59.614 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.172413+0000", 2023-11-05T08:33:59.614 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0012752340000000001 2023-11-05T08:33:59.615 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.615 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.615 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:59.615 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.174926+0000", 2023-11-05T08:33:59.615 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0025137050000000002 2023-11-05T08:33:59.616 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.616 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.616 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:59.616 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.175007+0000", 2023-11-05T08:33:59.616 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.0413000000000004e-05 2023-11-05T08:33:59.616 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.617 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.617 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:59.617 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.175060+0000", 2023-11-05T08:33:59.617 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3687000000000003e-05 2023-11-05T08:33:59.617 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.618 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:59.618 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.618 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.618 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.618 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:163 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:59.619 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.670890+0000", 2023-11-05T08:33:59.619 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 89.863398020999995, 2023-11-05T08:33:59.619 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.02132307, 2023-11-05T08:33:59.619 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:59.619 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:59.620 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:59.620 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:59.620 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:59.620 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 163 2023-11-05T08:33:59.620 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.621 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:59.621 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.621 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:59.621 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670890+0000", 2023-11-05T08:33:59.621 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.622 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.622 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.622 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:59.622 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670890+0000", 2023-11-05T08:33:59.622 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.622 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.623 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.623 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:59.623 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670891+0000", 2023-11-05T08:33:59.623 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3300000000000002e-07 2023-11-05T08:33:59.623 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.624 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.624 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:59.624 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670897+0000", 2023-11-05T08:33:59.624 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.4600000000000002e-06 2023-11-05T08:33:59.624 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.625 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.625 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:59.625 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670906+0000", 2023-11-05T08:33:59.625 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.3130000000000004e-06 2023-11-05T08:33:59.625 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.625 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.626 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:59.626 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670913+0000", 2023-11-05T08:33:59.626 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9859999999999997e-06 2023-11-05T08:33:59.626 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.626 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.627 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:59.627 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.687360+0000", 2023-11-05T08:33:59.627 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.016447244 2023-11-05T08:33:59.627 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.627 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.628 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:59.628 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.687588+0000", 2023-11-05T08:33:59.628 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00022757699999999999 2023-11-05T08:33:59.628 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.628 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.629 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:59.629 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.687946+0000", 2023-11-05T08:33:59.629 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00035789399999999999 2023-11-05T08:33:59.629 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.629 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.629 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:59.630 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.691929+0000", 2023-11-05T08:33:59.630 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0039835690000000002 2023-11-05T08:33:59.630 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.630 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.630 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:59.631 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.691970+0000", 2023-11-05T08:33:59.631 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.0351e-05 2023-11-05T08:33:59.631 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.631 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.631 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:59.632 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692111+0000", 2023-11-05T08:33:59.632 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014114499999999999 2023-11-05T08:33:59.632 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.632 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.632 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:59.633 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692214+0000", 2023-11-05T08:33:59.633 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000102898 2023-11-05T08:33:59.633 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.633 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:59.633 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.633 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.634 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.634 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:59.634 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.670924+0000", 2023-11-05T08:33:59.634 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 89.863364613000002, 2023-11-05T08:33:59.634 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.021621676999999999, 2023-11-05T08:33:59.635 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:59.635 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:59.635 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:59.635 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:59.635 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:59.636 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 164 2023-11-05T08:33:59.636 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.636 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:59.636 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.636 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:59.637 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670924+0000", 2023-11-05T08:33:59.637 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.637 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.637 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.637 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:59.637 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670924+0000", 2023-11-05T08:33:59.638 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.638 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.638 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.638 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:59.638 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.670925+0000", 2023-11-05T08:33:59.639 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.8299999999999996e-07 2023-11-05T08:33:59.639 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.639 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.639 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:59.639 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671022+0000", 2023-11-05T08:33:59.640 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.7445e-05 2023-11-05T08:33:59.640 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.640 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.640 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:59.640 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671033+0000", 2023-11-05T08:33:59.641 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1206000000000001e-05 2023-11-05T08:33:59.641 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.641 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.641 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:59.641 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671045+0000", 2023-11-05T08:33:59.642 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1797000000000001e-05 2023-11-05T08:33:59.642 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.642 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.642 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:59.642 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.688874+0000", 2023-11-05T08:33:59.643 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017829424 2023-11-05T08:33:59.643 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.643 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.643 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:59.643 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.689097+0000", 2023-11-05T08:33:59.644 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000222828 2023-11-05T08:33:59.644 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.644 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.644 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:59.644 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.689444+0000", 2023-11-05T08:33:59.645 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00034672099999999998 2023-11-05T08:33:59.645 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.645 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.645 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:59.645 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692240+0000", 2023-11-05T08:33:59.646 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0027959030000000002 2023-11-05T08:33:59.646 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.646 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.646 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:59.646 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692339+0000", 2023-11-05T08:33:59.647 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.8801000000000005e-05 2023-11-05T08:33:59.647 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.647 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.647 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:59.647 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692435+0000", 2023-11-05T08:33:59.648 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6311999999999995e-05 2023-11-05T08:33:59.648 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.648 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.648 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:59.648 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692546+0000", 2023-11-05T08:33:59.648 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00011055700000000001 2023-11-05T08:33:59.649 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.649 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:59.649 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.649 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.649 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.650 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:59.650 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.671060+0000", 2023-11-05T08:33:59.650 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 89.863228946999996, 2023-11-05T08:33:59.650 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022080292000000001, 2023-11-05T08:33:59.650 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:59.651 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:59.651 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:59.651 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:59.651 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:59.651 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 165 2023-11-05T08:33:59.652 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.652 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:59.652 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.652 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:59.652 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671060+0000", 2023-11-05T08:33:59.653 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.653 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.653 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.653 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:59.653 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671060+0000", 2023-11-05T08:33:59.653 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.654 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.654 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.654 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:59.654 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671060+0000", 2023-11-05T08:33:59.654 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.1699999999999997e-07 2023-11-05T08:33:59.655 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.655 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.655 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:59.655 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671067+0000", 2023-11-05T08:33:59.655 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5690000000000001e-06 2023-11-05T08:33:59.655 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.656 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.656 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:59.656 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671076+0000", 2023-11-05T08:33:59.656 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.5270000000000004e-06 2023-11-05T08:33:59.656 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.657 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.657 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:59.657 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.671084+0000", 2023-11-05T08:33:59.657 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.3459999999999996e-06 2023-11-05T08:33:59.657 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.657 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.658 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:59.658 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.690832+0000", 2023-11-05T08:33:59.658 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.019747793 2023-11-05T08:33:59.658 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.658 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.659 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:59.659 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.690971+0000", 2023-11-05T08:33:59.659 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000139245 2023-11-05T08:33:59.659 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.659 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.659 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:59.660 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.691178+0000", 2023-11-05T08:33:59.660 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000206892 2023-11-05T08:33:59.660 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.660 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.660 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:59.661 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692704+0000", 2023-11-05T08:33:59.661 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00152655 2023-11-05T08:33:59.661 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.661 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.661 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:59.661 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.692904+0000", 2023-11-05T08:33:59.662 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000199376 2023-11-05T08:33:59.662 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.662 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.662 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:59.662 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.693044+0000", 2023-11-05T08:33:59.663 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013989199999999999 2023-11-05T08:33:59.663 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.663 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.663 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:59.663 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.693140+0000", 2023-11-05T08:33:59.664 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6285000000000004e-05 2023-11-05T08:33:59.664 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.664 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:59.664 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.664 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.665 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.665 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:185 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:59.665 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.866212+0000", 2023-11-05T08:33:59.665 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 89.668076952000007, 2023-11-05T08:33:59.665 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.019448929, 2023-11-05T08:33:59.666 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:59.666 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:59.666 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:59.666 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:59.666 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:59.667 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 185 2023-11-05T08:33:59.667 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.667 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:59.667 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.667 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:59.668 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866212+0000", 2023-11-05T08:33:59.668 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.668 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.668 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.668 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:59.669 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866212+0000", 2023-11-05T08:33:59.669 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.669 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.669 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.669 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:59.670 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866213+0000", 2023-11-05T08:33:59.670 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0559999999999999e-06 2023-11-05T08:33:59.670 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.670 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.670 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:59.671 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866308+0000", 2023-11-05T08:33:59.671 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.4984999999999999e-05 2023-11-05T08:33:59.671 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.671 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.671 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:59.672 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866320+0000", 2023-11-05T08:33:59.672 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2639e-05 2023-11-05T08:33:59.672 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.672 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.672 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:59.672 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866328+0000", 2023-11-05T08:33:59.673 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.1240000000000005e-06 2023-11-05T08:33:59.673 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.673 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.673 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:59.673 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.879899+0000", 2023-11-05T08:33:59.674 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.013571059999999999 2023-11-05T08:33:59.674 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.674 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.674 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:59.674 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.880034+0000", 2023-11-05T08:33:59.675 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000134676 2023-11-05T08:33:59.675 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.675 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.675 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:59.675 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.880258+0000", 2023-11-05T08:33:59.675 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00022358100000000001 2023-11-05T08:33:59.676 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.676 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.676 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:59.676 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885299+0000", 2023-11-05T08:33:59.676 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.005041678 2023-11-05T08:33:59.677 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.677 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.677 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:59.677 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885436+0000", 2023-11-05T08:33:59.677 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000136472 2023-11-05T08:33:59.678 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.678 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.678 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:59.678 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885557+0000", 2023-11-05T08:33:59.678 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000121644 2023-11-05T08:33:59.679 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.679 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.679 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:59.679 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885660+0000", 2023-11-05T08:33:59.679 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000103014 2023-11-05T08:33:59.679 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.680 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:59.680 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.680 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.680 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.680 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:186 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:59.681 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.866343+0000", 2023-11-05T08:33:59.681 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 89.667945008999993, 2023-11-05T08:33:59.681 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.021691589000000001, 2023-11-05T08:33:59.681 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:59.681 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:59.682 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:59.682 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:59.682 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:59.682 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 186 2023-11-05T08:33:59.682 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.683 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:59.683 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.683 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:59.683 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866343+0000", 2023-11-05T08:33:59.683 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.684 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.684 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.684 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:59.684 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866343+0000", 2023-11-05T08:33:59.684 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.685 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.685 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.685 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:59.685 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866344+0000", 2023-11-05T08:33:59.685 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.6000000000000003e-07 2023-11-05T08:33:59.685 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.686 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.686 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:59.686 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866351+0000", 2023-11-05T08:33:59.686 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3090000000000001e-06 2023-11-05T08:33:59.686 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.687 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.687 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:59.687 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866360+0000", 2023-11-05T08:33:59.687 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.2599999999999994e-06 2023-11-05T08:33:59.687 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.688 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.688 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:59.688 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866367+0000", 2023-11-05T08:33:59.688 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.7599999999999997e-06 2023-11-05T08:33:59.688 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.688 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.689 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:59.689 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.882269+0000", 2023-11-05T08:33:59.689 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.015902218999999999 2023-11-05T08:33:59.689 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.689 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.690 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:59.690 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.882545+0000", 2023-11-05T08:33:59.690 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00027617199999999998 2023-11-05T08:33:59.690 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.690 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.691 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:59.691 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.882925+0000", 2023-11-05T08:33:59.691 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00037995900000000003 2023-11-05T08:33:59.691 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.691 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.691 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:59.692 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.887718+0000", 2023-11-05T08:33:59.692 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0047935069999999998 2023-11-05T08:33:59.692 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.692 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.692 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:59.693 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.887836+0000", 2023-11-05T08:33:59.693 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000117401 2023-11-05T08:33:59.693 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.693 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.693 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:59.694 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.887931+0000", 2023-11-05T08:33:59.694 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.4802000000000005e-05 2023-11-05T08:33:59.694 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.694 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.694 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:59.694 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888035+0000", 2023-11-05T08:33:59.695 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010444 2023-11-05T08:33:59.695 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.695 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:59.695 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.695 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.696 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.696 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:187 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:59.696 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.866377+0000", 2023-11-05T08:33:59.696 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 89.667911047999993, 2023-11-05T08:33:59.696 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022216751, 2023-11-05T08:33:59.697 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:59.697 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:59.697 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:59.697 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:59.697 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:59.698 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 187 2023-11-05T08:33:59.698 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.698 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:59.698 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.698 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:59.698 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866377+0000", 2023-11-05T08:33:59.699 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.699 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.699 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.699 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:59.699 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866377+0000", 2023-11-05T08:33:59.700 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.700 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.700 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.700 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:59.700 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866378+0000", 2023-11-05T08:33:59.701 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9999999999999997e-07 2023-11-05T08:33:59.701 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.701 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.701 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:59.701 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866384+0000", 2023-11-05T08:33:59.701 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.4099999999999999e-06 2023-11-05T08:33:59.702 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.702 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.702 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:59.702 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866393+0000", 2023-11-05T08:33:59.702 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9719999999999998e-06 2023-11-05T08:33:59.703 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.703 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.703 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:59.703 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866400+0000", 2023-11-05T08:33:59.703 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.2740000000000002e-06 2023-11-05T08:33:59.703 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.704 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.704 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:59.704 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.883936+0000", 2023-11-05T08:33:59.704 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017536513 2023-11-05T08:33:59.704 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.705 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.705 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:59.705 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.884061+0000", 2023-11-05T08:33:59.705 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012435300000000001 2023-11-05T08:33:59.705 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.705 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.706 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:59.706 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.884277+0000", 2023-11-05T08:33:59.706 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00021665500000000001 2023-11-05T08:33:59.706 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.706 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.707 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:59.707 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888193+0000", 2023-11-05T08:33:59.707 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0039160319999999998 2023-11-05T08:33:59.709 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.709 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.710 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:59.710 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888235+0000", 2023-11-05T08:33:59.710 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1267999999999999e-05 2023-11-05T08:33:59.710 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.710 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.711 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:59.711 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888373+0000", 2023-11-05T08:33:59.711 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000138789 2023-11-05T08:33:59.711 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.711 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.712 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:59.712 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888594+0000", 2023-11-05T08:33:59.712 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00022078500000000001 2023-11-05T08:33:59.712 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.712 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:59.713 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.713 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.713 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.713 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:188 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:59.713 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.866411+0000", 2023-11-05T08:33:59.714 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 89.667877786999995, 2023-11-05T08:33:59.714 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.023678187999999999, 2023-11-05T08:33:59.714 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:59.714 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:59.714 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:59.715 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:59.715 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:59.715 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 188 2023-11-05T08:33:59.715 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.715 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:59.716 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.716 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:59.716 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866411+0000", 2023-11-05T08:33:59.716 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.716 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.717 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.717 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:59.717 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866411+0000", 2023-11-05T08:33:59.717 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.717 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.717 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.718 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:59.718 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866411+0000", 2023-11-05T08:33:59.718 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.44e-07 2023-11-05T08:33:59.718 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.718 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.719 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:59.719 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866531+0000", 2023-11-05T08:33:59.719 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000119664 2023-11-05T08:33:59.719 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.719 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.720 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:59.720 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866545+0000", 2023-11-05T08:33:59.720 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3875e-05 2023-11-05T08:33:59.720 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.720 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.721 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:59.721 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866554+0000", 2023-11-05T08:33:59.721 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9099999999999994e-06 2023-11-05T08:33:59.721 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.721 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.722 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:59.722 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.885784+0000", 2023-11-05T08:33:59.722 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.019230651000000001 2023-11-05T08:33:59.722 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.722 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.723 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:59.723 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.886020+0000", 2023-11-05T08:33:59.723 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00023538300000000001 2023-11-05T08:33:59.723 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.723 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.724 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:59.724 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.886386+0000", 2023-11-05T08:33:59.724 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00036659399999999998 2023-11-05T08:33:59.724 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.724 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.724 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:59.725 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888662+0000", 2023-11-05T08:33:59.725 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0022756019999999998 2023-11-05T08:33:59.725 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.725 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.725 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:59.726 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.889922+0000", 2023-11-05T08:33:59.726 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0012603149999999999 2023-11-05T08:33:59.726 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.726 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.726 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:59.727 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.890034+0000", 2023-11-05T08:33:59.727 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000111283 2023-11-05T08:33:59.727 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.727 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.727 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:59.728 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.890089+0000", 2023-11-05T08:33:59.728 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5266999999999998e-05 2023-11-05T08:33:59.728 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.728 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:59.728 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.729 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.729 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.729 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4281.0:189 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-11-05T08:33:59.729 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:29.866572+0000", 2023-11-05T08:33:59.729 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 89.667716829, 2023-11-05T08:33:59.730 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.024895937999999999, 2023-11-05T08:33:59.730 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:59.730 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:59.730 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:59.730 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4281", 2023-11-05T08:33:59.731 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.2:0/2732825876", 2023-11-05T08:33:59.731 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 189 2023-11-05T08:33:59.731 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.731 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:59.731 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.731 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:59.732 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866572+0000", 2023-11-05T08:33:59.732 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.732 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.732 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.732 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:59.733 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866572+0000", 2023-11-05T08:33:59.733 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.733 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.733 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.733 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:59.733 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866572+0000", 2023-11-05T08:33:59.734 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.8000000000000005e-07 2023-11-05T08:33:59.734 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.734 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.734 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:59.734 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866581+0000", 2023-11-05T08:33:59.734 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9800000000000004e-06 2023-11-05T08:33:59.735 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.735 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.735 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:59.735 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866591+0000", 2023-11-05T08:33:59.735 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6059999999999992e-06 2023-11-05T08:33:59.736 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.736 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.736 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:59.736 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.866598+0000", 2023-11-05T08:33:59.736 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.1029999999999996e-06 2023-11-05T08:33:59.736 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.737 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.737 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:59.737 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888850+0000", 2023-11-05T08:33:59.737 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022251615999999998 2023-11-05T08:33:59.737 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.738 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.738 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:59.738 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.888988+0000", 2023-11-05T08:33:59.738 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000138436 2023-11-05T08:33:59.738 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.739 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.739 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2", 2023-11-05T08:33:59.739 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.889206+0000", 2023-11-05T08:33:59.739 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00021818099999999999 2023-11-05T08:33:59.739 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.740 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.740 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2023-11-05T08:33:59.740 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.890621+0000", 2023-11-05T08:33:59.740 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001414219 2023-11-05T08:33:59.740 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.741 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.741 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2023-11-05T08:33:59.741 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.891316+0000", 2023-11-05T08:33:59.741 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00069570500000000002 2023-11-05T08:33:59.741 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.742 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.742 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:59.742 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.891406+0000", 2023-11-05T08:33:59.742 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9901999999999994e-05 2023-11-05T08:33:59.742 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.743 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.743 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:59.743 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:29.891468+0000", 2023-11-05T08:33:59.743 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1409999999999996e-05 2023-11-05T08:33:59.743 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.743 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:59.744 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.744 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.744 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.744 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:292 3.2s0 3:41aa5db9:::smithi09039252-7:head [delete] snapc 8=[8,7,6,5] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e41)", 2023-11-05T08:33:59.744 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:32:54.121023+0000", 2023-11-05T08:33:59.745 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 65.413265413999994, 2023-11-05T08:33:59.745 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.061679772000000001, 2023-11-05T08:33:59.745 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:59.745 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:59.745 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:59.745 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:59.746 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:59.746 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 292 2023-11-05T08:33:59.746 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.746 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:59.746 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.747 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:59.747 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.121023+0000", 2023-11-05T08:33:59.747 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.747 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.747 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.747 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:59.748 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.121023+0000", 2023-11-05T08:33:59.748 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.748 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.748 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.748 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:59.749 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.121026+0000", 2023-11-05T08:33:59.749 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0369999999999999e-06 2023-11-05T08:33:59.749 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.749 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.749 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:59.750 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.121039+0000", 2023-11-05T08:33:59.750 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3103e-05 2023-11-05T08:33:59.750 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.750 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.750 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:59.750 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.121086+0000", 2023-11-05T08:33:59.751 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7095000000000002e-05 2023-11-05T08:33:59.751 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.751 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.751 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:59.751 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.121131+0000", 2023-11-05T08:33:59.752 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4455e-05 2023-11-05T08:33:59.752 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.752 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.752 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:59.752 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.129136+0000", 2023-11-05T08:33:59.753 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0080047439999999994 2023-11-05T08:33:59.753 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.753 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.753 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2023-11-05T08:33:59.753 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.129142+0000", 2023-11-05T08:33:59.753 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3759999999999996e-06 2023-11-05T08:33:59.754 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.754 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.754 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:59.754 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.158095+0000", 2023-11-05T08:33:59.754 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.028953402 2023-11-05T08:33:59.755 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.755 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.755 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:59.755 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.158534+0000", 2023-11-05T08:33:59.755 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00043911800000000001 2023-11-05T08:33:59.755 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.756 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.756 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:59.756 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.160380+0000", 2023-11-05T08:33:59.756 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001845923 2023-11-05T08:33:59.756 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.756 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.757 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:59.757 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.177604+0000", 2023-11-05T08:33:59.757 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017223405000000001 2023-11-05T08:33:59.757 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.757 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.758 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:59.758 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.182556+0000", 2023-11-05T08:33:59.758 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0049522740000000004 2023-11-05T08:33:59.758 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.758 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.758 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:59.759 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:32:54.182703+0000", 2023-11-05T08:33:59.759 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014684 2023-11-05T08:33:59.759 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.759 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:59.759 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.760 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.760 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.760 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:335 3.18s0 3:1aa38538:::smithi09039252-11:head [stat,delete] snapc 10=[10,7] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2023-11-05T08:33:59.760 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:04.225103+0000", 2023-11-05T08:33:59.760 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 55.309185540000001, 2023-11-05T08:33:59.760 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.028941109999999999, 2023-11-05T08:33:59.761 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:59.761 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:59.761 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:59.761 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:59.761 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:59.762 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 335 2023-11-05T08:33:59.762 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.762 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:59.762 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.762 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:59.763 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.225103+0000", 2023-11-05T08:33:59.763 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.763 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.763 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.763 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:59.763 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.225103+0000", 2023-11-05T08:33:59.764 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.764 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.764 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.764 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:59.764 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.225106+0000", 2023-11-05T08:33:59.765 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0929999999999999e-06 2023-11-05T08:33:59.765 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.765 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.765 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:59.765 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.225118+0000", 2023-11-05T08:33:59.766 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2017e-05 2023-11-05T08:33:59.766 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.766 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.766 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:59.766 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.225155+0000", 2023-11-05T08:33:59.767 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.6711999999999997e-05 2023-11-05T08:33:59.767 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.767 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.767 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:59.767 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.225172+0000", 2023-11-05T08:33:59.768 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7367000000000002e-05 2023-11-05T08:33:59.768 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.768 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.768 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:59.768 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.225312+0000", 2023-11-05T08:33:59.769 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000140323 2023-11-05T08:33:59.769 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.769 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.769 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op must wait for map", 2023-11-05T08:33:59.769 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.225355+0000", 2023-11-05T08:33:59.770 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.2746e-05 2023-11-05T08:33:59.770 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.770 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.770 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:59.770 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.237049+0000", 2023-11-05T08:33:59.771 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011693360999999999 2023-11-05T08:33:59.771 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.771 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.771 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:59.771 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.237652+0000", 2023-11-05T08:33:59.772 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000603749 2023-11-05T08:33:59.772 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.772 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.772 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:59.772 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.245013+0000", 2023-11-05T08:33:59.772 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0073607000000000004 2023-11-05T08:33:59.773 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.773 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.773 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:59.773 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.249533+0000", 2023-11-05T08:33:59.773 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0045203939999999996 2023-11-05T08:33:59.774 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.774 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.774 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:59.774 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.253639+0000", 2023-11-05T08:33:59.774 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0041051730000000002 2023-11-05T08:33:59.775 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.775 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.775 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:59.775 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:04.254044+0000", 2023-11-05T08:33:59.775 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00040547499999999998 2023-11-05T08:33:59.776 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.776 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:59.776 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.776 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.776 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.777 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(osd.5.22:1 3.10s0 3:084c43b8:::smithi09039252-5:head [assert-version v10,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:59.777 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:05.258544+0000", 2023-11-05T08:33:59.777 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 54.275744854000003, 2023-11-05T08:33:59.777 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.024845459, 2023-11-05T08:33:59.777 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:59.777 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2023-11-05T08:33:59.778 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:59.778 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.778 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:59.778 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258544+0000", 2023-11-05T08:33:59.778 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.779 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.779 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.779 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:59.779 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258544+0000", 2023-11-05T08:33:59.779 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.779 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.780 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.780 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:59.780 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258547+0000", 2023-11-05T08:33:59.780 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0740000000000001e-06 2023-11-05T08:33:59.780 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.781 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.781 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:59.781 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258561+0000", 2023-11-05T08:33:59.781 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4525999999999999e-05 2023-11-05T08:33:59.781 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.781 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.782 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:59.782 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258601+0000", 2023-11-05T08:33:59.782 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.9433e-05 2023-11-05T08:33:59.782 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.782 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.783 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:59.783 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258625+0000", 2023-11-05T08:33:59.783 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3819e-05 2023-11-05T08:33:59.783 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.783 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.784 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:59.784 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258788+0000", 2023-11-05T08:33:59.784 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00016302600000000001 2023-11-05T08:33:59.784 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.784 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.785 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op must wait for map", 2023-11-05T08:33:59.785 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.258812+0000", 2023-11-05T08:33:59.785 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4043e-05 2023-11-05T08:33:59.785 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.785 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.786 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:59.786 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.275660+0000", 2023-11-05T08:33:59.786 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.016848823999999998 2023-11-05T08:33:59.786 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.786 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.787 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:59.787 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.276035+0000", 2023-11-05T08:33:59.787 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00037483200000000002 2023-11-05T08:33:59.787 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.787 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.788 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:59.788 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.283389+0000", 2023-11-05T08:33:59.788 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.007353882 2023-11-05T08:33:59.788 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.788 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:59.788 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.789 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.789 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.789 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:351 3.18s0 3:1aa38538:::smithi09039252-11:head [append 0~491520] snapc 10=[7] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:59.789 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:05.264529+0000", 2023-11-05T08:33:59.789 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 54.269759626000003, 2023-11-05T08:33:59.790 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.029015691, 2023-11-05T08:33:59.790 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:59.790 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:59.790 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:59.790 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:59.791 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:59.791 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 351 2023-11-05T08:33:59.791 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.791 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:59.791 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.792 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:59.792 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.264529+0000", 2023-11-05T08:33:59.792 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.792 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.792 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.793 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:59.793 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.264529+0000", 2023-11-05T08:33:59.793 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.793 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.793 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.793 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:59.794 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.264532+0000", 2023-11-05T08:33:59.794 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3730000000000001e-06 2023-11-05T08:33:59.794 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.794 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.795 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:59.795 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.266714+0000", 2023-11-05T08:33:59.795 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0021819880000000002 2023-11-05T08:33:59.795 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.795 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.795 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:59.796 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.266743+0000", 2023-11-05T08:33:59.796 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9218999999999999e-05 2023-11-05T08:33:59.796 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.796 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.796 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:59.797 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.266759+0000", 2023-11-05T08:33:59.797 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5102999999999999e-05 2023-11-05T08:33:59.797 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.797 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.797 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:59.798 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.266928+0000", 2023-11-05T08:33:59.798 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00016909299999999999 2023-11-05T08:33:59.798 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.798 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.798 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op must wait for map", 2023-11-05T08:33:59.799 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.266953+0000", 2023-11-05T08:33:59.799 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5344999999999999e-05 2023-11-05T08:33:59.799 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.799 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.799 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:59.800 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.276760+0000", 2023-11-05T08:33:59.800 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0098067109999999992 2023-11-05T08:33:59.800 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.800 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.800 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:59.800 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.277064+0000", 2023-11-05T08:33:59.801 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00030468299999999998 2023-11-05T08:33:59.801 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.801 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.801 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:59.801 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.279923+0000", 2023-11-05T08:33:59.802 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0028589119999999999 2023-11-05T08:33:59.802 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.802 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.802 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:59.802 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.292166+0000", 2023-11-05T08:33:59.803 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012242723 2023-11-05T08:33:59.803 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.803 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.803 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:59.803 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.292949+0000", 2023-11-05T08:33:59.803 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00078316699999999998 2023-11-05T08:33:59.804 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.804 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.804 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:59.804 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.293545+0000", 2023-11-05T08:33:59.804 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00059537399999999997 2023-11-05T08:33:59.805 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.805 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:59.805 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.805 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.805 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.806 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:354 3.18s0 3:1aa38538:::smithi09039252-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:33:59.806 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:05.281130+0000", 2023-11-05T08:33:59.806 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 54.253158032999998, 2023-11-05T08:33:59.806 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.021364054, 2023-11-05T08:33:59.806 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:59.807 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2023-11-05T08:33:59.807 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:59.807 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:59.807 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:59.807 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 354 2023-11-05T08:33:59.808 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.808 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:59.808 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.808 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:59.808 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.281130+0000", 2023-11-05T08:33:59.808 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.809 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.809 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.809 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:59.809 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.281130+0000", 2023-11-05T08:33:59.809 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.810 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.810 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.810 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:59.810 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.281131+0000", 2023-11-05T08:33:59.810 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.0299999999999998e-07 2023-11-05T08:33:59.811 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.811 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.811 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:59.811 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.281135+0000", 2023-11-05T08:33:59.811 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2799999999999999e-06 2023-11-05T08:33:59.812 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.812 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.812 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:59.812 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.281146+0000", 2023-11-05T08:33:59.812 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.119e-05 2023-11-05T08:33:59.812 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.813 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.813 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:59.813 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.281154+0000", 2023-11-05T08:33:59.813 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.7719999999999994e-06 2023-11-05T08:33:59.813 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.814 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.814 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:59.814 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.298629+0000", 2023-11-05T08:33:59.814 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017475195999999998 2023-11-05T08:33:59.814 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.815 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.815 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:59.815 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.298839+0000", 2023-11-05T08:33:59.815 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00020991799999999999 2023-11-05T08:33:59.815 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.816 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.816 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:59.816 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:05.302495+0000", 2023-11-05T08:33:59.816 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0036558950000000002 2023-11-05T08:33:59.816 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.817 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:59.817 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.817 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.817 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.817 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:638 3.10s0 3:0a3c6106:::smithi09039252-48:head [create,setxattr TahZbq^WXaahVeQfQsXqNXgcd (1963) in=25b,setxattr `Pj`q[kZqXoPYYj_`hNV^bohgce[Ull^lRXTctln (982) in=40b,setxattr ^gfWYcnTRmaWZQkrf`asZ\\XVPcRmRdRckrhUheci (10312) in=40b,setxattr hVm[Q^c]jmYfcedO_eoe[N_fohZ^nghUPgNQYY`h (15648) in=40b,setxattr \\jlYtrYeh_rthhUaqacc]`Uger\\]rPngbjPhbaWV (454) in=40b,setxattr nOgORnqOheegXnS]tSkTQTreXqqt`ojnVPsQbabf (771) in=40b,setxattr QR]c[o[]SToV\\goc\\pjtRRf`WYSngR[eQ`taQgcc (12488) in=40b,setxattr pY\\tNq\\a[n\\fajnSVYNng[\\kcqZ^VhqNmqRXUTlO (1121) in=40b,setxattr [laaiUa\\cY`kdXP\\SNRq[ig]rcanlRgnTtdfZi_W (1403) in=40b,setxattr aaf[PNUURnfSbT^fR_ge[lfbq]keQjYgrbO\\SbTa (1226) in=40b,setxattr SfaWR`YpWpbnn]sR]^eO[Wbi\\Se_TkrcQp[pcomR (4768) in=40b,setxattr pS^PVQeeOqs_S^YVlb^n\\h``m_Q\\prek^_qjcmgR (1121) in=40b,setxattr [aWfmed^jZheSjc^Xmb]TrYaU[[c]jb\\[UdNQngi (593) in=40b,setxattr kPO^emkeWSaQtnWaRVXk[hbN[TP^]ThcqpOfordV (2805) in=40b,setxattr PkhW`mV]l_O[nUSprX^tRPoRakfUNbOiRedpZb^n (421) in=40b,setxattr [`esP (1963) in=5b,setxattr SlhkjUdadks\\hNrmRohr_fl`itUjb_Vjia\\Vdqf` (10033) in=40b,setxattr TUWTXRdZcQc\\lOTbYmrWiirilafTQQrNdPoj_^sZ (3084) in=40b,setxattr ddbfaqshSg (1963) in=10b,setxattr nSgahmseaQO^hNUcamf\\m[SsQam^Uc``at``i`kU (1435) in=40b] snapc 34=[2f,7] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2023-11-05T08:33:59.818 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:40.606065+0000", 2023-11-05T08:33:59.818 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 18.92822348, 2023-11-05T08:33:59.818 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022517651, 2023-11-05T08:33:59.818 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:59.818 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:59.818 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:59.819 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:59.819 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:59.819 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 638 2023-11-05T08:33:59.819 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.819 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:59.820 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.820 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:59.820 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:40.606065+0000", 2023-11-05T08:33:59.820 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.820 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.820 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.821 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:59.821 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:40.606065+0000", 2023-11-05T08:33:59.821 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.821 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.821 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.822 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:59.822 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:40.606068+0000", 2023-11-05T08:33:59.822 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.01e-06 2023-11-05T08:33:59.822 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.822 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.823 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:59.823 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:40.606338+0000", 2023-11-05T08:33:59.823 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00026997799999999998 2023-11-05T08:33:59.823 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.823 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.824 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:59.824 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:40.606371+0000", 2023-11-05T08:33:59.824 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3278999999999998e-05 2023-11-05T08:33:59.824 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.824 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.825 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:59.825 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:40.606388+0000", 2023-11-05T08:33:59.825 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7073000000000001e-05 2023-11-05T08:33:59.825 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.825 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.825 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:59.826 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:40.606499+0000", 2023-11-05T08:33:59.826 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000110962 2023-11-05T08:33:59.826 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.826 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.826 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op must wait for map", 2023-11-05T08:33:59.827 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:40.606515+0000", 2023-11-05T08:33:59.827 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5352000000000001e-05 2023-11-05T08:33:59.827 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.827 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.827 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:59.828 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:40.614281+0000", 2023-11-05T08:33:59.828 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0077663560000000003 2023-11-05T08:33:59.828 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.828 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.828 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:59.828 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:40.614375+0000", 2023-11-05T08:33:59.829 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.4346999999999996e-05 2023-11-05T08:33:59.829 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.829 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.829 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:59.829 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:40.615206+0000", 2023-11-05T08:33:59.830 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00083111000000000003 2023-11-05T08:33:59.830 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.830 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.830 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:59.830 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:40.619735+0000", 2023-11-05T08:33:59.831 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0045284009999999996 2023-11-05T08:33:59.831 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.831 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.831 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:59.831 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:40.628377+0000", 2023-11-05T08:33:59.832 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0086421710000000006 2023-11-05T08:33:59.832 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.832 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.832 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:59.832 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:40.628583+0000", 2023-11-05T08:33:59.833 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00020561200000000001 2023-11-05T08:33:59.833 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.833 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:59.833 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.833 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.834 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.834 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.4691.0:806 3.2s0 3:41aa5db9:::smithi09039252-7:head [append 0~573440] snapc 44=[44,42,41] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2023-11-05T08:33:59.834 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2023-11-05T08:33:55.754990+0000", 2023-11-05T08:33:59.834 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 3.7792983769999999, 2023-11-05T08:33:59.834 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.022339569, 2023-11-05T08:33:59.835 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2023-11-05T08:33:59.835 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:33:59.835 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2023-11-05T08:33:59.835 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.4691", 2023-11-05T08:33:59.835 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:33:59.835 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 806 2023-11-05T08:33:59.836 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.836 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2023-11-05T08:33:59.836 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.836 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2023-11-05T08:33:59.836 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:55.754990+0000", 2023-11-05T08:33:59.837 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.837 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.837 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.837 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2023-11-05T08:33:59.837 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:55.754990+0000", 2023-11-05T08:33:59.838 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2023-11-05T08:33:59.838 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.838 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.838 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2023-11-05T08:33:59.838 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:55.754993+0000", 2023-11-05T08:33:59.839 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3060000000000002e-06 2023-11-05T08:33:59.839 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.839 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.839 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2023-11-05T08:33:59.839 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:55.756131+0000", 2023-11-05T08:33:59.839 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001137999 2023-11-05T08:33:59.840 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.840 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.840 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2023-11-05T08:33:59.840 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:55.756160+0000", 2023-11-05T08:33:59.840 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8887e-05 2023-11-05T08:33:59.841 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.841 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.841 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2023-11-05T08:33:59.841 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:55.756176+0000", 2023-11-05T08:33:59.841 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5760000000000002e-05 2023-11-05T08:33:59.842 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.842 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.842 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2023-11-05T08:33:59.842 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:55.756339+0000", 2023-11-05T08:33:59.842 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00016287300000000001 2023-11-05T08:33:59.843 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.843 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.843 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2023-11-05T08:33:59.843 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:55.756569+0000", 2023-11-05T08:33:59.843 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00022991400000000001 2023-11-05T08:33:59.843 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.844 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.844 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_started", 2023-11-05T08:33:59.844 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:55.759101+0000", 2023-11-05T08:33:59.844 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0025323730000000001 2023-11-05T08:33:59.844 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.845 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.845 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_committed", 2023-11-05T08:33:59.845 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:55.768990+0000", 2023-11-05T08:33:59.845 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0098887479999999993 2023-11-05T08:33:59.845 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.846 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.846 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2023-11-05T08:33:59.846 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:55.777113+0000", 2023-11-05T08:33:59.846 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0081231259999999996 2023-11-05T08:33:59.846 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2023-11-05T08:33:59.847 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-11-05T08:33:59.847 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2023-11-05T08:33:59.847 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2023-11-05T08:33:55.777330+0000", 2023-11-05T08:33:59.847 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00021658300000000001 2023-11-05T08:33:59.847 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.847 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:59.848 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.848 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-11-05T08:33:59.848 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-11-05T08:33:59.848 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:33:59.850 INFO:teuthology.orchestra.run.smithi002.stderr:noscrub is unset 2023-11-05T08:33:59.850 INFO:teuthology.orchestra.run.smithi002.stderr:reweighted osd.12 to 1 (10000) 2023-11-05T08:33:59.851 INFO:tasks.rados.rados.0.smithi090.stdout:365: done (2 left) 2023-11-05T08:33:59.852 INFO:tasks.rados.rados.0.smithi090.stdout:366: snap_create 2023-11-05T08:33:59.852 INFO:tasks.rados.rados.0.smithi090.stdout:362: finishing write tid 1 to smithi09039252-39 2023-11-05T08:33:59.852 INFO:tasks.rados.rados.0.smithi090.stdout:363: finishing rollback tid 0 to smithi09039252-35 2023-11-05T08:33:59.852 INFO:tasks.rados.rados.0.smithi090.stdout:363: finishing rollback tid 1 to smithi09039252-35 2023-11-05T08:33:59.852 INFO:tasks.rados.rados.0.smithi090.stdout:362: finishing write tid 2 to smithi09039252-39 2023-11-05T08:33:59.853 INFO:tasks.rados.rados.0.smithi090.stdout:363: finishing rollback tid 2 to smithi09039252-35 2023-11-05T08:33:59.853 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 35 v 31 (ObjNum 136 snap 36 seq_num 136) dirty exists 2023-11-05T08:33:59.853 INFO:tasks.rados.rados.0.smithi090.stdout:362: finishing write tid 3 to smithi09039252-39 2023-11-05T08:33:59.853 INFO:tasks.rados.rados.0.smithi090.stdout:362: finishing write tid 4 to smithi09039252-39 2023-11-05T08:33:59.853 INFO:tasks.rados.rados.0.smithi090.stdout:362: finishing write tid 5 to smithi09039252-39 2023-11-05T08:33:59.854 INFO:tasks.rados.rados.0.smithi090.stdout:362: finishing write tid 6 to smithi09039252-39 2023-11-05T08:33:59.854 INFO:tasks.rados.rados.0.smithi090.stdout:362: oid 39 updating version 0 to 77 2023-11-05T08:33:59.854 INFO:tasks.rados.rados.0.smithi090.stdout:362: oid 39 updating version 77 to 79 2023-11-05T08:33:59.854 INFO:tasks.rados.rados.0.smithi090.stdout:362: oid 39 version 79 is already newer than 78 2023-11-05T08:33:59.854 INFO:tasks.rados.rados.0.smithi090.stdout:362: oid 39 version 79 is already newer than 76 2023-11-05T08:33:59.855 INFO:tasks.rados.rados.0.smithi090.stdout:362: oid 39 version 79 is already newer than 75 2023-11-05T08:33:59.855 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 39 v 79 (ObjNum 138 snap 37 seq_num 138) dirty exists 2023-11-05T08:33:59.855 INFO:tasks.rados.rados.0.smithi090.stdout:362: left oid 39 (ObjNum 138 snap 37 seq_num 138) 2023-11-05T08:33:59.856 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-11-05T08:33:59.857 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 13 1 2023-11-05T08:33:59.857 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:33:59.845+0000 7fdc5cd58640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:33:59.950 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:33:59.946+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:33:59.993 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:33:59.994 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2023-11-05T08:33:59.994 INFO:teuthology.orchestra.run.smithi069.stdout: "num_ops": 0 2023-11-05T08:33:59.994 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:34:00.007 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-11-05T08:34:00.051 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:34:00.048+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:34:00.125 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:34:00.126 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2023-11-05T08:34:00.126 INFO:teuthology.orchestra.run.smithi069.stdout: "complaint_time": 30, 2023-11-05T08:34:00.126 INFO:teuthology.orchestra.run.smithi069.stdout: "num_blocked_ops": 0 2023-11-05T08:34:00.126 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:34:00.139 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-11-05T08:34:00.152 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:34:00.149+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:34:00.253 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:34:00.248+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:34:00.261 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:34:00.262 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 20, 2023-11-05T08:34:00.262 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 600, 2023-11-05T08:34:00.262 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [ 2023-11-05T08:34:00.262 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.263 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:274 3.es0 3:74767c3c:::smithi09039252-8:head [append 0~737280] snapc 8=[8,7,6,5,3] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:34:00.263 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:53.134268+0000", 2023-11-05T08:34:00.263 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 67.125155972000002, 2023-11-05T08:34:00.263 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.014769740999999999, 2023-11-05T08:34:00.263 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:00.264 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:00.264 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:34:00.264 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:34:00.264 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:00.264 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 274 2023-11-05T08:34:00.264 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.265 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:00.265 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.265 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:00.265 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.134268+0000", 2023-11-05T08:34:00.266 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.266 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.266 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.266 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:00.266 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.134268+0000", 2023-11-05T08:34:00.266 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.267 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.267 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.267 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:00.267 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.134270+0000", 2023-11-05T08:34:00.267 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8560000000000001e-06 2023-11-05T08:34:00.268 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.268 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.268 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:00.268 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.135843+0000", 2023-11-05T08:34:00.268 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0015724210000000001 2023-11-05T08:34:00.269 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.269 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.269 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:00.269 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.135876+0000", 2023-11-05T08:34:00.269 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.3895e-05 2023-11-05T08:34:00.269 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.270 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.270 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:00.270 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.135929+0000", 2023-11-05T08:34:00.270 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.2930000000000003e-05 2023-11-05T08:34:00.270 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.271 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.271 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:00.271 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.136066+0000", 2023-11-05T08:34:00.271 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000136673 2023-11-05T08:34:00.271 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.271 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.272 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:34:00.272 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.136335+0000", 2023-11-05T08:34:00.272 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00026863599999999999 2023-11-05T08:34:00.272 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.272 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.273 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:00.273 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.138586+0000", 2023-11-05T08:34:00.273 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0022515629999999998 2023-11-05T08:34:00.273 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.273 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.274 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:00.274 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.148630+0000", 2023-11-05T08:34:00.274 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.010043778 2023-11-05T08:34:00.274 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.274 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.275 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:34:00.275 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.148798+0000", 2023-11-05T08:34:00.275 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00016756799999999999 2023-11-05T08:34:00.275 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.275 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.276 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:00.276 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.149038+0000", 2023-11-05T08:34:00.276 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00024042099999999999 2023-11-05T08:34:00.276 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.276 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:00.277 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.277 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.277 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.277 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:277 3.es0 3:74767c3c:::smithi09039252-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:34:00.277 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:53.143452+0000", 2023-11-05T08:34:00.278 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 67.115971989000002, 2023-11-05T08:34:00.278 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0099967070000000005, 2023-11-05T08:34:00.278 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:00.278 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2023-11-05T08:34:00.279 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:34:00.279 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:34:00.279 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:00.279 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 277 2023-11-05T08:34:00.279 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.279 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:00.280 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.280 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:00.280 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143452+0000", 2023-11-05T08:34:00.280 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.280 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.281 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.281 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:00.281 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143452+0000", 2023-11-05T08:34:00.281 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.281 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.282 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.282 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:00.282 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143453+0000", 2023-11-05T08:34:00.282 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0640000000000001e-06 2023-11-05T08:34:00.282 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.283 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.283 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:00.283 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143459+0000", 2023-11-05T08:34:00.283 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.5330000000000002e-06 2023-11-05T08:34:00.283 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.284 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.284 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:00.284 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143491+0000", 2023-11-05T08:34:00.284 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2156000000000002e-05 2023-11-05T08:34:00.284 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.285 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.285 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:00.285 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.143504+0000", 2023-11-05T08:34:00.285 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3156000000000001e-05 2023-11-05T08:34:00.285 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.286 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.286 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:00.286 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.150256+0000", 2023-11-05T08:34:00.286 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0067515980000000001 2023-11-05T08:34:00.286 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.287 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.287 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for rw locks", 2023-11-05T08:34:00.287 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.150342+0000", 2023-11-05T08:34:00.287 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.5951000000000004e-05 2023-11-05T08:34:00.287 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.288 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.288 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:00.288 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.151851+0000", 2023-11-05T08:34:00.288 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0015094590000000001 2023-11-05T08:34:00.288 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.289 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.289 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:34:00.289 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.151933+0000", 2023-11-05T08:34:00.289 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.1898999999999996e-05 2023-11-05T08:34:00.289 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.289 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.290 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:00.290 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:53.153449+0000", 2023-11-05T08:34:00.290 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0015158909999999999 2023-11-05T08:34:00.290 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.290 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:00.291 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.291 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.291 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.291 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:380 3.es0 3:7b35d524:::smithi09039252-34:head [delete] snapc 17=[17,7] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e57)", 2023-11-05T08:34:00.291 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:10.278638+0000", 2023-11-05T08:34:00.292 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 49.980786010000003, 2023-11-05T08:34:00.292 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.010515907, 2023-11-05T08:34:00.292 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:00.292 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:00.292 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:34:00.292 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:34:00.293 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:00.293 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 380 2023-11-05T08:34:00.293 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.293 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:00.293 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.294 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:00.294 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:10.278638+0000", 2023-11-05T08:34:00.294 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.294 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.294 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.295 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:00.295 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:10.278638+0000", 2023-11-05T08:34:00.295 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.295 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.295 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.296 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:00.296 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:10.278641+0000", 2023-11-05T08:34:00.296 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.9100000000000001e-06 2023-11-05T08:34:00.296 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.296 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.297 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:00.297 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:10.278654+0000", 2023-11-05T08:34:00.297 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2532e-05 2023-11-05T08:34:00.297 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.297 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.297 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:00.298 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:10.278690+0000", 2023-11-05T08:34:00.298 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.6486000000000003e-05 2023-11-05T08:34:00.298 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.298 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.298 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:00.299 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:10.278708+0000", 2023-11-05T08:34:00.299 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8142e-05 2023-11-05T08:34:00.299 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.299 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.299 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:00.300 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:10.278888+0000", 2023-11-05T08:34:00.300 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00017937399999999999 2023-11-05T08:34:00.300 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.300 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.300 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op must wait for map", 2023-11-05T08:34:00.301 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:10.278913+0000", 2023-11-05T08:34:00.301 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5488999999999999e-05 2023-11-05T08:34:00.301 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.301 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.301 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:00.302 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:10.283146+0000", 2023-11-05T08:34:00.302 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.004232446 2023-11-05T08:34:00.302 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.302 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.302 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:34:00.302 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:10.283306+0000", 2023-11-05T08:34:00.303 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00015997400000000001 2023-11-05T08:34:00.303 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.303 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.303 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:00.303 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:10.284790+0000", 2023-11-05T08:34:00.304 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00148408 2023-11-05T08:34:00.304 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.304 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.304 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:00.304 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:10.287627+0000", 2023-11-05T08:34:00.305 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0028376489999999998 2023-11-05T08:34:00.305 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.305 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.305 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:34:00.305 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:10.289000+0000", 2023-11-05T08:34:00.306 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001372828 2023-11-05T08:34:00.306 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.306 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.306 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:00.307 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:10.289154+0000", 2023-11-05T08:34:00.307 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00015399700000000001 2023-11-05T08:34:00.307 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.307 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:00.308 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.308 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.308 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.308 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.2s1 65/45 ECSubWrite(tid=243, reqid=client.4691.0:470, at_version=65'18, trim_to=0'0, roll_forward_to=55'16))", 2023-11-05T08:34:00.308 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:18.554843+0000", 2023-11-05T08:34:00.309 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 41.704581556000001, 2023-11-05T08:34:00.309 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0095377659999999996, 2023-11-05T08:34:00.309 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:00.309 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:00.309 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:00.310 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.310 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:00.310 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.554843+0000", 2023-11-05T08:34:00.310 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.310 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.311 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.311 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:00.311 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.554843+0000", 2023-11-05T08:34:00.311 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.311 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.312 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.312 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:00.312 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.554844+0000", 2023-11-05T08:34:00.312 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.353e-06 2023-11-05T08:34:00.312 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.313 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.313 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:00.313 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.556081+0000", 2023-11-05T08:34:00.313 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0012370700000000001 2023-11-05T08:34:00.313 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.314 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.314 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:00.314 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.556114+0000", 2023-11-05T08:34:00.314 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2792e-05 2023-11-05T08:34:00.314 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.314 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.315 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:00.315 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.556169+0000", 2023-11-05T08:34:00.315 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.5331000000000003e-05 2023-11-05T08:34:00.315 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.315 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.316 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:00.316 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.556309+0000", 2023-11-05T08:34:00.316 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000139906 2023-11-05T08:34:00.316 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.316 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.317 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:00.317 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.556337+0000", 2023-11-05T08:34:00.317 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7376e-05 2023-11-05T08:34:00.317 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.317 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.317 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:00.318 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.564283+0000", 2023-11-05T08:34:00.318 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0079467340000000004 2023-11-05T08:34:00.318 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.318 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.319 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:00.319 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.564380+0000", 2023-11-05T08:34:00.319 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.7204e-05 2023-11-05T08:34:00.319 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.319 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:00.320 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.320 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.320 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.320 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:484 3.es0 3:71b68062:::smithi09039252-25:head [stat,delete] snapc 1c=[1a,7] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2023-11-05T08:34:00.320 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:18.555264+0000", 2023-11-05T08:34:00.320 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 41.704159849, 2023-11-05T08:34:00.321 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0088314239999999992, 2023-11-05T08:34:00.321 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:00.321 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:00.321 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:34:00.322 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:34:00.322 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:00.322 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 484 2023-11-05T08:34:00.322 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.322 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:00.323 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.323 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:00.323 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555264+0000", 2023-11-05T08:34:00.323 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.323 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.323 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.324 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:00.324 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555264+0000", 2023-11-05T08:34:00.324 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.324 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.324 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.325 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:00.325 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555267+0000", 2023-11-05T08:34:00.325 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2029999999999999e-06 2023-11-05T08:34:00.325 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.325 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.326 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:00.326 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555276+0000", 2023-11-05T08:34:00.326 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.8670000000000006e-06 2023-11-05T08:34:00.326 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.326 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.327 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:00.327 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555306+0000", 2023-11-05T08:34:00.327 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.9249000000000001e-05 2023-11-05T08:34:00.327 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.327 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.328 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:00.328 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555323+0000", 2023-11-05T08:34:00.328 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6898999999999999e-05 2023-11-05T08:34:00.328 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.328 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.329 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:00.329 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555436+0000", 2023-11-05T08:34:00.329 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00011365400000000001 2023-11-05T08:34:00.329 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.330 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.330 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:34:00.330 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555573+0000", 2023-11-05T08:34:00.330 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000136735 2023-11-05T08:34:00.331 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.331 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.331 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:00.331 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.556564+0000", 2023-11-05T08:34:00.332 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00099118899999999996 2023-11-05T08:34:00.332 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.332 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.332 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:00.333 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.563037+0000", 2023-11-05T08:34:00.333 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0064729319999999998 2023-11-05T08:34:00.333 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.333 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.334 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:34:00.334 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.563963+0000", 2023-11-05T08:34:00.334 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00092605599999999997 2023-11-05T08:34:00.334 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.335 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.335 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:00.335 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.564096+0000", 2023-11-05T08:34:00.335 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00013264 2023-11-05T08:34:00.336 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.337 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:00.337 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.338 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.338 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.338 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:505 3.es0 3:775e6266:::smithi09039252-12:head [stat,delete] snapc 21=[21,7] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2023-11-05T08:34:00.338 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:20.379990+0000", 2023-11-05T08:34:00.339 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 39.879434568000001, 2023-11-05T08:34:00.339 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.01062186, 2023-11-05T08:34:00.339 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:00.339 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:00.340 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:34:00.340 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:34:00.340 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:00.340 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 505 2023-11-05T08:34:00.341 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.341 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:00.341 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.341 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:00.342 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:20.379990+0000", 2023-11-05T08:34:00.342 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.342 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.343 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.343 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:00.343 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:20.379990+0000", 2023-11-05T08:34:00.343 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.343 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.344 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.344 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:00.344 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:20.379993+0000", 2023-11-05T08:34:00.344 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.817e-06 2023-11-05T08:34:00.344 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.345 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.345 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:00.345 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:20.380009+0000", 2023-11-05T08:34:00.345 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5512999999999999e-05 2023-11-05T08:34:00.345 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.346 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.346 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:00.346 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:20.380054+0000", 2023-11-05T08:34:00.346 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.4984999999999997e-05 2023-11-05T08:34:00.346 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.347 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.347 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:00.347 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:20.380078+0000", 2023-11-05T08:34:00.347 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3883000000000002e-05 2023-11-05T08:34:00.347 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.348 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.348 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:00.348 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:20.380312+0000", 2023-11-05T08:34:00.348 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00023429700000000001 2023-11-05T08:34:00.348 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.348 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.349 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op must wait for map", 2023-11-05T08:34:00.349 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:20.380330+0000", 2023-11-05T08:34:00.349 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.802e-05 2023-11-05T08:34:00.349 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.349 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.350 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:00.350 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:20.384598+0000", 2023-11-05T08:34:00.350 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0042682420000000002 2023-11-05T08:34:00.350 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.350 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.351 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:34:00.351 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:20.384765+0000", 2023-11-05T08:34:00.351 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000166591 2023-11-05T08:34:00.351 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.351 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.352 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:00.352 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:20.386117+0000", 2023-11-05T08:34:00.352 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001352036 2023-11-05T08:34:00.352 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.352 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.353 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:00.353 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:20.389679+0000", 2023-11-05T08:34:00.353 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.003562225 2023-11-05T08:34:00.353 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.353 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.354 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:34:00.354 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:20.390456+0000", 2023-11-05T08:34:00.354 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00077675499999999996 2023-11-05T08:34:00.354 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.354 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.355 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:00.355 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:20.390612+0000", 2023-11-05T08:34:00.355 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000155496 2023-11-05T08:34:00.355 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.355 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:00.355 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.356 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.356 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.356 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:653 3.es0 3:7a868416:::smithi09039252-27:head [append 0~0] snapc 38=[38,36,7] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2023-11-05T08:34:00.356 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:43.713833+0000", 2023-11-05T08:34:00.356 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 16.545591048999999, 2023-11-05T08:34:00.357 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.015387629999999999, 2023-11-05T08:34:00.357 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:00.357 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:00.357 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:34:00.357 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:34:00.358 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:00.358 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 653 2023-11-05T08:34:00.358 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.358 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:00.358 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.359 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:00.359 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.713833+0000", 2023-11-05T08:34:00.359 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.359 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.359 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.360 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:00.360 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.713833+0000", 2023-11-05T08:34:00.360 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.360 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.360 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.361 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:00.361 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.713835+0000", 2023-11-05T08:34:00.361 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3099999999999999e-06 2023-11-05T08:34:00.361 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.361 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.362 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:00.362 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.713844+0000", 2023-11-05T08:34:00.362 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.2840000000000006e-06 2023-11-05T08:34:00.362 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.362 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.363 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:00.363 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.713865+0000", 2023-11-05T08:34:00.363 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0764999999999999e-05 2023-11-05T08:34:00.363 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.363 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.364 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:00.364 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.713876+0000", 2023-11-05T08:34:00.364 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.171e-05 2023-11-05T08:34:00.365 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.365 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.365 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:00.365 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.714051+0000", 2023-11-05T08:34:00.365 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00017453400000000001 2023-11-05T08:34:00.366 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.366 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.366 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op must wait for map", 2023-11-05T08:34:00.366 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.714066+0000", 2023-11-05T08:34:00.367 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5038000000000001e-05 2023-11-05T08:34:00.367 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.367 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.367 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:00.368 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.719836+0000", 2023-11-05T08:34:00.368 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.005770318 2023-11-05T08:34:00.368 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.368 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.368 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:34:00.369 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.720016+0000", 2023-11-05T08:34:00.369 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000179416 2023-11-05T08:34:00.369 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.369 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.370 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:00.370 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.721239+0000", 2023-11-05T08:34:00.370 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001223385 2023-11-05T08:34:00.370 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.371 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.371 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:00.371 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.724695+0000", 2023-11-05T08:34:00.371 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0034562059999999999 2023-11-05T08:34:00.372 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.372 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.372 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:34:00.372 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.729113+0000", 2023-11-05T08:34:00.373 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0044178730000000001 2023-11-05T08:34:00.373 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.373 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.373 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:00.373 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.729221+0000", 2023-11-05T08:34:00.374 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000107791 2023-11-05T08:34:00.374 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.374 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:00.374 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.374 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.375 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.375 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:654 3.es0 3:7a868416:::smithi09039252-27:head [rollback 7] snapc 38=[38,36,7] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2023-11-05T08:34:00.375 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:43.713910+0000", 2023-11-05T08:34:00.375 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 16.545514111999999, 2023-11-05T08:34:00.376 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.016303631999999998, 2023-11-05T08:34:00.376 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:00.376 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:00.376 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:34:00.376 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:34:00.377 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:00.377 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 654 2023-11-05T08:34:00.377 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.377 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:00.378 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.378 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:00.378 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.713910+0000", 2023-11-05T08:34:00.378 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.379 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.379 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.379 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:00.379 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.713910+0000", 2023-11-05T08:34:00.380 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.380 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.380 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.380 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:00.381 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.713911+0000", 2023-11-05T08:34:00.381 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.5899999999999995e-07 2023-11-05T08:34:00.381 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.381 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.382 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:00.382 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.713915+0000", 2023-11-05T08:34:00.382 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.6500000000000002e-06 2023-11-05T08:34:00.382 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.382 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.383 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:00.383 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.713927+0000", 2023-11-05T08:34:00.383 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2714999999999999e-05 2023-11-05T08:34:00.383 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.383 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.383 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:00.384 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.713935+0000", 2023-11-05T08:34:00.384 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.011e-06 2023-11-05T08:34:00.384 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.384 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.384 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:00.385 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.714145+0000", 2023-11-05T08:34:00.385 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00020977300000000001 2023-11-05T08:34:00.385 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.385 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.386 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:34:00.386 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.714158+0000", 2023-11-05T08:34:00.386 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2699e-05 2023-11-05T08:34:00.386 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.386 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.387 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:00.387 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.721780+0000", 2023-11-05T08:34:00.387 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0076222490000000002 2023-11-05T08:34:00.387 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.387 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.387 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:34:00.388 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.721960+0000", 2023-11-05T08:34:00.388 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000180285 2023-11-05T08:34:00.388 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.388 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.388 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:00.389 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.722869+0000", 2023-11-05T08:34:00.389 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00090917099999999996 2023-11-05T08:34:00.389 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.389 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.389 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:00.390 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.727217+0000", 2023-11-05T08:34:00.390 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0043475199999999997 2023-11-05T08:34:00.390 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.390 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.390 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:34:00.391 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.730127+0000", 2023-11-05T08:34:00.391 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0029101090000000001 2023-11-05T08:34:00.391 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.391 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.391 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:00.392 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.730214+0000", 2023-11-05T08:34:00.392 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.6691000000000001e-05 2023-11-05T08:34:00.392 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.392 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:00.392 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.393 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.393 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.393 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:655 3.es0 3:7a868416:::smithi09039252-27:head [append 0~0] snapc 38=[38,36,7] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2023-11-05T08:34:00.393 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:43.713947+0000", 2023-11-05T08:34:00.393 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 16.545477007999999, 2023-11-05T08:34:00.393 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.017421432000000001, 2023-11-05T08:34:00.394 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:00.394 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:00.394 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:34:00.394 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:34:00.394 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:00.395 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 655 2023-11-05T08:34:00.395 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.395 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:00.395 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.395 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:00.396 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.713947+0000", 2023-11-05T08:34:00.396 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.396 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.396 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.396 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:00.397 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.713947+0000", 2023-11-05T08:34:00.397 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.397 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.397 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.397 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:00.398 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.713948+0000", 2023-11-05T08:34:00.398 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.3200000000000005e-07 2023-11-05T08:34:00.398 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.398 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.398 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:00.399 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.713951+0000", 2023-11-05T08:34:00.399 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.4379999999999999e-06 2023-11-05T08:34:00.399 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.399 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.399 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:00.400 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.713961+0000", 2023-11-05T08:34:00.400 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.628e-06 2023-11-05T08:34:00.400 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.400 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.400 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:00.401 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.713969+0000", 2023-11-05T08:34:00.401 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.6930000000000007e-06 2023-11-05T08:34:00.401 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.401 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.402 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:00.402 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.714196+0000", 2023-11-05T08:34:00.402 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000227495 2023-11-05T08:34:00.402 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.402 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.403 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:34:00.403 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.714208+0000", 2023-11-05T08:34:00.403 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.199e-05 2023-11-05T08:34:00.403 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.403 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.403 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:00.404 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.723324+0000", 2023-11-05T08:34:00.404 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0091161060000000006 2023-11-05T08:34:00.404 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.404 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.404 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:34:00.405 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.723495+0000", 2023-11-05T08:34:00.405 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00017114699999999999 2023-11-05T08:34:00.405 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.405 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.405 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:00.406 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.724221+0000", 2023-11-05T08:34:00.406 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00072564600000000004 2023-11-05T08:34:00.406 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.406 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.406 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:00.407 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.727840+0000", 2023-11-05T08:34:00.407 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0036185359999999999 2023-11-05T08:34:00.407 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.407 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.407 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:34:00.408 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.731276+0000", 2023-11-05T08:34:00.408 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0034361560000000001 2023-11-05T08:34:00.408 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.408 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.408 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:00.409 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:43.731369+0000", 2023-11-05T08:34:00.409 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.2965000000000002e-05 2023-11-05T08:34:00.409 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.409 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:00.409 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.410 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.410 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.410 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:671 3.es0 3:7a868416:::smithi09039252-27:head [append 0~802816] snapc 38=[7] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2023-11-05T08:34:00.410 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:45.783655+0000", 2023-11-05T08:34:00.410 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 14.475769712, 2023-11-05T08:34:00.411 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.011290691, 2023-11-05T08:34:00.411 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:00.411 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:00.411 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:34:00.411 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:34:00.412 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:00.412 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 671 2023-11-05T08:34:00.412 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.412 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:00.412 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.413 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:00.413 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.783655+0000", 2023-11-05T08:34:00.413 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.413 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.413 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.414 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:00.414 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.783655+0000", 2023-11-05T08:34:00.414 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.414 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.414 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.415 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:00.415 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.783657+0000", 2023-11-05T08:34:00.415 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2570000000000002e-06 2023-11-05T08:34:00.415 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.415 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.416 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:00.416 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.785069+0000", 2023-11-05T08:34:00.416 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001412553 2023-11-05T08:34:00.416 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.416 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.417 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:00.417 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.785092+0000", 2023-11-05T08:34:00.417 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2429000000000002e-05 2023-11-05T08:34:00.417 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.417 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.418 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:00.418 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.785105+0000", 2023-11-05T08:34:00.418 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3173000000000001e-05 2023-11-05T08:34:00.418 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.418 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.419 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:00.419 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.785308+0000", 2023-11-05T08:34:00.419 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00020346200000000001 2023-11-05T08:34:00.419 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.419 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.420 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:34:00.420 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.785451+0000", 2023-11-05T08:34:00.420 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00014265799999999999 2023-11-05T08:34:00.420 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.420 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.421 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:00.421 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.787077+0000", 2023-11-05T08:34:00.421 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0016254819999999999 2023-11-05T08:34:00.421 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.421 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.422 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:00.422 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.794363+0000", 2023-11-05T08:34:00.422 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0072867590000000003 2023-11-05T08:34:00.422 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.422 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.423 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:34:00.423 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.794817+0000", 2023-11-05T08:34:00.423 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00045376900000000003 2023-11-05T08:34:00.423 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.423 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.424 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:00.424 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.794945+0000", 2023-11-05T08:34:00.424 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00012814900000000001 2023-11-05T08:34:00.424 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.424 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:00.425 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.425 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.425 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.425 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:672 3.es0 3:7a868416:::smithi09039252-27:head [append 0~360448] snapc 38=[7] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2023-11-05T08:34:00.425 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:45.786907+0000", 2023-11-05T08:34:00.426 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 14.472517132, 2023-11-05T08:34:00.426 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.010018526, 2023-11-05T08:34:00.426 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:00.426 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:00.426 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:34:00.427 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:34:00.427 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:00.427 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 672 2023-11-05T08:34:00.427 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.427 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:00.428 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.428 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:00.428 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.786907+0000", 2023-11-05T08:34:00.428 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.428 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.428 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.429 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:00.429 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.786907+0000", 2023-11-05T08:34:00.429 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.429 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.429 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.430 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:00.430 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.786909+0000", 2023-11-05T08:34:00.430 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1299999999999999e-06 2023-11-05T08:34:00.430 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.430 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.431 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:00.431 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.787417+0000", 2023-11-05T08:34:00.431 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00050770500000000001 2023-11-05T08:34:00.431 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.431 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.432 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:00.432 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.787447+0000", 2023-11-05T08:34:00.432 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0195000000000001e-05 2023-11-05T08:34:00.432 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.432 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.433 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:00.433 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.787459+0000", 2023-11-05T08:34:00.433 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2366e-05 2023-11-05T08:34:00.433 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.433 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.434 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:00.434 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.789730+0000", 2023-11-05T08:34:00.434 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0022701549999999998 2023-11-05T08:34:00.434 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.434 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.435 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:34:00.435 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.789919+0000", 2023-11-05T08:34:00.435 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00018969599999999999 2023-11-05T08:34:00.435 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.435 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.436 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:00.436 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.791021+0000", 2023-11-05T08:34:00.436 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001101278 2023-11-05T08:34:00.436 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.436 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.437 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:00.437 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.794537+0000", 2023-11-05T08:34:00.437 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0035161369999999999 2023-11-05T08:34:00.437 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.437 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.438 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:34:00.438 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.796803+0000", 2023-11-05T08:34:00.438 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.002266395 2023-11-05T08:34:00.438 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.438 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.439 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:00.439 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.796926+0000", 2023-11-05T08:34:00.439 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000122469 2023-11-05T08:34:00.439 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.439 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:00.440 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.440 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.440 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.440 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:673 3.es0 3:7a868416:::smithi09039252-27:head [setxattr _header (57) in=7b] snapc 38=[7] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2023-11-05T08:34:00.440 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:45.787506+0000", 2023-11-05T08:34:00.441 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 14.471918514, 2023-11-05T08:34:00.441 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0097367189999999996, 2023-11-05T08:34:00.441 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:00.441 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:00.441 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:34:00.442 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:34:00.442 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:00.442 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 673 2023-11-05T08:34:00.442 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.442 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:00.443 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.443 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:00.443 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.787506+0000", 2023-11-05T08:34:00.443 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.443 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.444 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.444 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:00.444 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.787506+0000", 2023-11-05T08:34:00.444 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.444 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.444 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.445 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:00.445 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.787507+0000", 2023-11-05T08:34:00.445 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1060000000000001e-06 2023-11-05T08:34:00.445 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.445 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.446 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:00.446 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.787512+0000", 2023-11-05T08:34:00.446 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.4770000000000002e-06 2023-11-05T08:34:00.446 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.446 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.447 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:00.447 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.787531+0000", 2023-11-05T08:34:00.447 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9052e-05 2023-11-05T08:34:00.447 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.447 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.448 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:00.448 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.787541+0000", 2023-11-05T08:34:00.448 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.6229999999999992e-06 2023-11-05T08:34:00.448 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.448 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.449 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:00.449 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.792584+0000", 2023-11-05T08:34:00.449 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0050430340000000001 2023-11-05T08:34:00.449 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.449 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.450 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:34:00.450 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.792730+0000", 2023-11-05T08:34:00.450 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00014642100000000001 2023-11-05T08:34:00.450 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.450 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.451 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:00.451 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.793793+0000", 2023-11-05T08:34:00.451 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001062863 2023-11-05T08:34:00.451 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.451 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.452 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:00.452 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.795678+0000", 2023-11-05T08:34:00.452 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001884262 2023-11-05T08:34:00.452 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.452 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.453 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:34:00.453 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.797100+0000", 2023-11-05T08:34:00.453 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0014225399999999999 2023-11-05T08:34:00.453 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.453 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.454 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:00.454 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.797242+0000", 2023-11-05T08:34:00.454 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00014234100000000001 2023-11-05T08:34:00.454 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.454 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:00.455 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.455 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.455 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.455 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:674 3.es0 3:7a868416:::smithi09039252-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e92)", 2023-11-05T08:34:00.455 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:45.787554+0000", 2023-11-05T08:34:00.456 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 14.471870483, 2023-11-05T08:34:00.456 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.012409609, 2023-11-05T08:34:00.456 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:00.456 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2023-11-05T08:34:00.456 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:34:00.457 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:34:00.457 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:00.457 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 674 2023-11-05T08:34:00.457 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.457 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:00.457 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.458 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:00.458 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.787554+0000", 2023-11-05T08:34:00.458 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.458 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.458 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.459 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:00.459 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.787554+0000", 2023-11-05T08:34:00.459 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.459 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.460 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.460 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:00.460 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.787554+0000", 2023-11-05T08:34:00.460 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.4000000000000001e-07 2023-11-05T08:34:00.460 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.460 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.461 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:00.461 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.787558+0000", 2023-11-05T08:34:00.461 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.18e-06 2023-11-05T08:34:00.461 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.461 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.462 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:00.462 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.787568+0000", 2023-11-05T08:34:00.462 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0596e-05 2023-11-05T08:34:00.462 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.462 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.463 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:00.463 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.787576+0000", 2023-11-05T08:34:00.463 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.9070000000000007e-06 2023-11-05T08:34:00.463 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.463 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.464 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:00.464 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.795833+0000", 2023-11-05T08:34:00.464 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0082570679999999994 2023-11-05T08:34:00.464 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.464 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.465 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for rw locks", 2023-11-05T08:34:00.465 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.796079+0000", 2023-11-05T08:34:00.465 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00024616800000000001 2023-11-05T08:34:00.465 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.465 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.466 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:00.466 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.798003+0000", 2023-11-05T08:34:00.466 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0019239109999999999 2023-11-05T08:34:00.466 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.466 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.467 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:34:00.467 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.798131+0000", 2023-11-05T08:34:00.467 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00012796800000000001 2023-11-05T08:34:00.467 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.467 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.468 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:00.468 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.799963+0000", 2023-11-05T08:34:00.468 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001832071 2023-11-05T08:34:00.468 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.468 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:00.469 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.469 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.469 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.469 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.18s2 97/45 ECSubWrite(tid=317, reqid=osd.13.0:317, at_version=97'42, trim_to=0'0, roll_forward_to=68'40))", 2023-11-05T08:34:00.469 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.665570+0000", 2023-11-05T08:34:00.470 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 9.5938540079999992, 2023-11-05T08:34:00.470 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.011413658, 2023-11-05T08:34:00.470 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:00.470 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:00.470 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:00.470 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.471 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:00.471 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.665570+0000", 2023-11-05T08:34:00.471 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.471 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.471 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.472 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:00.472 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.665570+0000", 2023-11-05T08:34:00.472 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.472 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.472 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.473 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:00.473 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.665572+0000", 2023-11-05T08:34:00.473 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6759999999999999e-06 2023-11-05T08:34:00.473 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.473 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.474 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:00.474 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.665634+0000", 2023-11-05T08:34:00.474 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.2162000000000006e-05 2023-11-05T08:34:00.474 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.474 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.475 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:00.475 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.665666+0000", 2023-11-05T08:34:00.475 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2255000000000002e-05 2023-11-05T08:34:00.475 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.475 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.475 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:00.476 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.665689+0000", 2023-11-05T08:34:00.476 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2949999999999999e-05 2023-11-05T08:34:00.476 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.476 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.476 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:00.477 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.665836+0000", 2023-11-05T08:34:00.477 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00014675800000000001 2023-11-05T08:34:00.477 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.477 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.477 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:00.478 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.665861+0000", 2023-11-05T08:34:00.478 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5021999999999999e-05 2023-11-05T08:34:00.478 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.478 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.478 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:00.479 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.676900+0000", 2023-11-05T08:34:00.479 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.011038918 2023-11-05T08:34:00.479 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.479 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.479 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:00.479 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.676984+0000", 2023-11-05T08:34:00.480 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.3917000000000003e-05 2023-11-05T08:34:00.480 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.480 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:00.480 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.480 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.481 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.481 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.18s2 97/45 ECSubWrite(tid=319, reqid=osd.13.0:319, at_version=97'44, trim_to=0'0, roll_forward_to=68'40))", 2023-11-05T08:34:00.481 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.666108+0000", 2023-11-05T08:34:00.481 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 9.5933159729999993, 2023-11-05T08:34:00.481 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.018622788000000001, 2023-11-05T08:34:00.482 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:00.482 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:00.482 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:00.482 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.482 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:00.483 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.666108+0000", 2023-11-05T08:34:00.483 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.483 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.483 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.483 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:00.483 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.666108+0000", 2023-11-05T08:34:00.484 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.484 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.484 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.484 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:00.484 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.666110+0000", 2023-11-05T08:34:00.485 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.7570000000000001e-06 2023-11-05T08:34:00.485 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.485 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.485 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:00.485 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.666167+0000", 2023-11-05T08:34:00.486 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.7383999999999998e-05 2023-11-05T08:34:00.486 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.486 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.486 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:00.486 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.666198+0000", 2023-11-05T08:34:00.487 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0966000000000001e-05 2023-11-05T08:34:00.487 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.487 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.487 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:00.487 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.666222+0000", 2023-11-05T08:34:00.488 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3405999999999999e-05 2023-11-05T08:34:00.488 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.488 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.488 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:00.488 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.671801+0000", 2023-11-05T08:34:00.489 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.005579171 2023-11-05T08:34:00.489 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.489 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.489 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:00.489 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.671826+0000", 2023-11-05T08:34:00.490 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5378999999999999e-05 2023-11-05T08:34:00.490 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.490 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.490 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:00.490 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.684639+0000", 2023-11-05T08:34:00.491 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.012812472 2023-11-05T08:34:00.491 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.491 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.491 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:00.491 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.684731+0000", 2023-11-05T08:34:00.491 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.2252999999999999e-05 2023-11-05T08:34:00.492 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.492 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:00.492 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.492 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.492 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.493 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.13s4 97/45 ECSubWrite(tid=168, reqid=osd.11.0:168, at_version=97'25, trim_to=0'0, roll_forward_to=92'23))", 2023-11-05T08:34:00.493 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.675380+0000", 2023-11-05T08:34:00.493 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 9.5840441619999996, 2023-11-05T08:34:00.493 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0093184380000000001, 2023-11-05T08:34:00.493 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:00.494 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:00.494 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:00.494 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.494 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:00.494 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.675380+0000", 2023-11-05T08:34:00.495 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.495 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.495 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.495 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:00.495 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.675380+0000", 2023-11-05T08:34:00.495 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.496 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.496 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.496 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:00.496 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.675382+0000", 2023-11-05T08:34:00.496 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9130000000000001e-06 2023-11-05T08:34:00.497 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.497 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.497 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:00.497 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.675427+0000", 2023-11-05T08:34:00.497 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.5364999999999997e-05 2023-11-05T08:34:00.498 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.498 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.498 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:00.498 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.675453+0000", 2023-11-05T08:34:00.499 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5822000000000001e-05 2023-11-05T08:34:00.499 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.499 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.499 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:00.499 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.675496+0000", 2023-11-05T08:34:00.500 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.2472000000000003e-05 2023-11-05T08:34:00.500 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.500 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.500 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:00.500 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.675719+0000", 2023-11-05T08:34:00.500 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000223043 2023-11-05T08:34:00.501 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.501 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.501 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:00.501 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.675748+0000", 2023-11-05T08:34:00.501 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.9281999999999998e-05 2023-11-05T08:34:00.502 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.502 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.502 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:00.502 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.684609+0000", 2023-11-05T08:34:00.502 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0088614579999999991 2023-11-05T08:34:00.503 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.503 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.503 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:00.503 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.684699+0000", 2023-11-05T08:34:00.503 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.9083000000000001e-05 2023-11-05T08:34:00.504 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.504 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:00.504 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.504 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.504 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.505 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.1s5 97/40 ECSubWrite(tid=141, reqid=osd.4.0:140, at_version=0'0, trim_to=0'0, roll_forward_to=97'28))", 2023-11-05T08:34:00.505 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.687027+0000", 2023-11-05T08:34:00.505 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 9.5723971649999999, 2023-11-05T08:34:00.505 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.010586126, 2023-11-05T08:34:00.505 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:00.506 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:00.506 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:00.506 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.506 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:00.506 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.687027+0000", 2023-11-05T08:34:00.506 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.507 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.507 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.507 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:00.507 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.687027+0000", 2023-11-05T08:34:00.507 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.508 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.508 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.508 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:00.508 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.687029+0000", 2023-11-05T08:34:00.508 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.7659999999999999e-06 2023-11-05T08:34:00.508 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.509 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.509 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:00.509 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.687056+0000", 2023-11-05T08:34:00.509 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7189e-05 2023-11-05T08:34:00.509 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.510 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.510 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:00.510 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.687104+0000", 2023-11-05T08:34:00.510 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.8134999999999999e-05 2023-11-05T08:34:00.510 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.511 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.511 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:00.511 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.687127+0000", 2023-11-05T08:34:00.511 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3227e-05 2023-11-05T08:34:00.511 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.512 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.512 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:00.512 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.687284+0000", 2023-11-05T08:34:00.512 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00015671700000000001 2023-11-05T08:34:00.512 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.512 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.513 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:00.513 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.687309+0000", 2023-11-05T08:34:00.513 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4680000000000001e-05 2023-11-05T08:34:00.513 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.513 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.514 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:00.514 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.697522+0000", 2023-11-05T08:34:00.514 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.010213172 2023-11-05T08:34:00.514 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.514 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.515 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:00.515 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.697613+0000", 2023-11-05T08:34:00.515 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.1240000000000001e-05 2023-11-05T08:34:00.515 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.515 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:00.516 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.516 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.516 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.516 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.fs1 101/37 ECSubWrite(tid=175, reqid=client.4691.0:778, at_version=101'62, trim_to=0'0, roll_forward_to=101'59))", 2023-11-05T08:34:00.517 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:54.775260+0000", 2023-11-05T08:34:00.517 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 5.4841639000000004, 2023-11-05T08:34:00.517 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.011084057, 2023-11-05T08:34:00.517 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:00.517 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:00.518 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:00.518 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.518 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:00.518 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.775260+0000", 2023-11-05T08:34:00.518 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.519 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.519 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.519 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:00.519 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.775260+0000", 2023-11-05T08:34:00.519 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.520 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.520 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.520 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:00.520 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.775262+0000", 2023-11-05T08:34:00.520 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.717e-06 2023-11-05T08:34:00.520 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.521 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.521 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:00.521 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.775490+0000", 2023-11-05T08:34:00.521 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00022796900000000001 2023-11-05T08:34:00.521 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.522 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.522 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:00.522 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.775523+0000", 2023-11-05T08:34:00.522 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2489e-05 2023-11-05T08:34:00.522 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.523 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.523 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:00.523 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.775545+0000", 2023-11-05T08:34:00.523 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2887e-05 2023-11-05T08:34:00.523 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.524 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.524 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:00.524 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.775722+0000", 2023-11-05T08:34:00.524 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00017708700000000001 2023-11-05T08:34:00.524 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.525 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.525 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:00.525 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.775747+0000", 2023-11-05T08:34:00.525 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4905999999999998e-05 2023-11-05T08:34:00.525 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.526 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.526 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:00.526 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.786308+0000", 2023-11-05T08:34:00.526 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.010560216000000001 2023-11-05T08:34:00.526 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.527 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.527 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:00.527 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.786344+0000", 2023-11-05T08:34:00.527 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.6786000000000003e-05 2023-11-05T08:34:00.527 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.528 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:00.528 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.528 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.528 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.528 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.fs1 101/37 ECSubWrite(tid=176, reqid=client.4691.0:779, at_version=101'63, trim_to=0'0, roll_forward_to=101'59))", 2023-11-05T08:34:00.528 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:54.776508+0000", 2023-11-05T08:34:00.529 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 5.4829162650000001, 2023-11-05T08:34:00.529 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0098766340000000005, 2023-11-05T08:34:00.529 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:00.529 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:00.529 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:00.530 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.530 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:00.530 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.776508+0000", 2023-11-05T08:34:00.530 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.530 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.531 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.531 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:00.531 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.776508+0000", 2023-11-05T08:34:00.531 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.531 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.532 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.532 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:00.532 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.776510+0000", 2023-11-05T08:34:00.532 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.7239999999999999e-06 2023-11-05T08:34:00.532 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.533 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.533 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:00.533 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.776564+0000", 2023-11-05T08:34:00.533 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.4608000000000003e-05 2023-11-05T08:34:00.533 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.534 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.534 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:00.534 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.776596+0000", 2023-11-05T08:34:00.534 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2101999999999999e-05 2023-11-05T08:34:00.535 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.535 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.535 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:00.535 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.776620+0000", 2023-11-05T08:34:00.536 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3153e-05 2023-11-05T08:34:00.536 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.536 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.536 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:00.536 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.777445+0000", 2023-11-05T08:34:00.537 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00082527800000000004 2023-11-05T08:34:00.537 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.537 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.537 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:00.538 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.777479+0000", 2023-11-05T08:34:00.538 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.3935e-05 2023-11-05T08:34:00.538 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.538 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.538 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:00.538 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.786351+0000", 2023-11-05T08:34:00.539 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0088727179999999999 2023-11-05T08:34:00.539 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.539 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.539 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:00.539 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.786385+0000", 2023-11-05T08:34:00.540 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.3115999999999999e-05 2023-11-05T08:34:00.540 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.540 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:00.540 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.540 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.541 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.541 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:805 3.es0 3:7a868416:::smithi09039252-27:head [rollback 44] snapc 44=[44,42,41] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2023-11-05T08:34:00.541 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:55.747894+0000", 2023-11-05T08:34:00.541 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 4.5115299579999997, 2023-11-05T08:34:00.541 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.008661485, 2023-11-05T08:34:00.542 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:00.542 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2023-11-05T08:34:00.542 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:34:00.542 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:34:00.542 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:00.543 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 805 2023-11-05T08:34:00.543 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.543 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:00.543 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.543 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:00.544 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:55.747894+0000", 2023-11-05T08:34:00.544 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.544 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.544 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.544 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:00.544 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:55.747894+0000", 2023-11-05T08:34:00.545 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.545 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.545 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.545 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:00.545 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:55.747897+0000", 2023-11-05T08:34:00.546 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2100000000000002e-06 2023-11-05T08:34:00.546 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.546 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.546 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:00.546 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:55.747910+0000", 2023-11-05T08:34:00.547 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2342e-05 2023-11-05T08:34:00.547 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.547 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.547 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:00.547 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:55.747945+0000", 2023-11-05T08:34:00.548 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.5006000000000003e-05 2023-11-05T08:34:00.548 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.548 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.548 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:00.548 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:55.747962+0000", 2023-11-05T08:34:00.549 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.7249999999999999e-05 2023-11-05T08:34:00.549 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.549 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.549 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:00.549 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:55.748131+0000", 2023-11-05T08:34:00.550 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000169328 2023-11-05T08:34:00.550 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.550 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.550 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op must wait for map", 2023-11-05T08:34:00.550 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:55.748143+0000", 2023-11-05T08:34:00.551 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1686e-05 2023-11-05T08:34:00.551 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.551 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.551 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:00.551 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:55.756044+0000", 2023-11-05T08:34:00.552 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0079006059999999993 2023-11-05T08:34:00.552 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.552 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.552 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:34:00.552 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:55.756260+0000", 2023-11-05T08:34:00.553 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000216116 2023-11-05T08:34:00.553 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.553 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.553 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:00.553 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:55.756556+0000", 2023-11-05T08:34:00.554 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00029594099999999998 2023-11-05T08:34:00.554 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.554 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:00.554 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.554 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.555 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:00.555 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:34:00.556 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-11-05T08:34:00.557 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:34:00.554+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:34:00.657 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:34:00.654+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:34:00.687 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:34:00.687 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2023-11-05T08:34:00.688 INFO:teuthology.orchestra.run.smithi069.stdout: "num_ops": 0 2023-11-05T08:34:00.688 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:34:00.701 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-11-05T08:34:00.758 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:34:00.754+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:34:00.794 INFO:teuthology.orchestra.run.smithi002.stderr:reweighted osd.13 to 1 (10000) 2023-11-05T08:34:00.804 INFO:tasks.rados.rados.0.smithi090.stdout:362: done (2 left) 2023-11-05T08:34:00.804 INFO:tasks.rados.rados.0.smithi090.stdout:363: done (1 left) 2023-11-05T08:34:00.804 INFO:tasks.rados.rados.0.smithi090.stdout:366: done (0 left) 2023-11-05T08:34:00.804 INFO:tasks.rados.rados.0.smithi090.stdout:367: copy_from oid 19 from oid 38 current snap is 38 2023-11-05T08:34:00.805 INFO:tasks.rados.rados.0.smithi090.stdout:368: snap_remove snap 37 2023-11-05T08:34:00.820 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 1 2023-11-05T08:34:00.823 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:34:00.824 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2023-11-05T08:34:00.824 INFO:teuthology.orchestra.run.smithi069.stdout: "complaint_time": 30, 2023-11-05T08:34:00.824 INFO:teuthology.orchestra.run.smithi069.stdout: "num_blocked_ops": 0 2023-11-05T08:34:00.824 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:34:00.838 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-11-05T08:34:00.859 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:34:00.854+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:34:00.959 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:34:00.952+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:34:00.965 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:34:00.965 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 20, 2023-11-05T08:34:00.966 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 600, 2023-11-05T08:34:00.966 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [ 2023-11-05T08:34:00.966 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.966 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.as5 42/33 ECSubWrite(tid=38, reqid=osd.7.0:38, at_version=42'16, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:34:00.966 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:55.127656+0000", 2023-11-05T08:34:00.967 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 65.833788447000003, 2023-11-05T08:34:00.967 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.016413512000000002, 2023-11-05T08:34:00.967 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:00.967 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:00.967 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:00.968 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.968 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:00.968 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127656+0000", 2023-11-05T08:34:00.968 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.968 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.969 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.969 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:00.969 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127656+0000", 2023-11-05T08:34:00.969 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.969 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.970 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.970 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:00.970 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127656+0000", 2023-11-05T08:34:00.970 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.2699999999999999e-07 2023-11-05T08:34:00.970 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.971 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.971 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:00.971 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127732+0000", 2023-11-05T08:34:00.971 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.5978999999999999e-05 2023-11-05T08:34:00.971 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.972 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.972 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:00.972 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127763+0000", 2023-11-05T08:34:00.972 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0672e-05 2023-11-05T08:34:00.972 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.972 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.973 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:00.973 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127785+0000", 2023-11-05T08:34:00.973 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2059e-05 2023-11-05T08:34:00.973 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.973 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.974 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:00.974 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127975+0000", 2023-11-05T08:34:00.974 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000189946 2023-11-05T08:34:00.974 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.974 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.975 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:00.975 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127999+0000", 2023-11-05T08:34:00.975 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3555999999999999e-05 2023-11-05T08:34:00.975 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.975 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.976 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:00.976 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.143967+0000", 2023-11-05T08:34:00.976 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.015968781000000001 2023-11-05T08:34:00.976 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.976 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.977 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:00.977 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.144069+0000", 2023-11-05T08:34:00.977 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000101992 2023-11-05T08:34:00.977 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.977 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:00.978 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.978 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.978 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.978 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.as5 42/33 ECSubWrite(tid=39, reqid=osd.7.0:39, at_version=42'18, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:34:00.978 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:55.127880+0000", 2023-11-05T08:34:00.979 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 65.833563961999999, 2023-11-05T08:34:00.979 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.019355170000000001, 2023-11-05T08:34:00.979 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:00.979 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:00.979 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:00.979 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.980 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:00.980 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127880+0000", 2023-11-05T08:34:00.980 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.980 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.980 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.981 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:00.981 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127880+0000", 2023-11-05T08:34:00.981 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.981 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.981 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.981 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:00.982 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127882+0000", 2023-11-05T08:34:00.982 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.187e-06 2023-11-05T08:34:00.982 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.982 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.982 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:00.983 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127934+0000", 2023-11-05T08:34:00.983 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.2553000000000003e-05 2023-11-05T08:34:00.983 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.983 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.983 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:00.984 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127961+0000", 2023-11-05T08:34:00.984 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6999e-05 2023-11-05T08:34:00.984 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.984 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.984 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:00.984 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127983+0000", 2023-11-05T08:34:00.985 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1778999999999999e-05 2023-11-05T08:34:00.985 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.985 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.985 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:00.985 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.129155+0000", 2023-11-05T08:34:00.986 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001172022 2023-11-05T08:34:00.986 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.986 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.986 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:00.986 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.129180+0000", 2023-11-05T08:34:00.987 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4841000000000002e-05 2023-11-05T08:34:00.987 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.987 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.987 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:00.987 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.147146+0000", 2023-11-05T08:34:00.988 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.017965990000000001 2023-11-05T08:34:00.988 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.988 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.988 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:00.988 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.147236+0000", 2023-11-05T08:34:00.989 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.9798999999999999e-05 2023-11-05T08:34:00.989 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.989 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:00.989 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:00.989 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.990 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.990 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.es5 42/30 ECSubWrite(tid=43, reqid=osd.2.0:43, at_version=42'29, trim_to=0'0, roll_forward_to=41'27))", 2023-11-05T08:34:00.990 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:55.134988+0000", 2023-11-05T08:34:00.990 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 65.826456145999998, 2023-11-05T08:34:00.990 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.012343942, 2023-11-05T08:34:00.990 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:00.991 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:00.991 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:00.991 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.991 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:00.991 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.134988+0000", 2023-11-05T08:34:00.992 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.992 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.992 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.992 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:00.992 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.134988+0000", 2023-11-05T08:34:00.993 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:00.993 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.993 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.993 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:00.993 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.134989+0000", 2023-11-05T08:34:00.994 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.6000000000000003e-07 2023-11-05T08:34:00.994 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.994 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.994 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:00.994 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.135039+0000", 2023-11-05T08:34:00.995 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.9879999999999997e-05 2023-11-05T08:34:00.995 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.995 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.995 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:00.995 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.135069+0000", 2023-11-05T08:34:00.995 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0445000000000001e-05 2023-11-05T08:34:00.996 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.996 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.996 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:00.996 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.135094+0000", 2023-11-05T08:34:00.996 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4646000000000001e-05 2023-11-05T08:34:00.997 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.997 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.997 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:00.997 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.135758+0000", 2023-11-05T08:34:00.997 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00066366799999999996 2023-11-05T08:34:00.998 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.998 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.998 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:00.998 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.135780+0000", 2023-11-05T08:34:00.998 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2509000000000001e-05 2023-11-05T08:34:00.999 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:00.999 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:00.999 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:00.999 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.147237+0000", 2023-11-05T08:34:00.999 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.011456401999999999 2023-11-05T08:34:00.999 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.000 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.000 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:01.000 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.147332+0000", 2023-11-05T08:34:01.000 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.5631999999999995e-05 2023-11-05T08:34:01.000 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:01.001 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:01.001 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:01.001 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.001 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.001 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.bs4 66/37 ECSubWrite(tid=108, reqid=client.4691.0:496, at_version=66'23, trim_to=0'0, roll_forward_to=65'21))", 2023-11-05T08:34:01.002 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:19.399298+0000", 2023-11-05T08:34:01.002 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 41.562145979, 2023-11-05T08:34:01.002 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.012016639000000001, 2023-11-05T08:34:01.002 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:01.002 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:01.002 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:01.003 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.003 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:01.003 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.399298+0000", 2023-11-05T08:34:01.003 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:01.003 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.004 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.004 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:01.004 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.399298+0000", 2023-11-05T08:34:01.004 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:01.004 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.004 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.005 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:01.005 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.399299+0000", 2023-11-05T08:34:01.005 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.4399999999999999e-07 2023-11-05T08:34:01.005 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.005 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.006 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:01.006 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.399993+0000", 2023-11-05T08:34:01.006 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00069384999999999998 2023-11-05T08:34:01.006 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.006 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.007 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:01.007 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.400030+0000", 2023-11-05T08:34:01.007 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.6508999999999999e-05 2023-11-05T08:34:01.007 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.007 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.007 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:01.008 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.400046+0000", 2023-11-05T08:34:01.008 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6266e-05 2023-11-05T08:34:01.008 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.008 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.008 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:01.009 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.400213+0000", 2023-11-05T08:34:01.009 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00016678099999999999 2023-11-05T08:34:01.009 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.009 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.009 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:01.010 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.400240+0000", 2023-11-05T08:34:01.010 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6979999999999999e-05 2023-11-05T08:34:01.010 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.010 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.010 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:01.010 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.411240+0000", 2023-11-05T08:34:01.011 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.010999978000000001 2023-11-05T08:34:01.011 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.011 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.011 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:01.011 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.411315+0000", 2023-11-05T08:34:01.012 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.5531000000000006e-05 2023-11-05T08:34:01.012 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:01.012 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:01.012 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:01.012 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.012 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.013 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.18s1 97/45 ECSubWrite(tid=317, reqid=osd.13.0:317, at_version=97'42, trim_to=0'0, roll_forward_to=68'40))", 2023-11-05T08:34:01.013 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.668427+0000", 2023-11-05T08:34:01.013 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 10.293017762, 2023-11-05T08:34:01.013 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.014768801, 2023-11-05T08:34:01.013 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:01.014 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:01.014 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:01.014 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.014 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:01.014 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668427+0000", 2023-11-05T08:34:01.015 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:01.015 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.015 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.015 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:01.015 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668427+0000", 2023-11-05T08:34:01.016 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:01.016 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.016 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.016 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:01.016 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668429+0000", 2023-11-05T08:34:01.016 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.916e-06 2023-11-05T08:34:01.017 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.017 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.017 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:01.017 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668490+0000", 2023-11-05T08:34:01.017 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.0977999999999998e-05 2023-11-05T08:34:01.018 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.018 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.018 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:01.018 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668527+0000", 2023-11-05T08:34:01.018 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.7905999999999999e-05 2023-11-05T08:34:01.019 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.019 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.019 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:01.019 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668556+0000", 2023-11-05T08:34:01.019 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8605000000000001e-05 2023-11-05T08:34:01.020 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.020 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.020 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:01.020 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668745+0000", 2023-11-05T08:34:01.020 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00018850099999999999 2023-11-05T08:34:01.020 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.021 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.021 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:01.021 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668776+0000", 2023-11-05T08:34:01.021 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.1134999999999999e-05 2023-11-05T08:34:01.021 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.022 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.022 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:01.022 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.683106+0000", 2023-11-05T08:34:01.022 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.01433067 2023-11-05T08:34:01.022 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.023 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.023 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:01.023 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.683195+0000", 2023-11-05T08:34:01.023 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.9090000000000003e-05 2023-11-05T08:34:01.023 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:01.024 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:01.024 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:01.024 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.024 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.024 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.10s1 97/45 ECSubWrite(tid=318, reqid=osd.13.0:318, at_version=97'58, trim_to=0'0, roll_forward_to=93'56))", 2023-11-05T08:34:01.024 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.668619+0000", 2023-11-05T08:34:01.025 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 10.292825608999999, 2023-11-05T08:34:01.025 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.014652379, 2023-11-05T08:34:01.025 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:01.025 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:01.025 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:01.026 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.026 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:01.026 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668619+0000", 2023-11-05T08:34:01.026 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:01.026 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.026 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.027 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:01.027 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668619+0000", 2023-11-05T08:34:01.027 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:01.027 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.027 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.028 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:01.028 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668620+0000", 2023-11-05T08:34:01.028 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.017e-06 2023-11-05T08:34:01.028 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.028 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.029 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:01.029 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668710+0000", 2023-11-05T08:34:01.029 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.0587000000000005e-05 2023-11-05T08:34:01.029 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.029 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.029 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:01.030 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668736+0000", 2023-11-05T08:34:01.030 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6098999999999999e-05 2023-11-05T08:34:01.030 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.030 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.030 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:01.031 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668756+0000", 2023-11-05T08:34:01.031 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9599e-05 2023-11-05T08:34:01.031 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.031 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.031 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:01.031 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.669838+0000", 2023-11-05T08:34:01.032 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0010821240000000001 2023-11-05T08:34:01.032 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.032 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.032 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:01.032 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.669863+0000", 2023-11-05T08:34:01.033 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4496000000000001e-05 2023-11-05T08:34:01.033 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.033 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.033 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:01.033 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.683214+0000", 2023-11-05T08:34:01.034 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.013350879 2023-11-05T08:34:01.034 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.034 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.034 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:01.034 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.683271+0000", 2023-11-05T08:34:01.034 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.7578000000000003e-05 2023-11-05T08:34:01.035 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:01.035 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:01.035 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:01.035 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.035 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.036 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.10s5 97/45 ECSubWrite(tid=318, reqid=osd.13.0:318, at_version=97'58, trim_to=0'0, roll_forward_to=93'56))", 2023-11-05T08:34:01.036 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.668786+0000", 2023-11-05T08:34:01.036 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 10.292658447999999, 2023-11-05T08:34:01.036 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.014548782, 2023-11-05T08:34:01.036 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:01.036 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:01.037 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:01.037 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.037 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:01.037 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668786+0000", 2023-11-05T08:34:01.037 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:01.038 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.038 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.038 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:01.038 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668786+0000", 2023-11-05T08:34:01.038 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:01.039 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.039 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.039 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:01.039 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668787+0000", 2023-11-05T08:34:01.039 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.4e-07 2023-11-05T08:34:01.040 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.040 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.040 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:01.040 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668844+0000", 2023-11-05T08:34:01.040 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.7680999999999999e-05 2023-11-05T08:34:01.040 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.041 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.041 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:01.041 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668869+0000", 2023-11-05T08:34:01.041 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.41e-05 2023-11-05T08:34:01.041 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.042 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.042 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:01.042 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668891+0000", 2023-11-05T08:34:01.042 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2259000000000002e-05 2023-11-05T08:34:01.042 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.043 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.043 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:01.043 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.670415+0000", 2023-11-05T08:34:01.043 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0015243019999999999 2023-11-05T08:34:01.043 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.044 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.044 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:01.044 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.670492+0000", 2023-11-05T08:34:01.044 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.6576999999999995e-05 2023-11-05T08:34:01.044 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.044 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.045 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:01.045 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.683277+0000", 2023-11-05T08:34:01.045 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.012785179000000001 2023-11-05T08:34:01.045 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.045 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.046 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:01.046 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.683335+0000", 2023-11-05T08:34:01.046 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.7843999999999997e-05 2023-11-05T08:34:01.046 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:01.046 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:01.047 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:01.047 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.047 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.047 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.10s1 97/45 ECSubWrite(tid=320, reqid=osd.13.0:320, at_version=97'60, trim_to=0'0, roll_forward_to=93'56))", 2023-11-05T08:34:01.047 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.668922+0000", 2023-11-05T08:34:01.048 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 10.292522339, 2023-11-05T08:34:01.048 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.015049735999999999, 2023-11-05T08:34:01.048 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:01.048 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:01.048 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:01.049 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.049 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:01.049 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668922+0000", 2023-11-05T08:34:01.049 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:01.049 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.050 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.050 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:01.050 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668922+0000", 2023-11-05T08:34:01.050 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:01.050 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.050 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.051 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:01.051 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668923+0000", 2023-11-05T08:34:01.051 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.6300000000000005e-07 2023-11-05T08:34:01.051 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.051 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.052 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:01.052 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668964+0000", 2023-11-05T08:34:01.052 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.0878999999999999e-05 2023-11-05T08:34:01.052 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.052 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.053 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:01.053 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668986+0000", 2023-11-05T08:34:01.053 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2166000000000001e-05 2023-11-05T08:34:01.053 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.053 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.054 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:01.054 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.669003+0000", 2023-11-05T08:34:01.054 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6892999999999999e-05 2023-11-05T08:34:01.054 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.054 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.055 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:01.055 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.673668+0000", 2023-11-05T08:34:01.055 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0046656019999999996 2023-11-05T08:34:01.055 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.055 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.055 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:01.056 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.673693+0000", 2023-11-05T08:34:01.056 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4816000000000001e-05 2023-11-05T08:34:01.056 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.056 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.056 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:01.057 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.683900+0000", 2023-11-05T08:34:01.057 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.010206814999999999 2023-11-05T08:34:01.057 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.057 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.057 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:01.058 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.683972+0000", 2023-11-05T08:34:01.058 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.1902000000000004e-05 2023-11-05T08:34:01.058 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:01.058 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:01.058 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:01.059 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.059 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.059 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.18s1 97/45 ECSubWrite(tid=319, reqid=osd.13.0:319, at_version=97'44, trim_to=0'0, roll_forward_to=68'40))", 2023-11-05T08:34:01.059 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.669028+0000", 2023-11-05T08:34:01.059 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 10.292416286, 2023-11-05T08:34:01.059 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.015017037, 2023-11-05T08:34:01.060 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:01.060 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:01.060 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:01.060 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.060 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:01.061 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.669028+0000", 2023-11-05T08:34:01.061 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:01.061 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.061 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.061 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:01.061 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.669028+0000", 2023-11-05T08:34:01.062 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:01.062 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.062 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.062 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:01.062 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.669029+0000", 2023-11-05T08:34:01.063 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.7700000000000004e-07 2023-11-05T08:34:01.063 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.063 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.063 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:01.063 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.669081+0000", 2023-11-05T08:34:01.063 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.1851000000000002e-05 2023-11-05T08:34:01.064 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.064 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.064 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:01.064 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.669103+0000", 2023-11-05T08:34:01.064 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2146000000000001e-05 2023-11-05T08:34:01.065 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.065 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.065 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:01.065 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.669120+0000", 2023-11-05T08:34:01.065 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6629999999999998e-05 2023-11-05T08:34:01.065 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.066 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.066 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:01.066 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.673823+0000", 2023-11-05T08:34:01.066 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0047034039999999996 2023-11-05T08:34:01.066 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.067 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.067 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:01.067 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.673850+0000", 2023-11-05T08:34:01.067 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6672000000000001e-05 2023-11-05T08:34:01.067 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.068 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.068 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:01.068 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.683983+0000", 2023-11-05T08:34:01.068 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.010133222000000001 2023-11-05T08:34:01.068 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.068 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.069 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:01.069 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.684045+0000", 2023-11-05T08:34:01.069 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.2335000000000005e-05 2023-11-05T08:34:01.069 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:01.069 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:01.070 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:01.070 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.070 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.070 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.10s5 97/45 ECSubWrite(tid=320, reqid=osd.13.0:320, at_version=97'60, trim_to=0'0, roll_forward_to=93'56))", 2023-11-05T08:34:01.071 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.669144+0000", 2023-11-05T08:34:01.071 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 10.292300730000001, 2023-11-05T08:34:01.071 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.019507072, 2023-11-05T08:34:01.071 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:01.072 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:01.072 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:01.072 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.072 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:01.072 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.669144+0000", 2023-11-05T08:34:01.072 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:01.073 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.073 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.073 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:01.073 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.669144+0000", 2023-11-05T08:34:01.073 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:01.074 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.074 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.074 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:01.074 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.669144+0000", 2023-11-05T08:34:01.074 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.6300000000000005e-07 2023-11-05T08:34:01.075 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.075 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.075 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:01.075 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.669194+0000", 2023-11-05T08:34:01.075 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.9372000000000001e-05 2023-11-05T08:34:01.076 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.076 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.076 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:01.076 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.669215+0000", 2023-11-05T08:34:01.076 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.158e-05 2023-11-05T08:34:01.076 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.077 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.077 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:01.077 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.669231+0000", 2023-11-05T08:34:01.077 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5892e-05 2023-11-05T08:34:01.077 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.078 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.078 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:01.078 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.678029+0000", 2023-11-05T08:34:01.078 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0087977669999999997 2023-11-05T08:34:01.078 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.079 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.079 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:01.079 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.678056+0000", 2023-11-05T08:34:01.079 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7219000000000001e-05 2023-11-05T08:34:01.079 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.080 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.080 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:01.080 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.688571+0000", 2023-11-05T08:34:01.080 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.010514931999999999 2023-11-05T08:34:01.080 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.080 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.081 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:01.081 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.688651+0000", 2023-11-05T08:34:01.081 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.9647000000000005e-05 2023-11-05T08:34:01.081 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:01.081 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:01.082 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:01.082 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.082 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.082 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.11s5 97/45 ECSubWrite(tid=19, reqid=osd.10.0:19, at_version=97'29, trim_to=0'0, roll_forward_to=94'27))", 2023-11-05T08:34:01.082 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.670156+0000", 2023-11-05T08:34:01.083 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 10.291288027, 2023-11-05T08:34:01.083 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.01292214, 2023-11-05T08:34:01.083 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:01.083 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:01.083 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:01.083 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.084 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:01.084 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.670156+0000", 2023-11-05T08:34:01.084 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:01.084 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.084 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.085 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:01.085 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.670156+0000", 2023-11-05T08:34:01.085 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:01.085 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.085 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.086 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:01.086 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.670158+0000", 2023-11-05T08:34:01.086 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9360000000000002e-06 2023-11-05T08:34:01.086 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.086 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.086 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:01.087 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.670220+0000", 2023-11-05T08:34:01.087 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.1587999999999994e-05 2023-11-05T08:34:01.087 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.087 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.087 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:01.088 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.670252+0000", 2023-11-05T08:34:01.088 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2252000000000002e-05 2023-11-05T08:34:01.088 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.088 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.088 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:01.089 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.670274+0000", 2023-11-05T08:34:01.089 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1549999999999999e-05 2023-11-05T08:34:01.089 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.089 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.089 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:01.089 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.670436+0000", 2023-11-05T08:34:01.090 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00016249 2023-11-05T08:34:01.090 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.090 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.090 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:01.090 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.670461+0000", 2023-11-05T08:34:01.091 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5096000000000002e-05 2023-11-05T08:34:01.091 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.091 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.091 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:01.091 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.683006+0000", 2023-11-05T08:34:01.092 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.012544504 2023-11-05T08:34:01.092 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.092 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.092 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:01.092 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.683078+0000", 2023-11-05T08:34:01.093 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.2724000000000004e-05 2023-11-05T08:34:01.093 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:01.093 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:01.093 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:01.093 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.094 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.094 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.es5 97/30 ECSubWrite(tid=128, reqid=osd.2.0:128, at_version=97'100, trim_to=0'0, roll_forward_to=94'96))", 2023-11-05T08:34:01.094 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.673930+0000", 2023-11-05T08:34:01.094 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 10.287513930999999, 2023-11-05T08:34:01.094 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.014705367, 2023-11-05T08:34:01.094 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:01.095 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:01.095 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:01.095 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.095 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:01.095 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.673930+0000", 2023-11-05T08:34:01.096 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:01.096 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.096 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.096 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:01.096 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.673930+0000", 2023-11-05T08:34:01.097 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:01.097 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.097 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.097 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:01.097 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.673932+0000", 2023-11-05T08:34:01.097 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4789999999999999e-06 2023-11-05T08:34:01.098 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.098 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.098 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:01.098 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.673982+0000", 2023-11-05T08:34:01.098 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.0355999999999997e-05 2023-11-05T08:34:01.099 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.099 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.099 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:01.099 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.674009+0000", 2023-11-05T08:34:01.099 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6392000000000001e-05 2023-11-05T08:34:01.100 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.100 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.100 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:01.100 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.674029+0000", 2023-11-05T08:34:01.100 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0716e-05 2023-11-05T08:34:01.100 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.101 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.101 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:01.101 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.674184+0000", 2023-11-05T08:34:01.101 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00015448099999999999 2023-11-05T08:34:01.101 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.102 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.102 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:01.102 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.674208+0000", 2023-11-05T08:34:01.102 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4533e-05 2023-11-05T08:34:01.102 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.102 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.103 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:01.103 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.688543+0000", 2023-11-05T08:34:01.103 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.01433482 2023-11-05T08:34:01.103 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.103 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.104 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:01.104 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.688636+0000", 2023-11-05T08:34:01.104 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.2590000000000006e-05 2023-11-05T08:34:01.104 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:01.104 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:01.104 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:01.105 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.105 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.105 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.10s1 97/45 ECSubWrite(tid=321, reqid=osd.13.0:320, at_version=0'0, trim_to=0'0, roll_forward_to=97'60))", 2023-11-05T08:34:01.105 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.699620+0000", 2023-11-05T08:34:01.105 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 10.261824664000001, 2023-11-05T08:34:01.106 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.013813372000000001, 2023-11-05T08:34:01.106 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:01.106 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:01.106 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:01.106 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.107 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:01.107 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.699620+0000", 2023-11-05T08:34:01.107 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:01.107 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.107 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.107 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:01.108 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.699620+0000", 2023-11-05T08:34:01.108 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:01.108 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.108 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.108 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:01.108 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.699621+0000", 2023-11-05T08:34:01.109 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.7460000000000001e-06 2023-11-05T08:34:01.109 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.109 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.109 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:01.109 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.699658+0000", 2023-11-05T08:34:01.110 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.6412000000000003e-05 2023-11-05T08:34:01.110 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.110 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.110 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:01.110 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.699691+0000", 2023-11-05T08:34:01.111 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2656e-05 2023-11-05T08:34:01.111 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.111 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.111 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:01.111 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.699715+0000", 2023-11-05T08:34:01.111 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4148999999999999e-05 2023-11-05T08:34:01.112 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.112 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.112 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:01.112 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.700026+0000", 2023-11-05T08:34:01.112 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00031170900000000002 2023-11-05T08:34:01.113 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.113 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.113 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:01.113 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.700051+0000", 2023-11-05T08:34:01.113 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4556e-05 2023-11-05T08:34:01.114 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.114 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.114 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:01.114 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.713354+0000", 2023-11-05T08:34:01.114 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.01330345 2023-11-05T08:34:01.115 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.115 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.115 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:01.115 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.713433+0000", 2023-11-05T08:34:01.115 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.8694000000000002e-05 2023-11-05T08:34:01.115 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:01.116 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:01.116 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:01.116 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.116 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.116 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.10s5 97/45 ECSubWrite(tid=321, reqid=osd.13.0:320, at_version=0'0, trim_to=0'0, roll_forward_to=97'60))", 2023-11-05T08:34:01.117 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.699776+0000", 2023-11-05T08:34:01.117 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 10.26166812, 2023-11-05T08:34:01.117 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.013719832, 2023-11-05T08:34:01.117 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:01.117 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:01.117 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:01.118 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.118 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:01.118 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.699776+0000", 2023-11-05T08:34:01.118 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:01.118 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.119 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.119 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:01.119 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.699776+0000", 2023-11-05T08:34:01.119 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:01.119 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.120 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.120 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:01.120 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.699777+0000", 2023-11-05T08:34:01.120 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.8700000000000004e-07 2023-11-05T08:34:01.120 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.120 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.121 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:01.121 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.699806+0000", 2023-11-05T08:34:01.121 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8626e-05 2023-11-05T08:34:01.121 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.121 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.122 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:01.122 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.699828+0000", 2023-11-05T08:34:01.122 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2602000000000001e-05 2023-11-05T08:34:01.122 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.122 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.123 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:01.123 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.699846+0000", 2023-11-05T08:34:01.123 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.7266999999999999e-05 2023-11-05T08:34:01.123 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.123 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.123 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:01.124 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.700434+0000", 2023-11-05T08:34:01.124 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00058856999999999996 2023-11-05T08:34:01.124 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.124 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.124 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:01.125 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.700467+0000", 2023-11-05T08:34:01.125 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2611999999999999e-05 2023-11-05T08:34:01.125 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.125 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.125 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:01.126 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.713448+0000", 2023-11-05T08:34:01.126 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.012980818 2023-11-05T08:34:01.126 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.126 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.126 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:01.127 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.713496+0000", 2023-11-05T08:34:01.127 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.8350000000000003e-05 2023-11-05T08:34:01.127 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:01.127 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:01.127 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:01.128 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.128 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.128 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.18s1 97/45 ECSubWrite(tid=322, reqid=osd.13.0:319, at_version=0'0, trim_to=0'0, roll_forward_to=97'44))", 2023-11-05T08:34:01.128 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.699885+0000", 2023-11-05T08:34:01.128 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 10.261559332999999, 2023-11-05T08:34:01.128 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.013657423, 2023-11-05T08:34:01.129 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:01.129 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:01.129 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:01.129 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.129 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:01.130 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.699885+0000", 2023-11-05T08:34:01.130 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:01.130 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.130 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.130 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:01.131 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.699885+0000", 2023-11-05T08:34:01.131 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:01.131 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.131 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.131 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:01.131 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.699886+0000", 2023-11-05T08:34:01.132 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.8599999999999997e-07 2023-11-05T08:34:01.132 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.132 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.132 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:01.132 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.699911+0000", 2023-11-05T08:34:01.133 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5355999999999999e-05 2023-11-05T08:34:01.133 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.133 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.133 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:01.133 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.699936+0000", 2023-11-05T08:34:01.133 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5035999999999999e-05 2023-11-05T08:34:01.134 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.134 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.134 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:01.134 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.699954+0000", 2023-11-05T08:34:01.134 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.7535999999999999e-05 2023-11-05T08:34:01.135 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.135 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.135 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:01.135 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.701485+0000", 2023-11-05T08:34:01.135 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001530886 2023-11-05T08:34:01.135 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.136 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.136 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:01.136 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.701507+0000", 2023-11-05T08:34:01.136 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2175999999999999e-05 2023-11-05T08:34:01.136 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.137 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.137 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:01.137 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.713500+0000", 2023-11-05T08:34:01.137 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.011992674 2023-11-05T08:34:01.137 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.137 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.138 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:01.138 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.713542+0000", 2023-11-05T08:34:01.138 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.2972999999999997e-05 2023-11-05T08:34:01.138 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:01.138 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:01.139 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:01.139 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.139 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.139 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.11s5 97/45 ECSubWrite(tid=21, reqid=osd.10.0:20, at_version=0'0, trim_to=0'0, roll_forward_to=97'31))", 2023-11-05T08:34:01.139 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.700121+0000", 2023-11-05T08:34:01.140 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 10.261323105000001, 2023-11-05T08:34:01.140 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.015358910999999999, 2023-11-05T08:34:01.140 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:01.140 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:01.140 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:01.140 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.141 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:01.141 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.700121+0000", 2023-11-05T08:34:01.141 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:01.141 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.141 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.142 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:01.142 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.700121+0000", 2023-11-05T08:34:01.142 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:01.142 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.142 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.143 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:01.143 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.700123+0000", 2023-11-05T08:34:01.143 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.657e-06 2023-11-05T08:34:01.143 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.143 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.143 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:01.144 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.700150+0000", 2023-11-05T08:34:01.144 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7268999999999999e-05 2023-11-05T08:34:01.144 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.144 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.144 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:01.145 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.703027+0000", 2023-11-05T08:34:01.145 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.002876736 2023-11-05T08:34:01.145 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.145 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.145 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:01.145 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.703056+0000", 2023-11-05T08:34:01.146 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.9065e-05 2023-11-05T08:34:01.146 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.146 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.146 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:01.146 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.703188+0000", 2023-11-05T08:34:01.147 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000132008 2023-11-05T08:34:01.147 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.147 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.147 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:01.147 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.703208+0000", 2023-11-05T08:34:01.148 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0387999999999999e-05 2023-11-05T08:34:01.148 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.148 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.148 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:01.148 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.715366+0000", 2023-11-05T08:34:01.148 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.012157340000000001 2023-11-05T08:34:01.149 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.149 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.149 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:01.149 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.715480+0000", 2023-11-05T08:34:01.149 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000114448 2023-11-05T08:34:01.150 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:01.150 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:01.150 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:01.150 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.150 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.150 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.10s1 97/45 ECSubWrite(tid=324, reqid=osd.13.0:324, at_version=97'62, trim_to=0'0, roll_forward_to=97'60))", 2023-11-05T08:34:01.151 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.701788+0000", 2023-11-05T08:34:01.151 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 10.259656253999999, 2023-11-05T08:34:01.151 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.013789007000000001, 2023-11-05T08:34:01.151 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:01.153 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:01.154 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:01.154 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.154 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:01.154 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.701788+0000", 2023-11-05T08:34:01.154 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:01.155 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.155 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.155 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:01.155 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.701788+0000", 2023-11-05T08:34:01.155 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:01.155 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.156 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.156 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:01.156 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.701790+0000", 2023-11-05T08:34:01.156 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9400000000000001e-06 2023-11-05T08:34:01.156 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.157 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.157 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:01.157 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.701853+0000", 2023-11-05T08:34:01.157 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.2471000000000005e-05 2023-11-05T08:34:01.157 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.158 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.158 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:01.158 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.701888+0000", 2023-11-05T08:34:01.158 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.5009000000000003e-05 2023-11-05T08:34:01.158 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.159 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.159 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:01.159 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.701911+0000", 2023-11-05T08:34:01.159 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3136e-05 2023-11-05T08:34:01.159 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.159 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.160 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:01.160 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.702544+0000", 2023-11-05T08:34:01.160 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00063307700000000001 2023-11-05T08:34:01.160 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.160 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.161 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:01.161 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.702565+0000", 2023-11-05T08:34:01.161 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1291e-05 2023-11-05T08:34:01.161 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.161 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.162 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:01.162 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.715516+0000", 2023-11-05T08:34:01.162 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.012951275 2023-11-05T08:34:01.162 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.162 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.162 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:01.163 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.715577+0000", 2023-11-05T08:34:01.163 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.0807999999999998e-05 2023-11-05T08:34:01.163 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:01.163 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:01.163 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:01.164 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.164 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.164 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.10s5 97/45 ECSubWrite(tid=324, reqid=osd.13.0:324, at_version=97'62, trim_to=0'0, roll_forward_to=97'60))", 2023-11-05T08:34:01.164 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.701963+0000", 2023-11-05T08:34:01.164 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 10.259481859999999, 2023-11-05T08:34:01.164 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.013656227999999999, 2023-11-05T08:34:01.165 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:01.165 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:01.165 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:01.165 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.165 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:01.166 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.701963+0000", 2023-11-05T08:34:01.166 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:01.166 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.166 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.166 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:01.167 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.701963+0000", 2023-11-05T08:34:01.167 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:01.167 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.167 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.167 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:01.167 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.701964+0000", 2023-11-05T08:34:01.168 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.026e-06 2023-11-05T08:34:01.168 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.168 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.168 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:01.168 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.702057+0000", 2023-11-05T08:34:01.169 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.3160999999999998e-05 2023-11-05T08:34:01.169 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.169 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.169 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:01.169 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.702082+0000", 2023-11-05T08:34:01.169 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5534999999999999e-05 2023-11-05T08:34:01.170 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.170 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.170 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:01.170 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.702101+0000", 2023-11-05T08:34:01.170 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8490000000000001e-05 2023-11-05T08:34:01.171 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.171 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.171 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:01.171 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.702846+0000", 2023-11-05T08:34:01.171 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00074520999999999995 2023-11-05T08:34:01.172 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.172 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.172 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:01.172 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.702869+0000", 2023-11-05T08:34:01.172 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3394e-05 2023-11-05T08:34:01.173 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.173 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.173 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:01.173 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.715584+0000", 2023-11-05T08:34:01.173 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.012714723000000001 2023-11-05T08:34:01.174 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.174 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.174 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:01.174 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.715619+0000", 2023-11-05T08:34:01.174 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.4688999999999999e-05 2023-11-05T08:34:01.175 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:01.175 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:01.175 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:01.175 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.175 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.175 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.18s1 97/45 ECSubWrite(tid=323, reqid=osd.13.0:323, at_version=97'46, trim_to=0'0, roll_forward_to=97'44))", 2023-11-05T08:34:01.176 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.702129+0000", 2023-11-05T08:34:01.176 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 10.259315256000001, 2023-11-05T08:34:01.176 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.016379057999999998, 2023-11-05T08:34:01.176 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:01.176 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:01.177 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:01.177 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.177 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:01.177 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.702129+0000", 2023-11-05T08:34:01.177 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:01.178 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.178 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.178 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:01.178 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.702129+0000", 2023-11-05T08:34:01.178 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:01.178 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.179 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.179 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:01.179 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.702130+0000", 2023-11-05T08:34:01.179 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.4300000000000002e-07 2023-11-05T08:34:01.179 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.180 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.180 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:01.180 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.702175+0000", 2023-11-05T08:34:01.180 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.5062000000000003e-05 2023-11-05T08:34:01.180 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.180 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.181 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:01.181 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.702198+0000", 2023-11-05T08:34:01.181 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3036000000000001e-05 2023-11-05T08:34:01.181 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.181 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.182 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:01.182 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.702218+0000", 2023-11-05T08:34:01.182 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9949000000000002e-05 2023-11-05T08:34:01.182 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.182 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.183 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:01.183 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.707796+0000", 2023-11-05T08:34:01.183 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0055775260000000002 2023-11-05T08:34:01.183 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.183 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.183 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:01.184 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.707818+0000", 2023-11-05T08:34:01.184 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2385e-05 2023-11-05T08:34:01.184 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.184 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.184 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:01.185 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.718457+0000", 2023-11-05T08:34:01.185 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.010639431 2023-11-05T08:34:01.185 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.185 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.185 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:01.186 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.718508+0000", 2023-11-05T08:34:01.186 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.0825999999999998e-05 2023-11-05T08:34:01.186 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:01.186 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:01.186 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:01.187 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.187 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.187 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.es5 97/30 ECSubWrite(tid=129, reqid=osd.2.0:128, at_version=0'0, trim_to=0'0, roll_forward_to=97'100))", 2023-11-05T08:34:01.187 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.704114+0000", 2023-11-05T08:34:01.187 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 10.257330668, 2023-11-05T08:34:01.187 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.013277532999999999, 2023-11-05T08:34:01.188 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:01.188 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:01.188 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:01.188 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.188 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:01.189 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.704114+0000", 2023-11-05T08:34:01.189 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:01.189 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.189 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.189 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:01.189 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.704114+0000", 2023-11-05T08:34:01.190 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:01.190 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.190 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.190 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:01.190 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.704115+0000", 2023-11-05T08:34:01.191 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.68e-06 2023-11-05T08:34:01.191 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.191 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.191 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:01.191 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.704139+0000", 2023-11-05T08:34:01.191 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3238999999999999e-05 2023-11-05T08:34:01.192 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.192 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.192 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:01.192 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.704167+0000", 2023-11-05T08:34:01.192 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8591000000000001e-05 2023-11-05T08:34:01.193 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.193 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.193 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:01.193 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.704186+0000", 2023-11-05T08:34:01.193 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9031000000000001e-05 2023-11-05T08:34:01.193 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.194 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.194 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:01.194 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.705305+0000", 2023-11-05T08:34:01.194 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0011183740000000001 2023-11-05T08:34:01.194 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.195 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.195 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:01.195 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.705329+0000", 2023-11-05T08:34:01.195 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4006e-05 2023-11-05T08:34:01.195 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.195 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.196 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:01.196 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.717332+0000", 2023-11-05T08:34:01.196 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.012003131 2023-11-05T08:34:01.196 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:01.196 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:01.197 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:01.197 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.717391+0000", 2023-11-05T08:34:01.197 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.9481000000000002e-05 2023-11-05T08:34:01.197 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:01.197 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:01.197 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:01.198 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:01.198 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:01.198 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:34:01.200 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2023-11-05T08:34:01.200 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-11-05T08:34:01.198+0000 7f72b2c5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 42525) UID: 0 2023-11-05T08:34:01.297 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-05T08:34:01.300 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:34:01.298+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:34:01.374 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:34:01.374 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2023-11-05T08:34:01.375 INFO:teuthology.orchestra.run.smithi069.stdout: "num_ops": 0 2023-11-05T08:34:01.375 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:34:01.388 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2023-11-05T08:34:01.401 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:34:01.397+0000 7fdc5cd58640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:34:01.435 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-11-05T08:34:01.458 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1: {} 2023-11-05T08:34:01.458 INFO:teuthology.orchestra.run.smithi002.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-11-05T08:34:01.476 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2: {} 2023-11-05T08:34:01.477 INFO:teuthology.orchestra.run.smithi002.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-11-05T08:34:01.497 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3: {} 2023-11-05T08:34:01.497 INFO:teuthology.orchestra.run.smithi002.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-11-05T08:34:01.503 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:34:01.496+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:34:01.510 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:34:01.510 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2023-11-05T08:34:01.510 INFO:teuthology.orchestra.run.smithi069.stdout: "complaint_time": 30, 2023-11-05T08:34:01.510 INFO:teuthology.orchestra.run.smithi069.stdout: "num_blocked_ops": 0 2023-11-05T08:34:01.511 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:34:01.518 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4: {} 2023-11-05T08:34:01.518 INFO:teuthology.orchestra.run.smithi002.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-11-05T08:34:01.525 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2023-11-05T08:34:01.539 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5: {} 2023-11-05T08:34:01.539 INFO:teuthology.orchestra.run.smithi002.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-11-05T08:34:01.563 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6: {} 2023-11-05T08:34:01.563 INFO:teuthology.orchestra.run.smithi002.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-11-05T08:34:01.584 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7: {} 2023-11-05T08:34:01.585 INFO:teuthology.orchestra.run.smithi002.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-11-05T08:34:01.603 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:34:01.600+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:34:01.606 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8: {} 2023-11-05T08:34:01.607 INFO:teuthology.orchestra.run.smithi002.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-11-05T08:34:01.629 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9: {} 2023-11-05T08:34:01.630 INFO:teuthology.orchestra.run.smithi002.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-11-05T08:34:01.652 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10: {} 2023-11-05T08:34:01.652 INFO:teuthology.orchestra.run.smithi002.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-11-05T08:34:01.676 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11: {} 2023-11-05T08:34:01.677 INFO:teuthology.orchestra.run.smithi002.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-11-05T08:34:01.692 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:34:01.692 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:34:01.699 INFO:teuthology.orchestra.run.smithi002.stdout:osd.12: {} 2023-11-05T08:34:01.700 INFO:teuthology.orchestra.run.smithi002.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-11-05T08:34:01.704 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:34:01.700+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:34:01.706 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2023-11-05T08:34:01.723 INFO:teuthology.orchestra.run.smithi002.stdout:osd.13: {} 2023-11-05T08:34:01.723 INFO:teuthology.orchestra.run.smithi002.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-11-05T08:34:01.740 INFO:teuthology.orchestra.run.smithi002.stdout:osd.14: {} 2023-11-05T08:34:01.741 INFO:teuthology.orchestra.run.smithi002.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-11-05T08:34:01.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-05T08:34:01.737+0000 7f4512ffd640 0 -- 172.21.15.2:0/1019992729 >> [v2:172.21.15.90:6824/739151808,v1:172.21.15.90:6825/739151808] conn(0x7f451421c650 msgr2=0x7f451421ea60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-11-05T08:34:01.798 INFO:teuthology.orchestra.run.smithi002.stderr:reweighted osd.14 to 1 (10000) 2023-11-05T08:34:01.805 INFO:tasks.rados.rados.0.smithi090.stdout:368: done (1 left) 2023-11-05T08:34:01.805 INFO:tasks.rados.rados.0.smithi090.stdout:369: read oid 25 snap -1 2023-11-05T08:34:01.806 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:34:01.802+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:34:01.806 INFO:tasks.rados.rados.0.smithi090.stdout:369: expect deleted 2023-11-05T08:34:01.807 INFO:tasks.rados.rados.0.smithi090.stdout:370: read oid 49 snap -1 2023-11-05T08:34:01.807 INFO:tasks.rados.rados.0.smithi090.stdout:370: expect (ObjNum 120 snap 32 seq_num 120) 2023-11-05T08:34:01.807 INFO:tasks.rados.rados.0.smithi090.stdout:371: rollback oid 43 current snap is 38 2023-11-05T08:34:01.807 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 43 to 36 2023-11-05T08:34:01.807 INFO:tasks.rados.rados.0.smithi090.stdout:367: finishing copy_from to smithi09039252-19 2023-11-05T08:34:01.808 INFO:tasks.rados.rados.0.smithi090.stdout:367: got expected ENOENT (src dne) 2023-11-05T08:34:01.808 INFO:tasks.rados.rados.0.smithi090.stdout:372: setattr oid 14 current snap is 38 2023-11-05T08:34:01.808 INFO:tasks.rados.rados.0.smithi090.stdout:367: finishing copy_from racing read to smithi09039252-19 2023-11-05T08:34:01.809 INFO:tasks.rados.rados.0.smithi090.stdout:367: done (4 left) 2023-11-05T08:34:01.809 INFO:tasks.rados.rados.0.smithi090.stdout:373: delete oid 15 current snap is 38 2023-11-05T08:34:01.823 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 14 v 128 (ObjNum 3 snap 0 seq_num 3) dirty exists 2023-11-05T08:34:01.824 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2023-11-05T08:34:01.825 INFO:tasks.rados.rados.0.smithi090.stdout:371: finishing rollback tid 1 to smithi09039252-43 2023-11-05T08:34:01.825 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 43 v 40 (ObjNum 161 snap 0 seq_num 1) dirty dne 2023-11-05T08:34:01.826 INFO:tasks.rados.rados.0.smithi090.stdout:369: done (4 left) 2023-11-05T08:34:01.826 INFO:tasks.rados.rados.0.smithi090.stdout:371: done (3 left) 2023-11-05T08:34:01.827 INFO:tasks.rados.rados.0.smithi090.stdout:372: done (2 left) 2023-11-05T08:34:01.827 INFO:tasks.rados.rados.0.smithi090.stdout:373: done (1 left) 2023-11-05T08:34:01.827 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 26 current snap is 38 2023-11-05T08:34:01.827 INFO:tasks.rados.rados.0.smithi090.stdout:374: seq_num 140 ranges {6193152=802816,6995968=425984,7421952=720896,8142848=409600} 2023-11-05T08:34:01.833 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:34:01.833 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:34:01.852 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2023-11-05T08:34:01.857 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-11-05T08:34:01.905 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~66s 2023-11-05T08:34:01.910 INFO:tasks.rados.rados.0.smithi090.stdout:374: writing smithi09039252-26 from 6193152 to 6995968 tid 1 2023-11-05T08:34:01.911 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:34:01.909+0000 7fdc5cd58640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:34:01.913 INFO:tasks.rados.rados.0.smithi090.stdout:374: writing smithi09039252-26 from 6995968 to 7421952 tid 2 2023-11-05T08:34:01.919 INFO:tasks.rados.rados.0.smithi090.stdout:374: writing smithi09039252-26 from 7421952 to 8142848 tid 3 2023-11-05T08:34:01.922 INFO:tasks.rados.rados.0.smithi090.stdout:374: writing smithi09039252-26 from 8142848 to 8552448 tid 4 2023-11-05T08:34:01.922 INFO:tasks.rados.rados.0.smithi090.stdout:375: rollback oid 45 current snap is 38 2023-11-05T08:34:01.922 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 45 to 36 2023-11-05T08:34:01.922 INFO:tasks.rados.rados.0.smithi090.stdout:370: expect (ObjNum 120 snap 32 seq_num 120) 2023-11-05T08:34:01.966 INFO:teuthology.orchestra.run.smithi002.stdout:osd.15: {} 2023-11-05T08:34:01.969 INFO:teuthology.orchestra.run.smithi002.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-11-05T08:34:01.971 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-05T08:34:01.971 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-05T08:34:01.972 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:34:01.972 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:34:01.986 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2023-11-05T08:34:02.032 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:34:02.028+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:34:02.097 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:34:02.097 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:34:02.108 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:34:02.104+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:34:02.109 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-11-05T08:34:02.208 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:34:02.204+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:34:02.227 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:34:02.228 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:34:02.241 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-11-05T08:34:02.257 INFO:tasks.rados.rados.0.smithi090.stdout:374: finishing write tid 1 to smithi09039252-26 2023-11-05T08:34:02.258 INFO:tasks.rados.rados.0.smithi090.stdout:374: finishing write tid 2 to smithi09039252-26 2023-11-05T08:34:02.258 INFO:tasks.rados.rados.0.smithi090.stdout:370: done (2 left) 2023-11-05T08:34:02.258 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 20 current snap is 38 2023-11-05T08:34:02.259 INFO:tasks.rados.rados.0.smithi090.stdout:375: finishing rollback tid 0 to smithi09039252-45 2023-11-05T08:34:02.259 INFO:tasks.rados.rados.0.smithi090.stdout:375: finishing rollback tid 1 to smithi09039252-45 2023-11-05T08:34:02.259 INFO:tasks.rados.rados.0.smithi090.stdout:374: finishing write tid 3 to smithi09039252-26 2023-11-05T08:34:02.259 INFO:tasks.rados.rados.0.smithi090.stdout:376: seq_num 141 ranges {2310144=425984,2736128=573440,3309568=425984,3735552=524288,4259840=425984} 2023-11-05T08:34:02.274 INFO:tasks.rados.rados.0.smithi090.stdout:376: writing smithi09039252-20 from 2310144 to 2736128 tid 1 2023-11-05T08:34:02.278 INFO:tasks.rados.rados.0.smithi090.stdout:376: writing smithi09039252-20 from 2736128 to 3309568 tid 2 2023-11-05T08:34:02.281 INFO:tasks.rados.rados.0.smithi090.stdout:376: writing smithi09039252-20 from 3309568 to 3735552 tid 3 2023-11-05T08:34:02.285 INFO:tasks.rados.rados.0.smithi090.stdout:376: writing smithi09039252-20 from 3735552 to 4259840 tid 4 2023-11-05T08:34:02.289 INFO:tasks.rados.rados.0.smithi090.stdout:376: writing smithi09039252-20 from 4259840 to 4685824 tid 5 2023-11-05T08:34:02.289 INFO:tasks.rados.rados.0.smithi090.stdout:377: read oid 37 snap -1 2023-11-05T08:34:02.289 INFO:tasks.rados.rados.0.smithi090.stdout:377: expect (ObjNum 93 snap 16 seq_num 93) 2023-11-05T08:34:02.290 INFO:tasks.rados.rados.0.smithi090.stdout:378: read oid 36 snap -1 2023-11-05T08:34:02.290 INFO:tasks.rados.rados.0.smithi090.stdout:378: expect (ObjNum 17 snap 0 seq_num 17) 2023-11-05T08:34:02.290 INFO:tasks.rados.rados.0.smithi090.stdout:379: rollback oid 35 current snap is 38 2023-11-05T08:34:02.290 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 35 to 36 2023-11-05T08:34:02.290 INFO:tasks.rados.rados.0.smithi090.stdout:380: read oid 17 snap 33 2023-11-05T08:34:02.291 INFO:tasks.rados.rados.0.smithi090.stdout:380: expect deleted 2023-11-05T08:34:02.291 INFO:tasks.rados.rados.0.smithi090.stdout:381: read oid 29 snap -1 2023-11-05T08:34:02.291 INFO:tasks.rados.rados.0.smithi090.stdout:381: expect (ObjNum 114 snap 29 seq_num 114) 2023-11-05T08:34:02.291 INFO:tasks.rados.rados.0.smithi090.stdout:382: snap_remove snap 34 2023-11-05T08:34:02.310 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:34:02.305+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:34:02.372 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:34:02.373 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:34:02.388 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-11-05T08:34:02.432 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:34:02.430+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:34:02.512 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:34:02.508+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:34:02.557 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:34:02.558 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:34:02.571 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2023-11-05T08:34:02.614 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:34:02.612+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:34:02.694 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:34:02.695 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:34:02.709 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2023-11-05T08:34:02.715 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:34:02.708+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:34:02.816 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:34:02.811+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:34:02.820 INFO:teuthology.orchestra.run.smithi002.stderr:nodeep-scrub is unset 2023-11-05T08:34:02.820 INFO:teuthology.orchestra.run.smithi002.stderr:reweighted osd.15 to 1 (10000) 2023-11-05T08:34:02.828 INFO:tasks.rados.rados.0.smithi090.stdout:382: done (8 left) 2023-11-05T08:34:02.828 INFO:tasks.rados.rados.0.smithi090.stdout:append oid 16 current snap is 38 2023-11-05T08:34:02.829 INFO:tasks.rados.rados.0.smithi090.stdout:375: finishing rollback tid 2 to smithi09039252-45 2023-11-05T08:34:02.829 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 45 v 58 (ObjNum 80 snap 13 seq_num 80) dirty exists 2023-11-05T08:34:02.829 INFO:tasks.rados.rados.0.smithi090.stdout:374: finishing write tid 4 to smithi09039252-26 2023-11-05T08:34:02.829 INFO:tasks.rados.rados.0.smithi090.stdout:374: finishing write tid 5 to smithi09039252-26 2023-11-05T08:34:02.830 INFO:tasks.rados.rados.0.smithi090.stdout:374: finishing write tid 6 to smithi09039252-26 2023-11-05T08:34:02.830 INFO:tasks.rados.rados.0.smithi090.stdout:374: oid 26 updating version 0 to 82 2023-11-05T08:34:02.830 INFO:tasks.rados.rados.0.smithi090.stdout:374: oid 26 updating version 82 to 83 2023-11-05T08:34:02.830 INFO:tasks.rados.rados.0.smithi090.stdout:374: oid 26 updating version 83 to 85 2023-11-05T08:34:02.831 INFO:tasks.rados.rados.0.smithi090.stdout:374: oid 26 version 85 is already newer than 84 2023-11-05T08:34:02.831 INFO:tasks.rados.rados.0.smithi090.stdout:374: oid 26 version 85 is already newer than 81 2023-11-05T08:34:02.831 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 26 v 85 (ObjNum 140 snap 38 seq_num 140) dirty exists 2023-11-05T08:34:02.831 INFO:tasks.rados.rados.0.smithi090.stdout:374: left oid 26 (ObjNum 140 snap 38 seq_num 140) 2023-11-05T08:34:02.832 INFO:tasks.rados.rados.0.smithi090.stdout:376: finishing write tid 1 to smithi09039252-20 2023-11-05T08:34:02.832 INFO:tasks.rados.rados.0.smithi090.stdout:376: finishing write tid 2 to smithi09039252-20 2023-11-05T08:34:02.832 INFO:tasks.rados.rados.0.smithi090.stdout:379: finishing rollback tid 0 to smithi09039252-35 2023-11-05T08:34:02.833 INFO:tasks.rados.rados.0.smithi090.stdout:379: finishing rollback tid 1 to smithi09039252-35 2023-11-05T08:34:02.833 INFO:tasks.rados.rados.0.smithi090.stdout:378: expect (ObjNum 17 snap 0 seq_num 17) 2023-11-05T08:34:02.835 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:34:02.835 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:34:02.841 INFO:tasks.rados.rados.0.smithi090.stdout:379: finishing rollback tid 2 to smithi09039252-35 2023-11-05T08:34:02.841 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 35 v 34 (ObjNum 136 snap 36 seq_num 136) dirty exists 2023-11-05T08:34:02.842 INFO:tasks.rados.rados.0.smithi090.stdout:376: finishing write tid 3 to smithi09039252-20 2023-11-05T08:34:02.842 INFO:tasks.rados.rados.0.smithi090.stdout:376: finishing write tid 4 to smithi09039252-20 2023-11-05T08:34:02.842 INFO:tasks.rados.rados.0.smithi090.stdout:376: finishing write tid 5 to smithi09039252-20 2023-11-05T08:34:02.842 INFO:tasks.rados.rados.0.smithi090.stdout:376: finishing write tid 6 to smithi09039252-20 2023-11-05T08:34:02.843 INFO:tasks.rados.rados.0.smithi090.stdout:383: seq_num 142 ranges {1409024=442368,1851392=720896,2572288=540672,3112960=688128} 2023-11-05T08:34:02.856 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2023-11-05T08:34:02.858 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2023-11-05T08:34:02.858 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-05T08:34:02.870 INFO:tasks.rados.rados.0.smithi090.stdout:383: writing smithi09039252-16 from 1409024 to 1851392 tid 1 2023-11-05T08:34:02.882 INFO:tasks.rados.rados.0.smithi090.stdout:383: writing smithi09039252-16 from 1851392 to 2572288 tid 2 2023-11-05T08:34:02.889 INFO:tasks.rados.rados.0.smithi090.stdout:383: writing smithi09039252-16 from 2572288 to 3112960 tid 3 2023-11-05T08:34:02.898 INFO:tasks.rados.rados.0.smithi090.stdout:383: writing smithi09039252-16 from 3112960 to 3801088 tid 4 2023-11-05T08:34:02.898 INFO:tasks.rados.rados.0.smithi090.stdout:374: done (8 left) 2023-11-05T08:34:02.898 INFO:tasks.rados.rados.0.smithi090.stdout:375: done (7 left) 2023-11-05T08:34:02.898 INFO:tasks.rados.rados.0.smithi090.stdout:378: done (6 left) 2023-11-05T08:34:02.899 INFO:tasks.rados.rados.0.smithi090.stdout:379: done (5 left) 2023-11-05T08:34:02.899 INFO:tasks.rados.rados.0.smithi090.stdout:380: done (4 left) 2023-11-05T08:34:02.899 INFO:tasks.rados.rados.0.smithi090.stdout:384: snap_remove snap 36 2023-11-05T08:34:02.917 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:34:02.912+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:34:02.985 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:34:02.986 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:34:03.000 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2023-11-05T08:34:03.018 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:34:03.016+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:34:03.119 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:34:03.115+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:34:03.167 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:34:03.168 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:34:03.182 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2023-11-05T08:34:03.208 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:34:03.212 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-11-05T08:34:03.220 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:34:03.216+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:34:03.229 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":141,"stamp":"2023-11-05T08:34:02.822007+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":116531744,"num_objects":111,"num_object_clones":61,"num_object_copies":767,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":71,"num_objects_misplaced":8,"num_objects_unfound":0,"num_objects_dirty":111,"num_whiteouts":4,"num_read":480,"num_read_kb":152086,"num_write":856,"num_write_kb":119260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":23068672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1147,"ondisk_log_size":1147,"up":199,"acting":199,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":190,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":664896,"kb_used_data":198104,"kb_used_omap":29,"kb_used_meta":466530,"kb_avail":1509284544,"statfs":{"total":1546188226560,"available":1545507373056,"internally_reserved":0,"allocated":202858496,"data_stored":303144385,"data_compressed":2502735,"data_compressed_allocated":5427200,"data_compressed_original":11747328,"omap_allocated":30361,"internal_metadata":477727079},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7061504,"num_objects":6,"num_object_clones":6,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":20,"num_read_kb":30871,"num_write":43,"num_write_kb":6896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.049608"},"pg_stats":[{"pgid":"3.19","version":"108'58","reported_seq":359,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-05T08:34:02.297869+0000","last_change":"2023-11-05T08:34:01.803774+0000","last_active":"2023-11-05T08:34:02.297869+0000","last_peered":"2023-11-05T08:34:02.297869+0000","last_clean":"2023-11-05T08:34:02.297869+0000","last_became_active":"2023-11-05T08:32:59.211522+0000","last_became_peered":"2023-11-05T08:32:59.211522+0000","last_unstale":"2023-11-05T08:34:02.297869+0000","last_undegraded":"2023-11-05T08:34:02.297869+0000","last_fullsized":"2023-11-05T08:34:02.297869+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":5,"last_scrub":"39'22","last_scrub_stamp":"2023-11-05T08:33:01.870959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:01.870959+0000","objects_scrubbed":3,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:14.506291+0000","scrub_duration":0.0087276880000000008,"objects_trimmed":0,"snaptrim_duration":0.00096281600000000002,"stat_sum":{"num_bytes":5734400,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":3027,"num_write":11,"num_write_kb":4144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,11,9,13,7,1],"acting":[8,4,11,9,13,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.18","version":"103'57","reported_seq":328,"reported_epoch":104,"state":"active+clean+remapped","last_fresh":"2023-11-05T08:33:57.793865+0000","last_change":"2023-11-05T08:33:54.765083+0000","last_active":"2023-11-05T08:33:57.793865+0000","last_peered":"2023-11-05T08:33:57.793865+0000","last_clean":"2023-11-05T08:33:57.793865+0000","last_became_active":"2023-11-05T08:32:59.217241+0000","last_became_peered":"2023-11-05T08:32:59.217241+0000","last_unstale":"2023-11-05T08:33:57.793865+0000","last_undegraded":"2023-11-05T08:33:57.793865+0000","last_fullsized":"2023-11-05T08:33:57.793865+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":5,"last_scrub":"52'28","last_scrub_stamp":"2023-11-05T08:33:13.971775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:13.971775+0000","objects_scrubbed":5,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:36.131790+0000","scrub_duration":0.019915236999999999,"objects_trimmed":0,"snaptrim_duration":0.021818140999999999,"stat_sum":{"num_bytes":6160384,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":3739,"num_write":31,"num_write_kb":4336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,2147483647,4,5,7,12],"acting":[13,6,2,4,5,7,12],"avail_no_missing":["13(0)","2(2)","4(3)","5(4)","6(1)","7(5)","12(6)"],"object_location_counts":[{"shards":"2(2),4(3),5(4),6(1),7(5),12(6),13(0)","objects":4}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"}]},{"pgid":"3.17","version":"105'36","reported_seq":383,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-05T08:34:02.203149+0000","last_change":"2023-11-05T08:34:02.204287+0000","last_active":"2023-11-05T08:34:02.203149+0000","last_peered":"2023-11-05T08:34:02.203149+0000","last_clean":"2023-11-05T08:34:02.203149+0000","last_became_active":"2023-11-05T08:33:50.677365+0000","last_became_peered":"2023-11-05T08:33:50.677365+0000","last_unstale":"2023-11-05T08:34:02.203149+0000","last_undegraded":"2023-11-05T08:34:02.203149+0000","last_fullsized":"2023-11-05T08:34:02.203149+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"49'17","last_scrub_stamp":"2023-11-05T08:33:13.975356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:13.975356+0000","objects_scrubbed":3,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:33.996216+0000","scrub_duration":0.014066680999999999,"objects_trimmed":0,"snaptrim_duration":0.001635697,"stat_sum":{"num_bytes":2588672,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":45,"num_write_kb":2176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":999424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,10,9,7,1,13],"acting":[5,3,10,9,7,1,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.16","version":"97'48","reported_seq":326,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-05T08:34:01.820510+0000","last_change":"2023-11-05T08:34:01.821079+0000","last_active":"2023-11-05T08:34:01.820510+0000","last_peered":"2023-11-05T08:34:01.820510+0000","last_clean":"2023-11-05T08:34:01.820510+0000","last_became_active":"2023-11-05T08:32:59.210858+0000","last_became_peered":"2023-11-05T08:32:59.210858+0000","last_unstale":"2023-11-05T08:34:01.820510+0000","last_undegraded":"2023-11-05T08:34:01.820510+0000","last_fullsized":"2023-11-05T08:34:01.820510+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":5,"last_scrub":"46'13","last_scrub_stamp":"2023-11-05T08:33:00.659926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:00.659926+0000","objects_scrubbed":1,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:06.720172+0000","scrub_duration":0.0077151709999999998,"objects_trimmed":0,"snaptrim_duration":0.00073938499999999996,"stat_sum":{"num_bytes":4669440,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":15,"num_write_kb":4560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15,4,8,9,5,10],"acting":[11,15,4,8,9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.15","version":"36'15","reported_seq":268,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-05T08:34:01.805163+0000","last_change":"2023-11-05T08:34:01.806289+0000","last_active":"2023-11-05T08:34:01.805163+0000","last_peered":"2023-11-05T08:34:01.805163+0000","last_clean":"2023-11-05T08:34:01.805163+0000","last_became_active":"2023-11-05T08:33:50.675923+0000","last_became_peered":"2023-11-05T08:33:50.675923+0000","last_unstale":"2023-11-05T08:34:01.805163+0000","last_undegraded":"2023-11-05T08:34:01.805163+0000","last_fullsized":"2023-11-05T08:34:01.805163+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"36'15","last_scrub_stamp":"2023-11-05T08:33:52.537969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:52.537969+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:35:09.298059+0000","scrub_duration":0.0054037720000000003,"objects_trimmed":0,"snaptrim_duration":0.0020680849999999999,"stat_sum":{"num_bytes":32768,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,3,13,14,2,15],"acting":[9,6,3,13,14,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.14","version":"101'27","reported_seq":281,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-05T08:34:01.806326+0000","last_change":"2023-11-05T08:34:01.807260+0000","last_active":"2023-11-05T08:34:01.806326+0000","last_peered":"2023-11-05T08:34:01.806326+0000","last_clean":"2023-11-05T08:34:01.806326+0000","last_became_active":"2023-11-05T08:33:50.674132+0000","last_became_peered":"2023-11-05T08:33:50.674132+0000","last_unstale":"2023-11-05T08:34:01.806326+0000","last_undegraded":"2023-11-05T08:34:01.806326+0000","last_fullsized":"2023-11-05T08:34:01.806326+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"49'18","last_scrub_stamp":"2023-11-05T08:33:04.812402+0000","last_deep_scrub":"49'18","last_deep_scrub_stamp":"2023-11-05T08:33:04.812402+0000","last_clean_scrub_stamp":"2023-11-05T08:33:04.812402+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:09.862950+0000","scrub_duration":0.0086643829999999995,"objects_trimmed":0,"snaptrim_duration":0.0023598249999999999,"stat_sum":{"num_bytes":294912,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1966080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,15,1,13,8,4],"acting":[3,9,15,1,13,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":98,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-05T08:34:01.152657+0000","last_change":"2023-11-05T08:33:50.142139+0000","last_active":"2023-11-05T08:34:01.152657+0000","last_peered":"2023-11-05T08:34:01.152657+0000","last_clean":"2023-11-05T08:34:01.152657+0000","last_became_active":"2023-11-05T08:32:31.941858+0000","last_became_peered":"2023-11-05T08:32:31.941858+0000","last_unstale":"2023-11-05T08:34:01.152657+0000","last_undegraded":"2023-11-05T08:34:01.152657+0000","last_fullsized":"2023-11-05T08:34:01.152657+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:33:50.141700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:33:50.141700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:51.467865+0000","scrub_duration":0.0025514660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"35'9","reported_seq":178,"reported_epoch":104,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:33:57.794197+0000","last_change":"2023-11-05T08:32:54.153555+0000","last_active":"2023-11-05T08:33:57.794197+0000","last_peered":"2023-11-05T08:33:57.794197+0000","last_clean":"2023-11-05T08:32:48.102597+0000","last_became_active":"2023-11-05T08:32:54.153555+0000","last_became_peered":"2023-11-05T08:32:54.153555+0000","last_unstale":"2023-11-05T08:33:57.794197+0000","last_undegraded":"2023-11-05T08:32:54.121924+0000","last_fullsized":"2023-11-05T08:32:54.120000+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":32,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1089536,"num_objects":1,"num_object_clones":1,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":1064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,13,1,15,12,7,9],"acting":[2147483647,13,1,15,12,7,9],"avail_no_missing":["13(1)","1(2)","7(5)","9(6)","12(4)","15(3)"],"object_location_counts":[{"shards":"1(2),7(5),9(6),12(4),13(1),15(3)","objects":1}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":99,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-05T08:34:01.451381+0000","last_change":"2023-11-05T08:33:52.971633+0000","last_active":"2023-11-05T08:34:01.451381+0000","last_peered":"2023-11-05T08:34:01.451381+0000","last_clean":"2023-11-05T08:34:01.451381+0000","last_became_active":"2023-11-05T08:32:31.941207+0000","last_became_peered":"2023-11-05T08:32:31.941207+0000","last_unstale":"2023-11-05T08:34:01.451381+0000","last_undegraded":"2023-11-05T08:34:01.451381+0000","last_fullsized":"2023-11-05T08:34:01.451381+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:33:52.971456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:33:52.971456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:35:07.735098+0000","scrub_duration":0.002374414,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"39'12","reported_seq":197,"reported_epoch":108,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:34:01.929244+0000","last_change":"2023-11-05T08:32:54.151639+0000","last_active":"2023-11-05T08:34:01.929244+0000","last_peered":"2023-11-05T08:34:01.929244+0000","last_clean":"2023-11-05T08:32:49.074431+0000","last_became_active":"2023-11-05T08:32:54.151639+0000","last_became_peered":"2023-11-05T08:32:54.151639+0000","last_unstale":"2023-11-05T08:34:01.929244+0000","last_undegraded":"2023-11-05T08:32:54.120694+0000","last_fullsized":"2023-11-05T08:32:54.119652+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":34,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030081180000000002,"stat_sum":{"num_bytes":319488,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,2147483647,8,5,11],"acting":[4,6,9,2147483647,8,5,11],"avail_no_missing":["4(0)","5(5)","6(1)","8(4)","9(2)","11(6)"],"object_location_counts":[{"shards":"4(0),5(5),6(1),8(4),9(2),11(6)","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":99,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-11-05T08:33:59.863392+0000","last_change":"2023-11-05T08:33:50.446617+0000","last_active":"2023-11-05T08:33:59.863392+0000","last_peered":"2023-11-05T08:33:59.863392+0000","last_clean":"2023-11-05T08:33:59.863392+0000","last_became_active":"2023-11-05T08:32:31.940312+0000","last_became_peered":"2023-11-05T08:32:31.940312+0000","last_unstale":"2023-11-05T08:33:59.863392+0000","last_undegraded":"2023-11-05T08:33:59.863392+0000","last_fullsized":"2023-11-05T08:33:59.863392+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:33:50.446312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:33:50.446312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:57.987976+0000","scrub_duration":0.0037389620000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"39'12","reported_seq":196,"reported_epoch":108,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:34:01.803840+0000","last_change":"2023-11-05T08:32:54.141915+0000","last_active":"2023-11-05T08:34:01.803840+0000","last_peered":"2023-11-05T08:34:01.803840+0000","last_clean":"2023-11-05T08:32:49.088117+0000","last_became_active":"2023-11-05T08:32:54.141915+0000","last_became_peered":"2023-11-05T08:32:54.141915+0000","last_unstale":"2023-11-05T08:34:01.803840+0000","last_undegraded":"2023-11-05T08:32:54.115420+0000","last_fullsized":"2023-11-05T08:32:54.114438+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":34,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.011061272,"stat_sum":{"num_bytes":1376256,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,4,1,2147483647,14,13],"acting":[8,2,4,1,2147483647,14,13],"avail_no_missing":["8(0)","1(3)","2(1)","4(2)","13(6)","14(5)"],"object_location_counts":[{"shards":"1(3),2(1),4(2),8(0),13(6),14(5)","objects":1}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":96,"reported_epoch":107,"state":"active+undersized","last_fresh":"2023-11-05T08:34:01.632325+0000","last_change":"2023-11-05T08:32:51.087583+0000","last_active":"2023-11-05T08:34:01.632325+0000","last_peered":"2023-11-05T08:34:01.632325+0000","last_clean":"2023-11-05T08:32:49.071396+0000","last_became_active":"2023-11-05T08:32:51.087583+0000","last_became_peered":"2023-11-05T08:32:51.087583+0000","last_unstale":"2023-11-05T08:34:01.632325+0000","last_undegraded":"2023-11-05T08:34:01.632325+0000","last_fullsized":"2023-11-05T08:32:51.085658+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"103'31","reported_seq":222,"reported_epoch":108,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:34:02.250234+0000","last_change":"2023-11-05T08:32:54.139002+0000","last_active":"2023-11-05T08:34:02.250234+0000","last_peered":"2023-11-05T08:34:02.250234+0000","last_clean":"2023-11-05T08:32:49.088591+0000","last_became_active":"2023-11-05T08:32:54.139002+0000","last_became_peered":"2023-11-05T08:32:54.139002+0000","last_unstale":"2023-11-05T08:34:02.250234+0000","last_undegraded":"2023-11-05T08:32:54.118261+0000","last_fullsized":"2023-11-05T08:32:54.117408+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":34,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.020413489999999999,"stat_sum":{"num_bytes":5988352,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":12,"num_read_kb":2382,"num_write":18,"num_write_kb":5848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,4,13,7,2147483647,6],"acting":[15,8,4,13,7,2147483647,6],"avail_no_missing":["15(0)","4(2)","6(6)","7(4)","8(1)","13(3)"],"object_location_counts":[{"shards":"4(2),6(6),7(4),8(1),13(3),15(0)","objects":5}],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.3","version":"97'17","reported_seq":359,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-11-05T08:33:57.794215+0000","last_change":"2023-11-05T08:33:54.770853+0000","last_active":"2023-11-05T08:33:57.794215+0000","last_peered":"2023-11-05T08:33:57.794215+0000","last_clean":"2023-11-05T08:33:57.794215+0000","last_became_active":"2023-11-05T08:33:50.683698+0000","last_became_peered":"2023-11-05T08:33:50.683698+0000","last_unstale":"2023-11-05T08:33:57.794215+0000","last_undegraded":"2023-11-05T08:33:57.794215+0000","last_fullsized":"2023-11-05T08:33:57.794215+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'17","last_scrub_stamp":"2023-11-05T08:33:53.073917+0000","last_deep_scrub":"97'17","last_deep_scrub_stamp":"2023-11-05T08:33:53.073917+0000","last_clean_scrub_stamp":"2023-11-05T08:33:53.073917+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:58.526902+0000","scrub_duration":0.0068216960000000004,"objects_trimmed":0,"snaptrim_duration":0.018960549,"stat_sum":{"num_bytes":524288,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":1537,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,4,15,2,5,3],"acting":[13,7,4,15,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"}]},{"pgid":"2.2","version":"0'0","reported_seq":104,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-05T08:34:00.798911+0000","last_change":"2023-11-05T08:33:53.537740+0000","last_active":"2023-11-05T08:34:00.798911+0000","last_peered":"2023-11-05T08:34:00.798911+0000","last_clean":"2023-11-05T08:34:00.798911+0000","last_became_active":"2023-11-05T08:33:47.679907+0000","last_became_peered":"2023-11-05T08:33:47.679907+0000","last_unstale":"2023-11-05T08:34:00.798911+0000","last_undegraded":"2023-11-05T08:34:00.798911+0000","last_fullsized":"2023-11-05T08:34:00.798911+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:33:53.537442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:33:53.537442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:35:15.667752+0000","scrub_duration":0.003485906,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3],"acting":[12,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.0","version":"33'9","reported_seq":342,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-05T08:34:01.808241+0000","last_change":"2023-11-05T08:34:01.808699+0000","last_active":"2023-11-05T08:34:01.808241+0000","last_peered":"2023-11-05T08:34:01.808241+0000","last_clean":"2023-11-05T08:34:01.808241+0000","last_became_active":"2023-11-05T08:32:54.144790+0000","last_became_peered":"2023-11-05T08:32:54.144790+0000","last_unstale":"2023-11-05T08:34:01.808241+0000","last_undegraded":"2023-11-05T08:34:01.808241+0000","last_fullsized":"2023-11-05T08:34:01.808241+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'9","last_scrub_stamp":"2023-11-05T08:33:03.810524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:03.810524+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:07.644773+0000","scrub_duration":0.0066041449999999996,"objects_trimmed":0,"snaptrim_duration":0.00083072500000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,15,1,5,6,4],"acting":[14,2,15,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":87,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-11-05T08:33:56.832312+0000","last_change":"2023-11-05T08:32:31.941357+0000","last_active":"2023-11-05T08:33:56.832312+0000","last_peered":"2023-11-05T08:33:56.832312+0000","last_clean":"2023-11-05T08:33:56.832312+0000","last_became_active":"2023-11-05T08:32:31.940564+0000","last_became_peered":"2023-11-05T08:32:31.940564+0000","last_unstale":"2023-11-05T08:33:56.832312+0000","last_undegraded":"2023-11-05T08:33:56.832312+0000","last_fullsized":"2023-11-05T08:33:56.832312+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:58.168161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.1","version":"106'40","reported_seq":385,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-05T08:34:01.927993+0000","last_change":"2023-11-05T08:34:01.928527+0000","last_active":"2023-11-05T08:34:01.927993+0000","last_peered":"2023-11-05T08:34:01.927993+0000","last_clean":"2023-11-05T08:34:01.927993+0000","last_became_active":"2023-11-05T08:32:54.144895+0000","last_became_peered":"2023-11-05T08:32:54.144895+0000","last_unstale":"2023-11-05T08:34:01.927993+0000","last_undegraded":"2023-11-05T08:34:01.927993+0000","last_fullsized":"2023-11-05T08:34:01.927993+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'11","last_scrub_stamp":"2023-11-05T08:33:04.003557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:04.003557+0000","objects_scrubbed":3,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:15.185676+0000","scrub_duration":0.010761797,"objects_trimmed":0,"snaptrim_duration":0.00073924699999999995,"stat_sum":{"num_bytes":1949696,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":772,"num_write":7,"num_write_kb":2480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,5,10,2,13],"acting":[4,11,9,5,10,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":96,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-11-05T08:33:59.863171+0000","last_change":"2023-11-05T08:33:47.700202+0000","last_active":"2023-11-05T08:33:59.863171+0000","last_peered":"2023-11-05T08:33:59.863171+0000","last_clean":"2023-11-05T08:33:59.863171+0000","last_became_active":"2023-11-05T08:33:47.699438+0000","last_became_peered":"2023-11-05T08:33:47.699438+0000","last_unstale":"2023-11-05T08:33:59.863171+0000","last_undegraded":"2023-11-05T08:33:59.863171+0000","last_fullsized":"2023-11-05T08:33:59.863171+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"102'30","reported_seq":374,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-11-05T08:33:57.793872+0000","last_change":"2023-11-05T08:33:54.764012+0000","last_active":"2023-11-05T08:33:57.793872+0000","last_peered":"2023-11-05T08:33:57.793872+0000","last_clean":"2023-11-05T08:33:57.793872+0000","last_became_active":"2023-11-05T08:33:50.678084+0000","last_became_peered":"2023-11-05T08:33:50.678084+0000","last_unstale":"2023-11-05T08:33:57.793872+0000","last_undegraded":"2023-11-05T08:33:57.793872+0000","last_fullsized":"2023-11-05T08:33:57.793872+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'25","last_scrub_stamp":"2023-11-05T08:33:52.100890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:52.100890+0000","objects_scrubbed":1,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:35:04.875042+0000","scrub_duration":0.007958952,"objects_trimmed":0,"snaptrim_duration":0.012335969,"stat_sum":{"num_bytes":3670016,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,3,6,15,1,11],"acting":[13,2,3,6,15,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"}]},{"pgid":"2.3","version":"0'0","reported_seq":99,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-05T08:34:01.452342+0000","last_change":"2023-11-05T08:34:00.133458+0000","last_active":"2023-11-05T08:34:01.452342+0000","last_peered":"2023-11-05T08:34:01.452342+0000","last_clean":"2023-11-05T08:34:01.452342+0000","last_became_active":"2023-11-05T08:32:31.939705+0000","last_became_peered":"2023-11-05T08:32:31.939705+0000","last_unstale":"2023-11-05T08:34:01.452342+0000","last_undegraded":"2023-11-05T08:34:01.452342+0000","last_fullsized":"2023-11-05T08:34:01.452342+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:34:00.133184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:34:00.133184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:35:05.010050+0000","scrub_duration":0.0035944599999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"25'157","reported_seq":294,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-11-05T08:33:56.833280+0000","last_change":"2023-11-05T08:33:41.087862+0000","last_active":"2023-11-05T08:33:56.833280+0000","last_peered":"2023-11-05T08:33:56.833280+0000","last_clean":"2023-11-05T08:33:56.833280+0000","last_became_active":"2023-11-05T08:32:28.758834+0000","last_became_peered":"2023-11-05T08:32:28.758834+0000","last_unstale":"2023-11-05T08:33:56.833280+0000","last_undegraded":"2023-11-05T08:33:56.833280+0000","last_fullsized":"2023-11-05T08:33:56.833280+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'157","last_scrub_stamp":"2023-11-05T08:33:41.087696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:27.693099+0000","last_clean_scrub_stamp":"2023-11-05T08:33:41.087696+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:35:09.900066+0000","scrub_duration":0.0045831459999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.8","version":"108'34","reported_seq":222,"reported_epoch":108,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:34:02.300393+0000","last_change":"2023-11-05T08:32:54.151946+0000","last_active":"2023-11-05T08:34:02.300393+0000","last_peered":"2023-11-05T08:34:02.300393+0000","last_clean":"2023-11-05T08:32:49.090022+0000","last_became_active":"2023-11-05T08:32:54.151946+0000","last_became_peered":"2023-11-05T08:32:54.151946+0000","last_unstale":"2023-11-05T08:34:02.300393+0000","last_undegraded":"2023-11-05T08:32:54.119646+0000","last_fullsized":"2023-11-05T08:32:54.118811+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":34,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.013731910999999999,"stat_sum":{"num_bytes":4816896,"num_objects":6,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":10,"num_read_kb":1684,"num_write":27,"num_write_kb":4144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2147483647,2,1,14,15,6],"acting":[5,2147483647,2,1,14,15,6],"avail_no_missing":["5(0)","1(3)","2(2)","6(6)","14(4)","15(5)"],"object_location_counts":[{"shards":"1(3),2(2),5(0),6(6),14(4),15(5)","objects":6}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.9","version":"103'64","reported_seq":315,"reported_epoch":108,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:34:01.843421+0000","last_change":"2023-11-05T08:33:50.691854+0000","last_active":"2023-11-05T08:34:01.843421+0000","last_peered":"2023-11-05T08:34:01.843421+0000","last_clean":"2023-11-05T08:32:49.072716+0000","last_became_active":"2023-11-05T08:33:50.691854+0000","last_became_peered":"2023-11-05T08:33:50.691854+0000","last_unstale":"2023-11-05T08:34:01.843421+0000","last_undegraded":"2023-11-05T08:33:50.658262+0000","last_fullsized":"2023-11-05T08:33:50.657391+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":34,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015323719999999999,"stat_sum":{"num_bytes":14761984,"num_objects":13,"num_object_clones":9,"num_object_copies":91,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":44,"num_read_kb":36208,"num_write":100,"num_write_kb":14416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":10584064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,2147483647,3,15,8,10],"acting":[14,1,2147483647,3,15,8,10],"avail_no_missing":["14(0)","1(1)","3(3)","8(5)","10(6)","15(4)"],"object_location_counts":[{"shards":"1(1),3(3),8(5),10(6),14(0),15(4)","objects":13}],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.a","version":"108'47","reported_seq":407,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-05T08:34:01.823848+0000","last_change":"2023-11-05T08:34:01.818604+0000","last_active":"2023-11-05T08:34:01.823848+0000","last_peered":"2023-11-05T08:34:01.823848+0000","last_clean":"2023-11-05T08:34:01.823848+0000","last_became_active":"2023-11-05T08:33:50.684400+0000","last_became_peered":"2023-11-05T08:33:50.684400+0000","last_unstale":"2023-11-05T08:34:01.823848+0000","last_undegraded":"2023-11-05T08:34:01.823848+0000","last_fullsized":"2023-11-05T08:34:01.823848+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'44","last_scrub_stamp":"2023-11-05T08:34:01.079975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:34:01.079975+0000","objects_scrubbed":3,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:35:14.986189+0000","scrub_duration":0.0091389039999999998,"objects_trimmed":0,"snaptrim_duration":0.0080629699999999992,"stat_sum":{"num_bytes":3538944,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":2,"num_read":25,"num_read_kb":17912,"num_write":36,"num_write_kb":3696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":6406144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,9,14,6,11],"acting":[3,7,1,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.b","version":"108'58","reported_seq":253,"reported_epoch":108,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:34:02.347090+0000","last_change":"2023-11-05T08:32:51.095575+0000","last_active":"2023-11-05T08:34:02.347090+0000","last_peered":"2023-11-05T08:34:02.347090+0000","last_clean":"2023-11-05T08:32:49.072970+0000","last_became_active":"2023-11-05T08:32:51.095575+0000","last_became_peered":"2023-11-05T08:32:51.095575+0000","last_unstale":"2023-11-05T08:34:02.347090+0000","last_undegraded":"2023-11-05T08:32:51.085902+0000","last_fullsized":"2023-11-05T08:32:51.084936+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":34,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:12.100399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021279290000000002,"stat_sum":{"num_bytes":12435456,"num_objects":13,"num_object_clones":9,"num_object_copies":91,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":38,"num_read_kb":17439,"num_write":64,"num_write_kb":12768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2147483647,1,12,6,7,9],"acting":[4,2147483647,1,12,6,7,9],"avail_no_missing":["4(0)","1(2)","6(4)","7(5)","9(6)","12(3)"],"object_location_counts":[{"shards":"1(2),4(0),6(4),7(5),9(6),12(3)","objects":13}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.c","version":"79'6","reported_seq":186,"reported_epoch":108,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:34:01.820645+0000","last_change":"2023-11-05T08:32:51.099852+0000","last_active":"2023-11-05T08:34:01.820645+0000","last_peered":"2023-11-05T08:34:01.820645+0000","last_clean":"2023-11-05T08:32:49.087645+0000","last_became_active":"2023-11-05T08:32:51.099852+0000","last_became_peered":"2023-11-05T08:32:51.099852+0000","last_unstale":"2023-11-05T08:34:01.820645+0000","last_undegraded":"2023-11-05T08:32:51.089699+0000","last_fullsized":"2023-11-05T08:32:51.089041+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":34,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015630430000000001,"stat_sum":{"num_bytes":999424,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":7,"num_read_kb":914,"num_write":4,"num_write_kb":976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,2147483647,10,6,12,5],"acting":[11,8,2147483647,10,6,12,5],"avail_no_missing":["11(0)","5(6)","6(4)","8(1)","10(3)","12(5)"],"object_location_counts":[{"shards":"5(6),6(4),8(1),10(3),11(0),12(5)","objects":3}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.d","version":"103'36","reported_seq":257,"reported_epoch":108,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:34:01.810896+0000","last_change":"2023-11-05T08:33:50.690639+0000","last_active":"2023-11-05T08:34:01.810896+0000","last_peered":"2023-11-05T08:34:01.810896+0000","last_clean":"2023-11-05T08:32:49.088145+0000","last_became_active":"2023-11-05T08:33:50.690639+0000","last_became_peered":"2023-11-05T08:33:50.690639+0000","last_unstale":"2023-11-05T08:34:01.810896+0000","last_undegraded":"2023-11-05T08:33:50.664683+0000","last_fullsized":"2023-11-05T08:33:50.663817+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":34,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.024583713,"stat_sum":{"num_bytes":5013504,"num_objects":7,"num_object_clones":5,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":6009,"num_write":23,"num_write_kb":3536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3112960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,2147483647,8,4],"acting":[7,3,14,10,2147483647,8,4],"avail_no_missing":["7(0)","3(1)","4(6)","8(5)","10(3)","14(2)"],"object_location_counts":[{"shards":"3(1),4(6),7(0),8(5),10(3),14(2)","objects":7}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.e","version":"108'128","reported_seq":451,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-05T08:34:01.821498+0000","last_change":"2023-11-05T08:34:01.821498+0000","last_active":"2023-11-05T08:34:01.821498+0000","last_peered":"2023-11-05T08:34:01.821498+0000","last_clean":"2023-11-05T08:34:01.821498+0000","last_became_active":"2023-11-05T08:32:44.949185+0000","last_became_peered":"2023-11-05T08:32:44.949185+0000","last_unstale":"2023-11-05T08:34:01.821498+0000","last_undegraded":"2023-11-05T08:34:01.821498+0000","last_fullsized":"2023-11-05T08:34:01.821498+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'96","last_scrub_stamp":"2023-11-05T08:33:48.102281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:48.102281+0000","objects_scrubbed":12,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:35:04.316205+0000","scrub_duration":0.014167007000000001,"objects_trimmed":0,"snaptrim_duration":0.011717086999999999,"stat_sum":{"num_bytes":7028736,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":12450,"num_write":75,"num_write_kb":13632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1,11,8,6,12],"acting":[2,9,1,11,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.f","version":"108'85","reported_seq":284,"reported_epoch":108,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:34:01.938307+0000","last_change":"2023-11-05T08:32:51.104425+0000","last_active":"2023-11-05T08:34:01.938307+0000","last_peered":"2023-11-05T08:34:01.938307+0000","last_clean":"2023-11-05T08:32:49.098585+0000","last_became_active":"2023-11-05T08:32:51.104425+0000","last_became_peered":"2023-11-05T08:32:51.104425+0000","last_unstale":"2023-11-05T08:34:01.938307+0000","last_undegraded":"2023-11-05T08:32:51.091474+0000","last_fullsized":"2023-11-05T08:32:51.089784+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":34,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012093372,"stat_sum":{"num_bytes":22069248,"num_objects":20,"num_object_clones":16,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":50,"num_read_kb":25938,"num_write":122,"num_write_kb":23184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,14,13,8,10,2147483647],"acting":[11,2,14,13,8,10,2147483647],"avail_no_missing":["11(0)","2(1)","8(4)","10(5)","13(3)","14(2)"],"object_location_counts":[{"shards":"2(1),8(4),10(5),11(0),13(3),14(2)","objects":20}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.10","version":"103'68","reported_seq":332,"reported_epoch":104,"state":"active+clean+remapped","last_fresh":"2023-11-05T08:33:57.793801+0000","last_change":"2023-11-05T08:33:54.758708+0000","last_active":"2023-11-05T08:33:57.793801+0000","last_peered":"2023-11-05T08:33:57.793801+0000","last_clean":"2023-11-05T08:33:57.793801+0000","last_became_active":"2023-11-05T08:32:59.210399+0000","last_became_peered":"2023-11-05T08:32:59.210399+0000","last_unstale":"2023-11-05T08:33:57.793801+0000","last_undegraded":"2023-11-05T08:33:57.793801+0000","last_fullsized":"2023-11-05T08:33:57.793801+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":5,"last_scrub":"33'9","last_scrub_stamp":"2023-11-05T08:33:00.792248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:00.792248+0000","objects_scrubbed":3,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:19.462036+0000","scrub_duration":0.0073349119999999999,"objects_trimmed":0,"snaptrim_duration":0.017308477999999999,"stat_sum":{"num_bytes":6897664,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":5385,"num_write":48,"num_write_kb":8880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,7,4,8,2147483647,11],"acting":[13,6,7,4,8,6,11],"avail_no_missing":["13(0)","4(3)","6(1)","6(5)","7(2)","8(4)","11(6)"],"object_location_counts":[{"shards":"4(3),6(1),6(5),7(2),8(4),11(6),13(0)","objects":4}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"}]},{"pgid":"3.11","version":"97'31","reported_seq":382,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-05T08:34:01.802710+0000","last_change":"2023-11-05T08:34:01.803250+0000","last_active":"2023-11-05T08:34:01.802710+0000","last_peered":"2023-11-05T08:34:01.802710+0000","last_clean":"2023-11-05T08:34:01.802710+0000","last_became_active":"2023-11-05T08:32:59.214193+0000","last_became_peered":"2023-11-05T08:32:59.214193+0000","last_unstale":"2023-11-05T08:34:01.802710+0000","last_undegraded":"2023-11-05T08:34:01.802710+0000","last_fullsized":"2023-11-05T08:34:01.802710+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":5,"last_scrub":"46'17","last_scrub_stamp":"2023-11-05T08:33:00.070510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:00.070510+0000","objects_scrubbed":3,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:03.268732+0000","scrub_duration":0.0081678800000000006,"objects_trimmed":0,"snaptrim_duration":0.00078513400000000005,"stat_sum":{"num_bytes":3981312,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":16565,"num_write":5,"num_write_kb":1936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,15,9,5,6,1],"acting":[10,11,15,9,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.12","version":"37'7","reported_seq":297,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-05T08:34:01.804941+0000","last_change":"2023-11-05T08:34:01.806121+0000","last_active":"2023-11-05T08:34:01.804941+0000","last_peered":"2023-11-05T08:34:01.804941+0000","last_clean":"2023-11-05T08:34:01.804941+0000","last_became_active":"2023-11-05T08:33:50.685746+0000","last_became_peered":"2023-11-05T08:33:50.685746+0000","last_unstale":"2023-11-05T08:34:01.804941+0000","last_undegraded":"2023-11-05T08:34:01.804941+0000","last_fullsized":"2023-11-05T08:34:01.804941+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"37'7","last_scrub_stamp":"2023-11-05T08:33:02.650387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:02.650387+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:19.568123+0000","scrub_duration":0.0067664470000000001,"objects_trimmed":0,"snaptrim_duration":0.002076825,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8,6,4,3,15],"acting":[9,7,8,6,4,3,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.13","version":"97'25","reported_seq":357,"reported_epoch":108,"state":"active+clean+remapped","last_fresh":"2023-11-05T08:34:01.821656+0000","last_change":"2023-11-05T08:34:01.822172+0000","last_active":"2023-11-05T08:34:01.821656+0000","last_peered":"2023-11-05T08:34:01.821656+0000","last_clean":"2023-11-05T08:34:01.821656+0000","last_became_active":"2023-11-05T08:32:59.212969+0000","last_became_peered":"2023-11-05T08:32:59.212969+0000","last_unstale":"2023-11-05T08:34:01.821656+0000","last_undegraded":"2023-11-05T08:34:01.821656+0000","last_fullsized":"2023-11-05T08:34:01.821656+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'14","last_scrub_stamp":"2023-11-05T08:33:02.644028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:02.644028+0000","objects_scrubbed":1,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:10.863657+0000","scrub_duration":0.007386386,"objects_trimmed":0,"snaptrim_duration":0.001657958,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,9,13,2147483647,4,10],"acting":[11,7,9,13,2,4,10],"avail_no_missing":["11(0)","2(4)","4(5)","7(1)","9(2)","10(6)","13(3)"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"49","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":115941376,"num_objects":109,"num_object_clones":61,"num_object_copies":763,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":71,"num_objects_misplaced":8,"num_objects_unfound":0,"num_objects_dirty":109,"num_whiteouts":4,"num_read":344,"num_read_kb":151968,"num_write":653,"num_write_kb":115800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":23068672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":185552896,"data_stored":291856384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":990,"ondisk_log_size":990,"up":182,"acting":182,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":15,"acting":15,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":13898,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":22,"seq":94489280533,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45884,"kb_used_data":14696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325956,"statfs":{"total":96636764160,"available":96589778944,"internally_reserved":0,"allocated":15048704,"data_stored":23455151,"data_compressed":164710,"data_compressed_allocated":372736,"data_compressed_original":1191936,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":1,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.256,"15min":1.256},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":3.4590000000000001,"5min":3.4590000000000001,"15min":3.4590000000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":1.341,"5min":1.341,"15min":1.341},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":2.7280000000000002,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":1.1479999999999999}]},{"osd":2,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":2.2080000000000002,"5min":2.2080000000000002,"15min":2.2080000000000002},"last":0.85699999999999998},{"interface":"front","average":{"1min":1.218,"5min":1.218,"15min":1.218},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":3.0139999999999998,"5min":3.0139999999999998,"15min":3.0139999999999998},"last":1.0720000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0449999999999999}]},{"osd":4,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.222,"15min":1.222},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":3.5979999999999999,"5min":3.5979999999999999,"15min":3.5979999999999999},"last":0.82799999999999996},{"interface":"front","average":{"1min":1.321,"5min":1.321,"15min":1.321},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":3.383,"5min":3.383,"15min":3.383},"last":0.94299999999999995}]},{"osd":5,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.466,"15min":1.466},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":3.5059999999999998,"5min":3.5059999999999998,"15min":3.5059999999999998},"last":0.89800000000000002},{"interface":"front","average":{"1min":1.5029999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":3.1579999999999999,"5min":3.1579999999999999,"15min":3.1579999999999999},"last":1.109}]},{"osd":6,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.625,"5min":1.625,"15min":1.625},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":3.4820000000000002,"5min":3.4820000000000002,"15min":3.4820000000000002},"last":0.998},{"interface":"front","average":{"1min":1.579,"5min":1.579,"15min":1.579},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":3.0059999999999998,"5min":3.0059999999999998,"15min":3.0059999999999998},"last":0.75900000000000001}]},{"osd":7,"last update":"Sun Nov 5 08:33:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"min":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"max":{"1min":2.165,"5min":2.165,"15min":2.165},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.54,"5min":1.54,"15min":1.54},"min":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"max":{"1min":2.2759999999999998,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":1.3400000000000001}]},{"osd":8,"last update":"Sun Nov 5 08:33:52 2023","interfaces":[{"interface":"back","average":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.6850000000000001},"min":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":3.5739999999999998,"5min":3.5739999999999998,"15min":3.5739999999999998},"last":1.212},{"interface":"front","average":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"min":{"1min":1.044,"5min":1.044,"15min":1.044},"max":{"1min":2.9569999999999999,"5min":2.9569999999999999,"15min":2.9569999999999999},"last":0.97499999999999998}]},{"osd":9,"last update":"Sun Nov 5 08:33:52 2023","interfaces":[{"interface":"back","average":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":1.7849999999999999},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":4.6520000000000001,"5min":4.6520000000000001,"15min":4.6520000000000001},"last":1.0940000000000001},{"interface":"front","average":{"1min":2.0720000000000001,"5min":2.0720000000000001,"15min":2.0720000000000001},"min":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"max":{"1min":4.7450000000000001,"5min":4.7450000000000001,"15min":4.7450000000000001},"last":1.129}]},{"osd":11,"last update":"Sun Nov 5 08:33:48 2023","interfaces":[{"interface":"back","average":{"1min":1.6619999999999999,"5min":1.6619999999999999,"15min":1.6619999999999999},"min":{"1min":1.2,"5min":1.2,"15min":1.2},"max":{"1min":2.5129999999999999,"5min":2.5129999999999999,"15min":2.5129999999999999},"last":1.3939999999999999},{"interface":"front","average":{"1min":1.7909999999999999,"5min":1.7909999999999999,"15min":1.7909999999999999},"min":{"1min":1.056,"5min":1.056,"15min":1.056},"max":{"1min":3.6619999999999999,"5min":3.6619999999999999,"15min":3.6619999999999999},"last":1.3129999999999999}]},{"osd":12,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.962,"5min":1.962,"15min":1.962},"min":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"max":{"1min":5.7270000000000003,"5min":5.7270000000000003,"15min":5.7270000000000003},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.9239999999999999,"5min":1.9239999999999999,"15min":1.9239999999999999},"min":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"max":{"1min":6.093,"5min":6.093,"15min":6.093},"last":1.2849999999999999}]},{"osd":14,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.9470000000000001,"5min":1.9470000000000001,"15min":1.9470000000000001},"min":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"max":{"1min":3.7050000000000001,"5min":3.7050000000000001,"15min":3.7050000000000001},"last":1.264},{"interface":"front","average":{"1min":1.982,"5min":1.982,"15min":1.982},"min":{"1min":1.099,"5min":1.099,"15min":1.099},"max":{"1min":3.2349999999999999,"5min":3.2349999999999999,"15min":3.2349999999999999},"last":1.0329999999999999}]},{"osd":15,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.9370000000000001,"5min":1.9370000000000001,"15min":1.9370000000000001},"min":{"1min":1.155,"5min":1.155,"15min":1.155},"max":{"1min":3.048,"5min":3.048,"15min":3.048},"last":1.242},{"interface":"front","average":{"1min":2.129,"5min":2.129,"15min":2.129},"min":{"1min":1.135,"5min":1.135,"15min":1.135},"max":{"1min":5.3920000000000003,"5min":5.3920000000000003,"15min":5.3920000000000003},"last":1.3620000000000001}]}]},{"osd":15,"up_from":21,"seq":90194313238,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37980,"kb_used_data":10956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94333860,"statfs":{"total":96636764160,"available":96597872640,"internally_reserved":0,"allocated":11218944,"data_stored":16767293,"data_compressed":165545,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,4,5,6,7,8,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":1,"last update":"Sun Nov 5 08:33:55 2023","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.139,"15min":1.139},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":3.0329999999999999,"5min":3.0329999999999999,"15min":3.0329999999999999},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":8.6170000000000009,"5min":8.6170000000000009,"15min":8.6170000000000009},"last":0.435}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.131}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96399999999999997}]},{"osd":4,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.125,"15min":1.125},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":2.4649999999999999,"5min":2.4649999999999999,"15min":2.4649999999999999},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":2.331,"5min":2.331,"15min":2.331},"last":0.67200000000000004}]},{"osd":5,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.113,"15min":1.113},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":2.9460000000000002,"5min":2.9460000000000002,"15min":2.9460000000000002},"last":0.48399999999999999},{"interface":"front","average":{"1min":1.452,"5min":1.452,"15min":1.452},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":3.8849999999999998,"5min":3.8849999999999998,"15min":3.8849999999999998},"last":1.042}]},{"osd":6,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.119,"15min":1.119},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":2.4769999999999999,"5min":2.4769999999999999,"15min":2.4769999999999999},"last":0.84199999999999997},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":3.0880000000000001,"5min":3.0880000000000001,"15min":3.0880000000000001},"last":0.88800000000000001}]},{"osd":7,"last update":"Sun Nov 5 08:33:44 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.169,"15min":1.169},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.996,"5min":2.996,"15min":2.996},"last":1.0600000000000001},{"interface":"front","average":{"1min":1.389,"5min":1.389,"15min":1.389},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":5.444,"5min":5.444,"15min":5.444},"last":0.54300000000000004}]},{"osd":8,"last update":"Sun Nov 5 08:33:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":3.7389999999999999,"5min":3.7389999999999999,"15min":3.7389999999999999},"last":1.1970000000000001},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":4.2489999999999997,"5min":4.2489999999999997,"15min":4.2489999999999997},"last":0.97999999999999998}]},{"osd":9,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.5900000000000001,"5min":1.5900000000000001,"15min":1.5900000000000001},"min":{"1min":1.044,"5min":1.044,"15min":1.044},"max":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"last":1.2330000000000001},{"interface":"front","average":{"1min":1.883,"5min":1.883,"15min":1.883},"min":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"max":{"1min":13.029999999999999,"5min":13.029999999999999,"15min":13.029999999999999},"last":0.77600000000000002}]},{"osd":11,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"min":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":2.9590000000000001,"5min":2.9590000000000001,"15min":2.9590000000000001},"last":1.1599999999999999},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"min":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"max":{"1min":2.6469999999999998,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":0.72699999999999998}]},{"osd":13,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.591,"5min":1.591,"15min":1.591},"min":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"max":{"1min":3.3580000000000001,"5min":3.3580000000000001,"15min":3.3580000000000001},"last":1.2709999999999999},{"interface":"front","average":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":1.7849999999999999},"min":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"max":{"1min":5.1379999999999999,"5min":5.1379999999999999,"15min":5.1379999999999999},"last":1.018}]},{"osd":14,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.718,"5min":1.718,"15min":1.718},"min":{"1min":0.999,"5min":0.999,"15min":0.999},"max":{"1min":5.7720000000000002,"5min":5.7720000000000002,"15min":5.7720000000000002},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.696,"5min":1.696,"15min":1.696},"min":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"max":{"1min":7.5039999999999996,"5min":7.5039999999999996,"15min":7.5039999999999996},"last":0.94299999999999995}]}]},{"osd":14,"up_from":22,"seq":94489280533,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40056,"kb_used_data":13036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331784,"statfs":{"total":96636764160,"available":96595746816,"internally_reserved":0,"allocated":13348864,"data_stored":25578375,"data_compressed":165545,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":1,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":1.45},{"interface":"front","average":{"1min":1.131,"5min":1.131,"15min":1.131},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.732,"5min":1.732,"15min":1.732},"last":0.78500000000000003}]},{"osd":2,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.86099999999999999},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.839,"5min":1.839,"15min":1.839},"last":1.179}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3149999999999999}]},{"osd":4,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":1.0640000000000001},{"interface":"front","average":{"1min":0.997,"5min":0.997,"15min":0.997},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.391,"5min":1.391,"15min":1.391},"last":1.347}]},{"osd":5,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.101,"5min":1.101,"15min":1.101},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.605,"5min":1.605,"15min":1.605},"last":0.92400000000000004}]},{"osd":6,"last update":"Sun Nov 5 08:33:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.575,"5min":1.575,"15min":1.575},"last":0.59799999999999998},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.9610000000000001,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.97299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.133}]},{"osd":8,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.7050000000000001,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.22,"5min":1.22,"15min":1.22},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":1.242}]},{"osd":10,"last update":"Sun Nov 5 08:33:44 2023","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.29,"15min":1.29},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.802,"5min":1.802,"15min":1.802},"last":1.028},{"interface":"front","average":{"1min":1.325,"5min":1.325,"15min":1.325},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.8049999999999999,"5min":1.8049999999999999,"15min":1.8049999999999999},"last":1.528}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.429}]},{"osd":12,"last update":"Sun Nov 5 08:33:53 2023","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":1.4430000000000001},"min":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"max":{"1min":2.2389999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.153},{"interface":"front","average":{"1min":1.544,"5min":1.544,"15min":1.544},"min":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"max":{"1min":2.1320000000000001,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":1.478}]},{"osd":13,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.5900000000000001,"5min":1.5900000000000001,"15min":1.5900000000000001},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":2.4590000000000001,"5min":2.4590000000000001,"15min":2.4590000000000001},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"min":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":2.028,"5min":2.028,"15min":2.028},"last":1.214}]},{"osd":15,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":2.2429999999999999,"5min":2.2429999999999999,"15min":2.2429999999999999},"last":1.4139999999999999},{"interface":"front","average":{"1min":1.6399999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":2.3559999999999999,"5min":2.3559999999999999,"15min":2.3559999999999999},"last":1.548}]}]},{"osd":0,"up_from":21,"seq":90194313223,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":4392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":4497408,"data_stored":4339168,"data_compressed":21230,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,9,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]}]},{"osd":3,"up_from":93,"seq":399431958532,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38104,"kb_used_data":9924,"kb_used_omap":6,"kb_used_meta":28153,"kb_avail":94333736,"statfs":{"total":96636764160,"available":96597745664,"internally_reserved":0,"allocated":10162176,"data_stored":12466071,"data_compressed":161651,"data_compressed_allocated":348160,"data_compressed_original":696320,"omap_allocated":6448,"internal_metadata":28829392},"hb_peers":[2,4,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2589999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91100000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2250000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4350000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4790000000000001}]}]},{"osd":6,"up_from":21,"seq":90194313237,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47140,"kb_used_data":15960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324700,"statfs":{"total":96636764160,"available":96588492800,"internally_reserved":0,"allocated":16343040,"data_stored":21761825,"data_compressed":163600,"data_compressed_allocated":352256,"data_compressed_original":704512,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,3,4,5,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.02,"15min":1.02},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":0.84999999999999998},{"interface":"front","average":{"1min":1.153,"5min":1.153,"15min":1.153},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.899,"5min":1.899,"15min":1.899},"last":1.0569999999999999}]},{"osd":2,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.76000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4370000000000001}]},{"osd":4,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.038,"15min":1.038},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":0.874},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.617,"5min":1.617,"15min":1.617},"last":0.64200000000000002}]},{"osd":5,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.532,"5min":1.532,"15min":1.532},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.212,"5min":1.212,"15min":1.212},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.577,"5min":1.577,"15min":1.577},"last":0.92100000000000004}]},{"osd":7,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.72,"5min":1.72,"15min":1.72},"last":0.95399999999999996},{"interface":"front","average":{"1min":1.069,"5min":1.069,"15min":1.069},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.6859999999999999,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":1.292}]},{"osd":8,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.958,"5min":1.958,"15min":1.958},"last":0.59999999999999998},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.772,"5min":1.772,"15min":1.772},"last":1.724}]},{"osd":9,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.484},{"interface":"front","average":{"1min":1.306,"5min":1.306,"15min":1.306},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.7749999999999999,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":1.26}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3999999999999999}]},{"osd":11,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.379,"15min":1.379},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":2.0579999999999998,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":1.093},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":2.1930000000000001,"5min":2.1930000000000001,"15min":2.1930000000000001},"last":1.5249999999999999}]},{"osd":12,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.548,"5min":1.548,"15min":1.548},"min":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":2.1139999999999999,"5min":2.1139999999999999,"15min":2.1139999999999999},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.841,"5min":1.841,"15min":1.841},"last":1.1379999999999999}]},{"osd":13,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":1.6930000000000001,"5min":1.6930000000000001,"15min":1.6930000000000001},"min":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"max":{"1min":2.2759999999999998,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":1.589},{"interface":"front","average":{"1min":1.532,"5min":1.532,"15min":1.532},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.9710000000000001,"5min":1.9710000000000001,"15min":1.9710000000000001},"last":1.5660000000000001}]},{"osd":14,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":2.319,"5min":2.319,"15min":2.319},"last":1.327},{"interface":"front","average":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":2.4870000000000001,"5min":2.4870000000000001,"15min":2.4870000000000001},"last":1.6559999999999999}]},{"osd":15,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.679,"5min":1.679,"15min":1.679},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.4009999999999998,"5min":2.4009999999999998,"15min":2.4009999999999998},"last":1.1970000000000001},{"interface":"front","average":{"1min":1.482,"5min":1.482,"15min":1.482},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"last":1.623}]}]},{"osd":1,"up_from":22,"seq":94489280533,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46080,"kb_used_data":14900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325760,"statfs":{"total":96636764160,"available":96589578240,"internally_reserved":0,"allocated":15257600,"data_stored":21203847,"data_compressed":165545,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,5,6,7,8,9,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":2,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.272,"5min":1.272,"15min":1.272},"last":0.84899999999999998},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.1240000000000001,"5min":2.1240000000000001,"15min":2.1240000000000001},"last":0.77100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.873}]},{"osd":4,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.6100000000000001,"5min":1.6100000000000001,"15min":1.6100000000000001},"last":1.117}]},{"osd":5,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.071,"15min":1.071},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":2.6110000000000002,"5min":2.6110000000000002,"15min":2.6110000000000002},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.7310000000000001,"5min":1.7310000000000001,"15min":1.7310000000000001},"last":1.349}]},{"osd":6,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.135,"15min":1.135},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.61599999999999999},{"interface":"front","average":{"1min":1.246,"5min":1.246,"15min":1.246},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.1759999999999999}]},{"osd":7,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.282,"15min":1.282},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.708,"5min":1.708,"15min":1.708},"last":0.82599999999999996},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":2.3090000000000002,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":1.254}]},{"osd":8,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.288,"15min":1.288},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":2.5630000000000002,"5min":2.5630000000000002,"15min":2.5630000000000002},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.238,"5min":1.238,"15min":1.238},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":1.397}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5089999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.46}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3109999999999999}]},{"osd":14,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"min":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"max":{"1min":2.2320000000000002,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":1.4810000000000001},{"interface":"front","average":{"1min":1.498,"5min":1.498,"15min":1.498},"min":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"max":{"1min":2.7610000000000001,"5min":2.7610000000000001,"15min":2.7610000000000001},"last":1.038}]},{"osd":15,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.5640000000000001,"5min":1.5640000000000001,"15min":1.5640000000000001},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":2.7250000000000001,"5min":2.7250000000000001,"15min":2.7250000000000001},"last":1.55},{"interface":"front","average":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"min":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":1.655}]}]},{"osd":2,"up_from":22,"seq":94489280533,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42868,"kb_used_data":11688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328972,"statfs":{"total":96636764160,"available":96592867328,"internally_reserved":0,"allocated":11968512,"data_stored":20028839,"data_compressed":172494,"data_compressed_allocated":389120,"data_compressed_original":1224704,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,3,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.5529999999999999,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.482,"5min":1.482,"15min":1.482},"last":0.94699999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94199999999999995}]},{"osd":5,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.5029999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.79800000000000004}]},{"osd":6,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.5},{"interface":"front","average":{"1min":1.077,"5min":1.077,"15min":1.077},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":0.32700000000000001}]},{"osd":7,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.167,"15min":1.167},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":2.0880000000000001,"5min":2.0880000000000001,"15min":2.0880000000000001},"last":0.629},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.744,"5min":1.744,"15min":1.744},"last":0.81499999999999995}]},{"osd":8,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.831,"5min":1.831,"15min":1.831},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.129,"5min":1.129,"15min":1.129},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":2.1859999999999999,"5min":2.1859999999999999,"15min":2.1859999999999999},"last":0.40500000000000003}]},{"osd":9,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.7969999999999999,"5min":1.7969999999999999,"15min":1.7969999999999999},"last":0.84499999999999997},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":2.27,"5min":2.27,"15min":2.27},"last":0.70499999999999996}]},{"osd":10,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":2.976,"5min":2.976,"15min":2.976},"last":0.44800000000000001},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":2.5739999999999998,"5min":2.5739999999999998,"15min":2.5739999999999998},"last":0.91000000000000003}]},{"osd":11,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.9009999999999998,"5min":2.9009999999999998,"15min":2.9009999999999998},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":2.6859999999999999,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.73499999999999999}]},{"osd":12,"last update":"Sun Nov 5 08:33:45 2023","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.2,"15min":1.2},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.75600000000000001}]}]},{"osd":4,"up_from":22,"seq":94489280533,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44584,"kb_used_data":13400,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327256,"statfs":{"total":96636764160,"available":96591110144,"internally_reserved":0,"allocated":13721600,"data_stored":19343677,"data_compressed":165545,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,5,6,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":1,"last update":"Sun Nov 5 08:33:43 2023","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.6659999999999999,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":0.995}]},{"osd":2,"last update":"Sun Nov 5 08:33:43 2023","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":0.82399999999999995},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.9850000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":1.4790000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2849999999999999}]},{"osd":5,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.0299999999999998,"5min":2.0299999999999998,"15min":2.0299999999999998},"last":1.24},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.7,"5min":1.7,"15min":1.7},"last":1.4390000000000001}]},{"osd":6,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":0.995,"15min":0.995},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.7290000000000001,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":1.238,"5min":1.238,"15min":1.238},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":2.2789999999999999,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":1.101}]},{"osd":7,"last update":"Sun Nov 5 08:33:43 2023","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.889,"5min":1.889,"15min":1.889},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":1.514}]},{"osd":8,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.171,"15min":1.171},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":2.2490000000000001,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.7789999999999999,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":1.1839999999999999}]},{"osd":9,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":2.5249999999999999,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":1.359},{"interface":"front","average":{"1min":1.468,"5min":1.468,"15min":1.468},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.3290000000000002,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":1.647}]},{"osd":10,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":2.0459999999999998,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.401},{"interface":"front","average":{"1min":1.508,"5min":1.508,"15min":1.508},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.3919999999999999,"5min":2.3919999999999999,"15min":2.3919999999999999},"last":1.569}]},{"osd":11,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.431,"15min":1.431},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.1280000000000001,"5min":2.1280000000000001,"15min":2.1280000000000001},"last":1.3260000000000001},{"interface":"front","average":{"1min":1.3,"5min":1.3,"15min":1.3},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.9379999999999999,"5min":1.9379999999999999,"15min":1.9379999999999999},"last":2.0960000000000001}]},{"osd":12,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.3,"5min":1.3,"15min":1.3},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":2.161,"5min":2.161,"15min":2.161},"last":1.9219999999999999}]},{"osd":13,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.5640000000000001,"5min":1.5640000000000001,"15min":1.5640000000000001},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":2.3420000000000001,"5min":2.3420000000000001,"15min":2.3420000000000001},"last":2.036},{"interface":"front","average":{"1min":1.538,"5min":1.538,"15min":1.538},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":2.3140000000000001,"5min":2.3140000000000001,"15min":2.3140000000000001},"last":1.6890000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5109999999999999}]},{"osd":15,"last update":"Sun Nov 5 08:33:57 2023","interfaces":[{"interface":"back","average":{"1min":1.6990000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":2.5750000000000002,"5min":2.5750000000000002,"15min":2.5750000000000002},"last":1.6080000000000001},{"interface":"front","average":{"1min":1.6719999999999999,"5min":1.6719999999999999,"15min":1.6719999999999999},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":2.3940000000000001,"5min":2.3940000000000001,"15min":2.3940000000000001},"last":1.879}]}]},{"osd":5,"up_from":22,"seq":94489280533,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34200,"kb_used_data":7180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337640,"statfs":{"total":96636764160,"available":96601743360,"internally_reserved":0,"allocated":7352320,"data_stored":8196459,"data_compressed":163600,"data_compressed_allocated":352256,"data_compressed_original":704512,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,6,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":1,"last update":"Sun Nov 5 08:33:45 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.97599999999999998}]},{"osd":2,"last update":"Sun Nov 5 08:33:45 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.575,"5min":1.575,"15min":1.575},"last":1.3939999999999999},{"interface":"front","average":{"1min":1.04,"5min":1.04,"15min":1.04},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.546,"5min":1.546,"15min":1.546},"last":1.26}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":4,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.645,"5min":1.645,"15min":1.645},"last":1.831},{"interface":"front","average":{"1min":1.036,"5min":1.036,"15min":1.036},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.77,"5min":1.77,"15min":1.77},"last":1.47}]},{"osd":6,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":1.3620000000000001},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":1.512}]},{"osd":7,"last update":"Sun Nov 5 08:33:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.696,"5min":1.696,"15min":1.696},"last":1.3089999999999999},{"interface":"front","average":{"1min":1.177,"5min":1.177,"15min":1.177},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":1.212}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5760000000000001}]},{"osd":9,"last update":"Sun Nov 5 08:33:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.3,"15min":1.3},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.6990000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":1.4359999999999999},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"min":{"1min":0.88,"5min":0.88,"15min":0.88},"max":{"1min":1.8180000000000001,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":1.931}]},{"osd":10,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.7909999999999999},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":1.8799999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4990000000000001}]},{"osd":12,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":1.635,"15min":1.635},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":3.488,"5min":3.488,"15min":3.488},"last":2.0089999999999999},{"interface":"front","average":{"1min":1.46,"5min":1.46,"15min":1.46},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.1970000000000001,"5min":2.1970000000000001,"15min":2.1970000000000001},"last":1.9690000000000001}]},{"osd":13,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.6799999999999999,"5min":1.6799999999999999,"15min":1.6799999999999999},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":3.714,"5min":3.714,"15min":3.714},"last":2.1819999999999999},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.5649999999999999,"15min":1.5649999999999999},"min":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"max":{"1min":2.121,"5min":2.121,"15min":2.121},"last":2.0369999999999999}]},{"osd":14,"last update":"Sun Nov 5 08:33:45 2023","interfaces":[{"interface":"back","average":{"1min":1.718,"5min":1.718,"15min":1.718},"min":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"max":{"1min":2.1920000000000002,"5min":2.1920000000000002,"15min":2.1920000000000002},"last":2.1309999999999998},{"interface":"front","average":{"1min":1.647,"5min":1.647,"15min":1.647},"min":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"max":{"1min":2.2930000000000001,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":2.1059999999999999}]},{"osd":15,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.974,"5min":1.974,"15min":1.974},"last":1.637},{"interface":"front","average":{"1min":1.8140000000000001,"5min":1.8140000000000001,"15min":1.8140000000000001},"min":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"max":{"1min":3.5870000000000002,"5min":3.5870000000000002,"15min":3.5870000000000002},"last":2.0600000000000001}]}]},{"osd":7,"up_from":22,"seq":94489280534,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39656,"kb_used_data":12632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94332184,"statfs":{"total":96636764160,"available":96596156416,"internally_reserved":0,"allocated":12935168,"data_stored":17976199,"data_compressed":165545,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1930000000000001}]},{"osd":1,"last update":"Sun Nov 5 08:33:47 2023","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":2.1360000000000001,"5min":2.1360000000000001,"15min":2.1360000000000001},"last":1.258},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.85299999999999998}]},{"osd":2,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.401,"15min":1.401},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":1.889,"5min":1.889,"15min":1.889},"last":1.875},{"interface":"front","average":{"1min":1.238,"5min":1.238,"15min":1.238},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.663,"5min":1.663,"15min":1.663},"last":1.349}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4370000000000001}]},{"osd":4,"last update":"Sun Nov 5 08:33:30 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.095,"15min":1.095},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.7949999999999999,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":2.294,"5min":2.294,"15min":2.294},"last":1.0840000000000001}]},{"osd":5,"last update":"Sun Nov 5 08:33:30 2023","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.591,"5min":1.591,"15min":1.591},"last":1.5740000000000001},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.726,"5min":1.726,"15min":1.726},"last":1.302}]},{"osd":6,"last update":"Sun Nov 5 08:33:30 2023","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.232,"15min":1.232},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.6040000000000001,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":1.7689999999999999},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":2.0510000000000002,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.7170000000000001}]},{"osd":8,"last update":"Sun Nov 5 08:33:30 2023","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.381,"5min":1.381,"15min":1.381},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.48,"5min":2.48,"15min":2.48},"last":1.657}]},{"osd":9,"last update":"Sun Nov 5 08:33:47 2023","interfaces":[{"interface":"back","average":{"1min":1.6799999999999999,"5min":1.6799999999999999,"15min":1.6799999999999999},"min":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"max":{"1min":2.823,"5min":2.823,"15min":2.823},"last":2.0099999999999998},{"interface":"front","average":{"1min":1.48,"5min":1.48,"15min":1.48},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":2.3450000000000002,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":1.9350000000000001}]},{"osd":10,"last update":"Sun Nov 5 08:33:30 2023","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":1.478,"15min":1.478},"min":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"max":{"1min":1.931,"5min":1.931,"15min":1.931},"last":1.397},{"interface":"front","average":{"1min":1.671,"5min":1.671,"15min":1.671},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.5550000000000002,"5min":2.5550000000000002,"15min":2.5550000000000002},"last":2.1539999999999999}]},{"osd":11,"last update":"Sun Nov 5 08:33:47 2023","interfaces":[{"interface":"back","average":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":2.093,"5min":2.093,"15min":2.093},"last":1.6220000000000001},{"interface":"front","average":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":2.6600000000000001,"5min":2.6600000000000001,"15min":2.6600000000000001},"last":2.2069999999999999}]},{"osd":12,"last update":"Sun Nov 5 08:33:52 2023","interfaces":[{"interface":"back","average":{"1min":1.841,"5min":1.841,"15min":1.841},"min":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"max":{"1min":2.7250000000000001,"5min":2.7250000000000001,"15min":2.7250000000000001},"last":2.306},{"interface":"front","average":{"1min":1.639,"5min":1.639,"15min":1.639},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.3780000000000001,"5min":2.3780000000000001,"15min":2.3780000000000001},"last":2.1880000000000002}]},{"osd":13,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":1.667,"5min":1.667,"15min":1.667},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.589,"5min":2.589,"15min":2.589},"last":2.0430000000000001},{"interface":"front","average":{"1min":1.625,"5min":1.625,"15min":1.625},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":2.3050000000000002,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":1.9690000000000001}]},{"osd":14,"last update":"Sun Nov 5 08:33:30 2023","interfaces":[{"interface":"back","average":{"1min":1.75,"5min":1.75,"15min":1.75},"min":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":2.4910000000000001,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":2.1200000000000001},{"interface":"front","average":{"1min":1.7529999999999999,"5min":1.7529999999999999,"15min":1.7529999999999999},"min":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"max":{"1min":2.9750000000000001,"5min":2.9750000000000001,"15min":2.9750000000000001},"last":1.823}]},{"osd":15,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":1.7010000000000001,"5min":1.7010000000000001,"15min":1.7010000000000001},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":3.1760000000000002,"5min":3.1760000000000002,"15min":3.1760000000000002},"last":2.0859999999999999},{"interface":"front","average":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":2.73,"5min":2.73,"15min":2.73},"last":2.258}]}]},{"osd":8,"up_from":22,"seq":94489280533,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50488,"kb_used_data":19304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321352,"statfs":{"total":96636764160,"available":96585064448,"internally_reserved":0,"allocated":19767296,"data_stored":34982791,"data_compressed":165545,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":1,"last update":"Sun Nov 5 08:33:46 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":1.3140000000000001},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":1.22}]},{"osd":2,"last update":"Sun Nov 5 08:33:46 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":1.1559999999999999},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":1.4950000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4330000000000001}]},{"osd":4,"last update":"Sun Nov 5 08:33:46 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":1.585},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":1.524}]},{"osd":5,"last update":"Sun Nov 5 08:33:33 2023","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.786,"5min":1.786,"15min":1.786},"last":1.6950000000000001},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":1.8280000000000001}]},{"osd":6,"last update":"Sun Nov 5 08:33:33 2023","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.6799999999999999,"5min":1.6799999999999999,"15min":1.6799999999999999},"last":1.7889999999999999},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":1.875}]},{"osd":7,"last update":"Sun Nov 5 08:33:33 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.556,"5min":1.556,"15min":1.556},"last":1.7430000000000001},{"interface":"front","average":{"1min":1.214,"5min":1.214,"15min":1.214},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":2.157,"5min":2.157,"15min":2.157},"last":1.9510000000000001}]},{"osd":9,"last update":"Sun Nov 5 08:33:33 2023","interfaces":[{"interface":"back","average":{"1min":2.1509999999999998,"5min":2.1509999999999998,"15min":2.1509999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":19.849,"5min":19.849,"15min":19.849},"last":2.234},{"interface":"front","average":{"1min":2.2170000000000001,"5min":2.2170000000000001,"15min":2.2170000000000001},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":19.867999999999999,"5min":19.867999999999999,"15min":19.867999999999999},"last":1.917}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]},{"osd":11,"last update":"Sun Nov 5 08:33:33 2023","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.298,"15min":1.298},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":2.0710000000000002,"5min":2.0710000000000002,"15min":2.0710000000000002},"last":2.21},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":2.0960000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":13,"last update":"Sun Nov 5 08:33:33 2023","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":2.0670000000000002},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.8560000000000001,"5min":1.8560000000000001,"15min":1.8560000000000001},"last":2.0310000000000001}]},{"osd":14,"last update":"Sun Nov 5 08:33:33 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":2.226,"5min":2.226,"15min":2.226},"last":1.9930000000000001},{"interface":"front","average":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":1.4430000000000001},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.891,"5min":1.891,"15min":1.891},"last":2.266}]},{"osd":15,"last update":"Sun Nov 5 08:33:55 2023","interfaces":[{"interface":"back","average":{"1min":1.552,"5min":1.552,"15min":1.552},"min":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"max":{"1min":2.1629999999999998,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":2.1360000000000001},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.4490000000000001,"15min":1.4490000000000001},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.1970000000000001,"5min":2.1970000000000001,"15min":2.1970000000000001},"last":2.169}]}]},{"osd":9,"up_from":21,"seq":90194313237,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42728,"kb_used_data":11544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329112,"statfs":{"total":96636764160,"available":96593010688,"internally_reserved":0,"allocated":11821056,"data_stored":14945159,"data_compressed":165545,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,6,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":1,"last update":"Sun Nov 5 08:34:00 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.506,"5min":1.506,"15min":1.506},"last":0.75},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.734,"5min":1.734,"15min":1.734},"last":1.0640000000000001}]},{"osd":2,"last update":"Sun Nov 5 08:34:00 2023","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.81200000000000006},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.5620000000000001,"5min":1.5620000000000001,"15min":1.5620000000000001},"last":0.94499999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.23}]},{"osd":4,"last update":"Sun Nov 5 08:34:00 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.095,"15min":1.095},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.478,"5min":1.478,"15min":1.478},"last":0.86499999999999999},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.595,"5min":1.595,"15min":1.595},"last":1.52}]},{"osd":6,"last update":"Sun Nov 5 08:33:32 2023","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.101,"15min":1.101},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":1.641,"5min":1.641,"15min":1.641},"last":1.1399999999999999},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.486,"5min":1.486,"15min":1.486},"last":1.3740000000000001}]},{"osd":7,"last update":"Sun Nov 5 08:34:00 2023","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"min":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"max":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.482},{"interface":"front","average":{"1min":1.369,"5min":1.369,"15min":1.369},"min":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":1.925,"5min":1.925,"15min":1.925},"last":1.6599999999999999}]},{"osd":8,"last update":"Sun Nov 5 08:33:32 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.202,"15min":1.202},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":1.4179999999999999},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.944,"5min":1.944,"15min":1.944},"last":1.617}]},{"osd":10,"last update":"Sun Nov 5 08:33:32 2023","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.752,"5min":1.752,"15min":1.752},"last":1.1870000000000001},{"interface":"front","average":{"1min":1.319,"5min":1.319,"15min":1.319},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":2.1240000000000001,"5min":2.1240000000000001,"15min":2.1240000000000001},"last":1.002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2669999999999999}]},{"osd":12,"last update":"Sun Nov 5 08:33:32 2023","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.381,"15min":1.381},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":2.0710000000000002,"5min":2.0710000000000002,"15min":2.0710000000000002},"last":1.2709999999999999},{"interface":"front","average":{"1min":1.462,"5min":1.462,"15min":1.462},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":2.0350000000000001,"5min":2.0350000000000001,"15min":2.0350000000000001},"last":1.4550000000000001}]},{"osd":13,"last update":"Sun Nov 5 08:33:32 2023","interfaces":[{"interface":"back","average":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"min":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"max":{"1min":2.8010000000000002,"5min":2.8010000000000002,"15min":2.8010000000000002},"last":1.748},{"interface":"front","average":{"1min":1.478,"5min":1.478,"15min":1.478},"min":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"max":{"1min":2.177,"5min":2.177,"15min":2.177},"last":1.7150000000000001}]},{"osd":14,"last update":"Sun Nov 5 08:33:32 2023","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.506,"15min":1.506},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.849,"5min":1.849,"15min":1.849},"last":1.772},{"interface":"front","average":{"1min":1.6699999999999999,"5min":1.6699999999999999,"15min":1.6699999999999999},"min":{"1min":1.294,"5min":1.294,"15min":1.294},"max":{"1min":2.9710000000000001,"5min":2.9710000000000001,"15min":2.9710000000000001},"last":1.591}]},{"osd":15,"last update":"Sun Nov 5 08:33:32 2023","interfaces":[{"interface":"back","average":{"1min":1.603,"5min":1.603,"15min":1.603},"min":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"max":{"1min":1.9670000000000001,"5min":1.9670000000000001,"15min":1.9670000000000001},"last":1.821},{"interface":"front","average":{"1min":1.716,"5min":1.716,"15min":1.716},"min":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"max":{"1min":3.0950000000000002,"5min":3.0950000000000002,"15min":3.0950000000000002},"last":1.6919999999999999}]}]},{"osd":10,"up_from":22,"seq":94489280533,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41780,"kb_used_data":14760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330060,"statfs":{"total":96636764160,"available":96593981440,"internally_reserved":0,"allocated":15114240,"data_stored":26340231,"data_compressed":165545,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,4,5,6,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":1,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.86399999999999999}]},{"osd":2,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.768,"5min":1.768,"15min":1.768},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.01,"5min":1.01,"15min":1.01},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":2.9729999999999999,"5min":2.9729999999999999,"15min":2.9729999999999999},"last":0.81699999999999995}]},{"osd":4,"last update":"Sun Nov 5 08:33:57 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":1.0600000000000001},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.482,"5min":1.482,"15min":1.482},"last":1.177}]},{"osd":5,"last update":"Sun Nov 5 08:33:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":3.4790000000000001,"5min":3.4790000000000001,"15min":3.4790000000000001},"last":1.151},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":3.5339999999999998,"5min":3.5339999999999998,"15min":3.5339999999999998},"last":0.90600000000000003}]},{"osd":6,"last update":"Sun Nov 5 08:33:57 2023","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.55,"5min":1.55,"15min":1.55},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.5469999999999999,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":1.0269999999999999}]},{"osd":8,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.105,"15min":1.105},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.661,"5min":1.661,"15min":1.661},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.673,"5min":1.673,"15min":1.673},"last":1.0820000000000001}]},{"osd":9,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.264,"15min":1.264},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":2.8199999999999998,"5min":2.8199999999999998,"15min":2.8199999999999998},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.319,"5min":1.319,"15min":1.319},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":1.827,"5min":1.827,"15min":1.827},"last":1.28}]},{"osd":11,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":1.877,"5min":1.877,"15min":1.877},"last":1.234},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":2.8889999999999998,"5min":2.8889999999999998,"15min":2.8889999999999998},"last":1.256}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":13,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.536,"5min":1.536,"15min":1.536},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":2.778,"5min":2.778,"15min":2.778},"last":1.355},{"interface":"front","average":{"1min":1.6200000000000001,"5min":1.6200000000000001,"15min":1.6200000000000001},"min":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"max":{"1min":3.681,"5min":3.681,"15min":3.681},"last":1.1279999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]},{"osd":15,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":2.0550000000000002,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":1.331},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.5289999999999999,"15min":1.5289999999999999},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.4620000000000002,"5min":2.4620000000000002,"15min":2.4620000000000002},"last":1.3080000000000001}]}]},{"osd":11,"up_from":21,"seq":90194313238,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47192,"kb_used_data":16008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324648,"statfs":{"total":96636764160,"available":96588439552,"internally_reserved":0,"allocated":16392192,"data_stored":25136007,"data_compressed":165545,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,4,5,6,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":1,"last update":"Sun Nov 5 08:33:53 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.044,"15min":1.044},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":1.069,"5min":1.069,"15min":1.069},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.85199999999999998}]},{"osd":2,"last update":"Sun Nov 5 08:33:48 2023","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.216,"15min":1.216},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":2.298,"5min":2.298,"15min":2.298},"last":1.117},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":2.9020000000000001,"5min":2.9020000000000001,"15min":2.9020000000000001},"last":1.1499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":4,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.167,"15min":1.167},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.7390000000000001,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":1.048},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.91900000000000004}]},{"osd":5,"last update":"Sun Nov 5 08:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.244,"15min":1.244},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":3.0459999999999998,"5min":3.0459999999999998,"15min":3.0459999999999998},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":3.3799999999999999,"5min":3.3799999999999999,"15min":3.3799999999999999},"last":1.3200000000000001}]},{"osd":6,"last update":"Sun Nov 5 08:33:30 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.244,"15min":1.244},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":2.0630000000000002,"5min":2.0630000000000002,"15min":2.0630000000000002},"last":1.4319999999999999},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":2.7770000000000001,"5min":2.7770000000000001,"15min":2.7770000000000001},"last":1.532}]},{"osd":7,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.76,"5min":1.76,"15min":1.76},"last":1.508},{"interface":"front","average":{"1min":1.395,"5min":1.395,"15min":1.395},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.909,"5min":1.909,"15min":1.909},"last":1.6850000000000001}]},{"osd":8,"last update":"Sun Nov 5 08:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.9079999999999999,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.948,"5min":1.948,"15min":1.948},"last":1.774}]},{"osd":9,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":2.089,"5min":2.089,"15min":2.089},"last":1.472},{"interface":"front","average":{"1min":1.5660000000000001,"5min":1.5660000000000001,"15min":1.5660000000000001},"min":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"max":{"1min":2.1720000000000002,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":1.6499999999999999}]},{"osd":10,"last update":"Sun Nov 5 08:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.599,"5min":1.599,"15min":1.599},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.6890000000000001,"5min":2.6890000000000001,"15min":2.6890000000000001},"last":1.7969999999999999},{"interface":"front","average":{"1min":1.371,"5min":1.371,"15min":1.371},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":2.3679999999999999,"5min":2.3679999999999999,"15min":2.3679999999999999},"last":1.569}]},{"osd":12,"last update":"Sun Nov 5 08:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"max":{"1min":2.423,"5min":2.423,"15min":2.423},"last":1.8660000000000001},{"interface":"front","average":{"1min":1.702,"5min":1.702,"15min":1.702},"min":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"max":{"1min":2.6160000000000001,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":1.75}]},{"osd":13,"last update":"Sun Nov 5 08:33:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.4450000000000001,"15min":1.4450000000000001},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":2.48,"5min":2.48,"15min":2.48},"last":1.623},{"interface":"front","average":{"1min":1.714,"5min":1.714,"15min":1.714},"min":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"max":{"1min":2.6219999999999999,"5min":2.6219999999999999,"15min":2.6219999999999999},"last":1.595}]},{"osd":14,"last update":"Sun Nov 5 08:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.8580000000000001,"5min":1.8580000000000001,"15min":1.8580000000000001},"min":{"1min":1.137,"5min":1.137,"15min":1.137},"max":{"1min":5.375,"5min":5.375,"15min":5.375},"last":1.724},{"interface":"front","average":{"1min":1.754,"5min":1.754,"15min":1.754},"min":{"1min":1.038,"5min":1.038,"15min":1.038},"max":{"1min":2.5369999999999999,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.9179999999999999}]},{"osd":15,"last update":"Sun Nov 5 08:33:30 2023","interfaces":[{"interface":"back","average":{"1min":2.125,"5min":2.125,"15min":2.125},"min":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":1.843},{"interface":"front","average":{"1min":1.762,"5min":1.762,"15min":1.762},"min":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"max":{"1min":5.2549999999999999,"5min":5.2549999999999999,"15min":5.2549999999999999},"last":1.819}]}]},{"osd":12,"up_from":22,"seq":94489280533,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34744,"kb_used_data":7724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337096,"statfs":{"total":96636764160,"available":96601186304,"internally_reserved":0,"allocated":7909376,"data_stored":10623293,"data_compressed":165545,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[2,3,4,5,6,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":1,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.603,"5min":1.603,"15min":1.603},"last":1.2909999999999999}]},{"osd":2,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":1.0309999999999999},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.64400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0980000000000001}]},{"osd":4,"last update":"Sun Nov 5 08:34:02 2023","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.429,"5min":1.429,"15min":1.429},"last":1.0680000000000001}]},{"osd":5,"last update":"Sun Nov 5 08:34:02 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":1.123},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":1.2809999999999999}]},{"osd":6,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.73299999999999998},{"interface":"front","average":{"1min":1.073,"5min":1.073,"15min":1.073},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":1.343}]},{"osd":9,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.208,"15min":1.208},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":2.2109999999999999,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":0.99099999999999999},{"interface":"front","average":{"1min":1.206,"5min":1.206,"15min":1.206},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.964,"5min":1.964,"15min":1.964},"last":0.96399999999999997}]},{"osd":11,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":2.1659999999999999,"5min":2.1659999999999999,"15min":2.1659999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.8680000000000001,"5min":1.8680000000000001,"15min":1.8680000000000001},"last":1.149}]},{"osd":13,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.25,"5min":2.25,"15min":2.25},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":2.399,"5min":2.399,"15min":2.399},"last":1.391}]},{"osd":14,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":2.1070000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.218},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":2.3420000000000001,"5min":2.3420000000000001,"15min":2.3420000000000001},"last":1.2450000000000001}]},{"osd":15,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.2959999999999998,"5min":2.2959999999999998,"15min":2.2959999999999998},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.4099999999999999,"15min":1.4099999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.873,"5min":1.873,"15min":1.873},"last":1.3169999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6949,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6949,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4165632,"data_stored":4165632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14594048,"data_stored":21921792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11247616,"data_stored":20656128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8929280,"data_stored":11505664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12468224,"data_stored":18362368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":6107136,"data_stored":7225344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15101952,"data_stored":20791296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12271616,"data_stored":18165760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":18612224,"data_stored":33574912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":10563584,"data_stored":13963264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":13856768,"data_stored":25358336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":15134720,"data_stored":24154112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":6656000,"data_stored":9641984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":13787136,"data_stored":21987328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":12091392,"data_stored":24596480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":9965568,"data_stored":15785984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-05T08:34:03.236 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-05T08:34:03.306 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:34:03.306 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:34:03.321 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2023-11-05T08:34:03.322 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:34:03.317+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:34:03.420 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-05T08:34:03.445 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:34:03.445 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:34:03.459 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-11-05T08:34:03.523 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:34:03.520+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:34:03.577 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:34:03.577 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:34:03.583 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:34:03.586 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-11-05T08:34:03.591 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-11-05T08:34:03.605 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":141,"stamp":"2023-11-05T08:34:02.822007+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":116531744,"num_objects":111,"num_object_clones":61,"num_object_copies":767,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":71,"num_objects_misplaced":8,"num_objects_unfound":0,"num_objects_dirty":111,"num_whiteouts":4,"num_read":480,"num_read_kb":152086,"num_write":856,"num_write_kb":119260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":23068672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1147,"ondisk_log_size":1147,"up":199,"acting":199,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":190,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":664896,"kb_used_data":198104,"kb_used_omap":29,"kb_used_meta":466530,"kb_avail":1509284544,"statfs":{"total":1546188226560,"available":1545507373056,"internally_reserved":0,"allocated":202858496,"data_stored":303144385,"data_compressed":2502735,"data_compressed_allocated":5427200,"data_compressed_original":11747328,"omap_allocated":30361,"internal_metadata":477727079},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7061504,"num_objects":6,"num_object_clones":6,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":20,"num_read_kb":30871,"num_write":43,"num_write_kb":6896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.049608"},"pg_stats":[{"pgid":"3.19","version":"108'58","reported_seq":359,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-05T08:34:02.297869+0000","last_change":"2023-11-05T08:34:01.803774+0000","last_active":"2023-11-05T08:34:02.297869+0000","last_peered":"2023-11-05T08:34:02.297869+0000","last_clean":"2023-11-05T08:34:02.297869+0000","last_became_active":"2023-11-05T08:32:59.211522+0000","last_became_peered":"2023-11-05T08:32:59.211522+0000","last_unstale":"2023-11-05T08:34:02.297869+0000","last_undegraded":"2023-11-05T08:34:02.297869+0000","last_fullsized":"2023-11-05T08:34:02.297869+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":5,"last_scrub":"39'22","last_scrub_stamp":"2023-11-05T08:33:01.870959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:01.870959+0000","objects_scrubbed":3,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:14.506291+0000","scrub_duration":0.0087276880000000008,"objects_trimmed":0,"snaptrim_duration":0.00096281600000000002,"stat_sum":{"num_bytes":5734400,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":3027,"num_write":11,"num_write_kb":4144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,11,9,13,7,1],"acting":[8,4,11,9,13,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.18","version":"103'57","reported_seq":328,"reported_epoch":104,"state":"active+clean+remapped","last_fresh":"2023-11-05T08:33:57.793865+0000","last_change":"2023-11-05T08:33:54.765083+0000","last_active":"2023-11-05T08:33:57.793865+0000","last_peered":"2023-11-05T08:33:57.793865+0000","last_clean":"2023-11-05T08:33:57.793865+0000","last_became_active":"2023-11-05T08:32:59.217241+0000","last_became_peered":"2023-11-05T08:32:59.217241+0000","last_unstale":"2023-11-05T08:33:57.793865+0000","last_undegraded":"2023-11-05T08:33:57.793865+0000","last_fullsized":"2023-11-05T08:33:57.793865+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":5,"last_scrub":"52'28","last_scrub_stamp":"2023-11-05T08:33:13.971775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:13.971775+0000","objects_scrubbed":5,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:36.131790+0000","scrub_duration":0.019915236999999999,"objects_trimmed":0,"snaptrim_duration":0.021818140999999999,"stat_sum":{"num_bytes":6160384,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":3739,"num_write":31,"num_write_kb":4336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,2147483647,4,5,7,12],"acting":[13,6,2,4,5,7,12],"avail_no_missing":["13(0)","2(2)","4(3)","5(4)","6(1)","7(5)","12(6)"],"object_location_counts":[{"shards":"2(2),4(3),5(4),6(1),7(5),12(6),13(0)","objects":4}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"}]},{"pgid":"3.17","version":"105'36","reported_seq":383,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-05T08:34:02.203149+0000","last_change":"2023-11-05T08:34:02.204287+0000","last_active":"2023-11-05T08:34:02.203149+0000","last_peered":"2023-11-05T08:34:02.203149+0000","last_clean":"2023-11-05T08:34:02.203149+0000","last_became_active":"2023-11-05T08:33:50.677365+0000","last_became_peered":"2023-11-05T08:33:50.677365+0000","last_unstale":"2023-11-05T08:34:02.203149+0000","last_undegraded":"2023-11-05T08:34:02.203149+0000","last_fullsized":"2023-11-05T08:34:02.203149+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"49'17","last_scrub_stamp":"2023-11-05T08:33:13.975356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:13.975356+0000","objects_scrubbed":3,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:33.996216+0000","scrub_duration":0.014066680999999999,"objects_trimmed":0,"snaptrim_duration":0.001635697,"stat_sum":{"num_bytes":2588672,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":45,"num_write_kb":2176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":999424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,10,9,7,1,13],"acting":[5,3,10,9,7,1,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.16","version":"97'48","reported_seq":326,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-05T08:34:01.820510+0000","last_change":"2023-11-05T08:34:01.821079+0000","last_active":"2023-11-05T08:34:01.820510+0000","last_peered":"2023-11-05T08:34:01.820510+0000","last_clean":"2023-11-05T08:34:01.820510+0000","last_became_active":"2023-11-05T08:32:59.210858+0000","last_became_peered":"2023-11-05T08:32:59.210858+0000","last_unstale":"2023-11-05T08:34:01.820510+0000","last_undegraded":"2023-11-05T08:34:01.820510+0000","last_fullsized":"2023-11-05T08:34:01.820510+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":5,"last_scrub":"46'13","last_scrub_stamp":"2023-11-05T08:33:00.659926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:00.659926+0000","objects_scrubbed":1,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:06.720172+0000","scrub_duration":0.0077151709999999998,"objects_trimmed":0,"snaptrim_duration":0.00073938499999999996,"stat_sum":{"num_bytes":4669440,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":15,"num_write_kb":4560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15,4,8,9,5,10],"acting":[11,15,4,8,9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.15","version":"36'15","reported_seq":268,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-05T08:34:01.805163+0000","last_change":"2023-11-05T08:34:01.806289+0000","last_active":"2023-11-05T08:34:01.805163+0000","last_peered":"2023-11-05T08:34:01.805163+0000","last_clean":"2023-11-05T08:34:01.805163+0000","last_became_active":"2023-11-05T08:33:50.675923+0000","last_became_peered":"2023-11-05T08:33:50.675923+0000","last_unstale":"2023-11-05T08:34:01.805163+0000","last_undegraded":"2023-11-05T08:34:01.805163+0000","last_fullsized":"2023-11-05T08:34:01.805163+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"36'15","last_scrub_stamp":"2023-11-05T08:33:52.537969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:52.537969+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:35:09.298059+0000","scrub_duration":0.0054037720000000003,"objects_trimmed":0,"snaptrim_duration":0.0020680849999999999,"stat_sum":{"num_bytes":32768,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,3,13,14,2,15],"acting":[9,6,3,13,14,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.14","version":"101'27","reported_seq":281,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-05T08:34:01.806326+0000","last_change":"2023-11-05T08:34:01.807260+0000","last_active":"2023-11-05T08:34:01.806326+0000","last_peered":"2023-11-05T08:34:01.806326+0000","last_clean":"2023-11-05T08:34:01.806326+0000","last_became_active":"2023-11-05T08:33:50.674132+0000","last_became_peered":"2023-11-05T08:33:50.674132+0000","last_unstale":"2023-11-05T08:34:01.806326+0000","last_undegraded":"2023-11-05T08:34:01.806326+0000","last_fullsized":"2023-11-05T08:34:01.806326+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"49'18","last_scrub_stamp":"2023-11-05T08:33:04.812402+0000","last_deep_scrub":"49'18","last_deep_scrub_stamp":"2023-11-05T08:33:04.812402+0000","last_clean_scrub_stamp":"2023-11-05T08:33:04.812402+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:09.862950+0000","scrub_duration":0.0086643829999999995,"objects_trimmed":0,"snaptrim_duration":0.0023598249999999999,"stat_sum":{"num_bytes":294912,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1966080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,15,1,13,8,4],"acting":[3,9,15,1,13,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":98,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-05T08:34:01.152657+0000","last_change":"2023-11-05T08:33:50.142139+0000","last_active":"2023-11-05T08:34:01.152657+0000","last_peered":"2023-11-05T08:34:01.152657+0000","last_clean":"2023-11-05T08:34:01.152657+0000","last_became_active":"2023-11-05T08:32:31.941858+0000","last_became_peered":"2023-11-05T08:32:31.941858+0000","last_unstale":"2023-11-05T08:34:01.152657+0000","last_undegraded":"2023-11-05T08:34:01.152657+0000","last_fullsized":"2023-11-05T08:34:01.152657+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:33:50.141700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:33:50.141700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:51.467865+0000","scrub_duration":0.0025514660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"35'9","reported_seq":178,"reported_epoch":104,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:33:57.794197+0000","last_change":"2023-11-05T08:32:54.153555+0000","last_active":"2023-11-05T08:33:57.794197+0000","last_peered":"2023-11-05T08:33:57.794197+0000","last_clean":"2023-11-05T08:32:48.102597+0000","last_became_active":"2023-11-05T08:32:54.153555+0000","last_became_peered":"2023-11-05T08:32:54.153555+0000","last_unstale":"2023-11-05T08:33:57.794197+0000","last_undegraded":"2023-11-05T08:32:54.121924+0000","last_fullsized":"2023-11-05T08:32:54.120000+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":32,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1089536,"num_objects":1,"num_object_clones":1,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":1064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,13,1,15,12,7,9],"acting":[2147483647,13,1,15,12,7,9],"avail_no_missing":["13(1)","1(2)","7(5)","9(6)","12(4)","15(3)"],"object_location_counts":[{"shards":"1(2),7(5),9(6),12(4),13(1),15(3)","objects":1}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":99,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-05T08:34:01.451381+0000","last_change":"2023-11-05T08:33:52.971633+0000","last_active":"2023-11-05T08:34:01.451381+0000","last_peered":"2023-11-05T08:34:01.451381+0000","last_clean":"2023-11-05T08:34:01.451381+0000","last_became_active":"2023-11-05T08:32:31.941207+0000","last_became_peered":"2023-11-05T08:32:31.941207+0000","last_unstale":"2023-11-05T08:34:01.451381+0000","last_undegraded":"2023-11-05T08:34:01.451381+0000","last_fullsized":"2023-11-05T08:34:01.451381+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:33:52.971456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:33:52.971456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:35:07.735098+0000","scrub_duration":0.002374414,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"39'12","reported_seq":197,"reported_epoch":108,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:34:01.929244+0000","last_change":"2023-11-05T08:32:54.151639+0000","last_active":"2023-11-05T08:34:01.929244+0000","last_peered":"2023-11-05T08:34:01.929244+0000","last_clean":"2023-11-05T08:32:49.074431+0000","last_became_active":"2023-11-05T08:32:54.151639+0000","last_became_peered":"2023-11-05T08:32:54.151639+0000","last_unstale":"2023-11-05T08:34:01.929244+0000","last_undegraded":"2023-11-05T08:32:54.120694+0000","last_fullsized":"2023-11-05T08:32:54.119652+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":34,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030081180000000002,"stat_sum":{"num_bytes":319488,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,2147483647,8,5,11],"acting":[4,6,9,2147483647,8,5,11],"avail_no_missing":["4(0)","5(5)","6(1)","8(4)","9(2)","11(6)"],"object_location_counts":[{"shards":"4(0),5(5),6(1),8(4),9(2),11(6)","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":99,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-11-05T08:33:59.863392+0000","last_change":"2023-11-05T08:33:50.446617+0000","last_active":"2023-11-05T08:33:59.863392+0000","last_peered":"2023-11-05T08:33:59.863392+0000","last_clean":"2023-11-05T08:33:59.863392+0000","last_became_active":"2023-11-05T08:32:31.940312+0000","last_became_peered":"2023-11-05T08:32:31.940312+0000","last_unstale":"2023-11-05T08:33:59.863392+0000","last_undegraded":"2023-11-05T08:33:59.863392+0000","last_fullsized":"2023-11-05T08:33:59.863392+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:33:50.446312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:33:50.446312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:57.987976+0000","scrub_duration":0.0037389620000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"39'12","reported_seq":196,"reported_epoch":108,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:34:01.803840+0000","last_change":"2023-11-05T08:32:54.141915+0000","last_active":"2023-11-05T08:34:01.803840+0000","last_peered":"2023-11-05T08:34:01.803840+0000","last_clean":"2023-11-05T08:32:49.088117+0000","last_became_active":"2023-11-05T08:32:54.141915+0000","last_became_peered":"2023-11-05T08:32:54.141915+0000","last_unstale":"2023-11-05T08:34:01.803840+0000","last_undegraded":"2023-11-05T08:32:54.115420+0000","last_fullsized":"2023-11-05T08:32:54.114438+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":34,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.011061272,"stat_sum":{"num_bytes":1376256,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,4,1,2147483647,14,13],"acting":[8,2,4,1,2147483647,14,13],"avail_no_missing":["8(0)","1(3)","2(1)","4(2)","13(6)","14(5)"],"object_location_counts":[{"shards":"1(3),2(1),4(2),8(0),13(6),14(5)","objects":1}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":96,"reported_epoch":107,"state":"active+undersized","last_fresh":"2023-11-05T08:34:01.632325+0000","last_change":"2023-11-05T08:32:51.087583+0000","last_active":"2023-11-05T08:34:01.632325+0000","last_peered":"2023-11-05T08:34:01.632325+0000","last_clean":"2023-11-05T08:32:49.071396+0000","last_became_active":"2023-11-05T08:32:51.087583+0000","last_became_peered":"2023-11-05T08:32:51.087583+0000","last_unstale":"2023-11-05T08:34:01.632325+0000","last_undegraded":"2023-11-05T08:34:01.632325+0000","last_fullsized":"2023-11-05T08:32:51.085658+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"103'31","reported_seq":222,"reported_epoch":108,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:34:02.250234+0000","last_change":"2023-11-05T08:32:54.139002+0000","last_active":"2023-11-05T08:34:02.250234+0000","last_peered":"2023-11-05T08:34:02.250234+0000","last_clean":"2023-11-05T08:32:49.088591+0000","last_became_active":"2023-11-05T08:32:54.139002+0000","last_became_peered":"2023-11-05T08:32:54.139002+0000","last_unstale":"2023-11-05T08:34:02.250234+0000","last_undegraded":"2023-11-05T08:32:54.118261+0000","last_fullsized":"2023-11-05T08:32:54.117408+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":34,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.020413489999999999,"stat_sum":{"num_bytes":5988352,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":12,"num_read_kb":2382,"num_write":18,"num_write_kb":5848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,4,13,7,2147483647,6],"acting":[15,8,4,13,7,2147483647,6],"avail_no_missing":["15(0)","4(2)","6(6)","7(4)","8(1)","13(3)"],"object_location_counts":[{"shards":"4(2),6(6),7(4),8(1),13(3),15(0)","objects":5}],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.3","version":"97'17","reported_seq":359,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-11-05T08:33:57.794215+0000","last_change":"2023-11-05T08:33:54.770853+0000","last_active":"2023-11-05T08:33:57.794215+0000","last_peered":"2023-11-05T08:33:57.794215+0000","last_clean":"2023-11-05T08:33:57.794215+0000","last_became_active":"2023-11-05T08:33:50.683698+0000","last_became_peered":"2023-11-05T08:33:50.683698+0000","last_unstale":"2023-11-05T08:33:57.794215+0000","last_undegraded":"2023-11-05T08:33:57.794215+0000","last_fullsized":"2023-11-05T08:33:57.794215+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'17","last_scrub_stamp":"2023-11-05T08:33:53.073917+0000","last_deep_scrub":"97'17","last_deep_scrub_stamp":"2023-11-05T08:33:53.073917+0000","last_clean_scrub_stamp":"2023-11-05T08:33:53.073917+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:58.526902+0000","scrub_duration":0.0068216960000000004,"objects_trimmed":0,"snaptrim_duration":0.018960549,"stat_sum":{"num_bytes":524288,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":1537,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,4,15,2,5,3],"acting":[13,7,4,15,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"}]},{"pgid":"2.2","version":"0'0","reported_seq":104,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-05T08:34:00.798911+0000","last_change":"2023-11-05T08:33:53.537740+0000","last_active":"2023-11-05T08:34:00.798911+0000","last_peered":"2023-11-05T08:34:00.798911+0000","last_clean":"2023-11-05T08:34:00.798911+0000","last_became_active":"2023-11-05T08:33:47.679907+0000","last_became_peered":"2023-11-05T08:33:47.679907+0000","last_unstale":"2023-11-05T08:34:00.798911+0000","last_undegraded":"2023-11-05T08:34:00.798911+0000","last_fullsized":"2023-11-05T08:34:00.798911+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:33:53.537442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:33:53.537442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:35:15.667752+0000","scrub_duration":0.003485906,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3],"acting":[12,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.0","version":"33'9","reported_seq":342,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-05T08:34:01.808241+0000","last_change":"2023-11-05T08:34:01.808699+0000","last_active":"2023-11-05T08:34:01.808241+0000","last_peered":"2023-11-05T08:34:01.808241+0000","last_clean":"2023-11-05T08:34:01.808241+0000","last_became_active":"2023-11-05T08:32:54.144790+0000","last_became_peered":"2023-11-05T08:32:54.144790+0000","last_unstale":"2023-11-05T08:34:01.808241+0000","last_undegraded":"2023-11-05T08:34:01.808241+0000","last_fullsized":"2023-11-05T08:34:01.808241+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'9","last_scrub_stamp":"2023-11-05T08:33:03.810524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:03.810524+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:07.644773+0000","scrub_duration":0.0066041449999999996,"objects_trimmed":0,"snaptrim_duration":0.00083072500000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,15,1,5,6,4],"acting":[14,2,15,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":87,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-11-05T08:33:56.832312+0000","last_change":"2023-11-05T08:32:31.941357+0000","last_active":"2023-11-05T08:33:56.832312+0000","last_peered":"2023-11-05T08:33:56.832312+0000","last_clean":"2023-11-05T08:33:56.832312+0000","last_became_active":"2023-11-05T08:32:31.940564+0000","last_became_peered":"2023-11-05T08:32:31.940564+0000","last_unstale":"2023-11-05T08:33:56.832312+0000","last_undegraded":"2023-11-05T08:33:56.832312+0000","last_fullsized":"2023-11-05T08:33:56.832312+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:33:58.168161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.1","version":"106'40","reported_seq":385,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-05T08:34:01.927993+0000","last_change":"2023-11-05T08:34:01.928527+0000","last_active":"2023-11-05T08:34:01.927993+0000","last_peered":"2023-11-05T08:34:01.927993+0000","last_clean":"2023-11-05T08:34:01.927993+0000","last_became_active":"2023-11-05T08:32:54.144895+0000","last_became_peered":"2023-11-05T08:32:54.144895+0000","last_unstale":"2023-11-05T08:34:01.927993+0000","last_undegraded":"2023-11-05T08:34:01.927993+0000","last_fullsized":"2023-11-05T08:34:01.927993+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'11","last_scrub_stamp":"2023-11-05T08:33:04.003557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:04.003557+0000","objects_scrubbed":3,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:15.185676+0000","scrub_duration":0.010761797,"objects_trimmed":0,"snaptrim_duration":0.00073924699999999995,"stat_sum":{"num_bytes":1949696,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":772,"num_write":7,"num_write_kb":2480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,5,10,2,13],"acting":[4,11,9,5,10,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":96,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-11-05T08:33:59.863171+0000","last_change":"2023-11-05T08:33:47.700202+0000","last_active":"2023-11-05T08:33:59.863171+0000","last_peered":"2023-11-05T08:33:59.863171+0000","last_clean":"2023-11-05T08:33:59.863171+0000","last_became_active":"2023-11-05T08:33:47.699438+0000","last_became_peered":"2023-11-05T08:33:47.699438+0000","last_unstale":"2023-11-05T08:33:59.863171+0000","last_undegraded":"2023-11-05T08:33:59.863171+0000","last_fullsized":"2023-11-05T08:33:59.863171+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"102'30","reported_seq":374,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-11-05T08:33:57.793872+0000","last_change":"2023-11-05T08:33:54.764012+0000","last_active":"2023-11-05T08:33:57.793872+0000","last_peered":"2023-11-05T08:33:57.793872+0000","last_clean":"2023-11-05T08:33:57.793872+0000","last_became_active":"2023-11-05T08:33:50.678084+0000","last_became_peered":"2023-11-05T08:33:50.678084+0000","last_unstale":"2023-11-05T08:33:57.793872+0000","last_undegraded":"2023-11-05T08:33:57.793872+0000","last_fullsized":"2023-11-05T08:33:57.793872+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'25","last_scrub_stamp":"2023-11-05T08:33:52.100890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:52.100890+0000","objects_scrubbed":1,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:35:04.875042+0000","scrub_duration":0.007958952,"objects_trimmed":0,"snaptrim_duration":0.012335969,"stat_sum":{"num_bytes":3670016,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,3,6,15,1,11],"acting":[13,2,3,6,15,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"}]},{"pgid":"2.3","version":"0'0","reported_seq":99,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-05T08:34:01.452342+0000","last_change":"2023-11-05T08:34:00.133458+0000","last_active":"2023-11-05T08:34:01.452342+0000","last_peered":"2023-11-05T08:34:01.452342+0000","last_clean":"2023-11-05T08:34:01.452342+0000","last_became_active":"2023-11-05T08:32:31.939705+0000","last_became_peered":"2023-11-05T08:32:31.939705+0000","last_unstale":"2023-11-05T08:34:01.452342+0000","last_undegraded":"2023-11-05T08:34:01.452342+0000","last_fullsized":"2023-11-05T08:34:01.452342+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:34:00.133184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:34:00.133184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:35:05.010050+0000","scrub_duration":0.0035944599999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"25'157","reported_seq":294,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-11-05T08:33:56.833280+0000","last_change":"2023-11-05T08:33:41.087862+0000","last_active":"2023-11-05T08:33:56.833280+0000","last_peered":"2023-11-05T08:33:56.833280+0000","last_clean":"2023-11-05T08:33:56.833280+0000","last_became_active":"2023-11-05T08:32:28.758834+0000","last_became_peered":"2023-11-05T08:32:28.758834+0000","last_unstale":"2023-11-05T08:33:56.833280+0000","last_undegraded":"2023-11-05T08:33:56.833280+0000","last_fullsized":"2023-11-05T08:33:56.833280+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'157","last_scrub_stamp":"2023-11-05T08:33:41.087696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:27.693099+0000","last_clean_scrub_stamp":"2023-11-05T08:33:41.087696+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:35:09.900066+0000","scrub_duration":0.0045831459999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.8","version":"108'34","reported_seq":222,"reported_epoch":108,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:34:02.300393+0000","last_change":"2023-11-05T08:32:54.151946+0000","last_active":"2023-11-05T08:34:02.300393+0000","last_peered":"2023-11-05T08:34:02.300393+0000","last_clean":"2023-11-05T08:32:49.090022+0000","last_became_active":"2023-11-05T08:32:54.151946+0000","last_became_peered":"2023-11-05T08:32:54.151946+0000","last_unstale":"2023-11-05T08:34:02.300393+0000","last_undegraded":"2023-11-05T08:32:54.119646+0000","last_fullsized":"2023-11-05T08:32:54.118811+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":34,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.013731910999999999,"stat_sum":{"num_bytes":4816896,"num_objects":6,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":10,"num_read_kb":1684,"num_write":27,"num_write_kb":4144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2147483647,2,1,14,15,6],"acting":[5,2147483647,2,1,14,15,6],"avail_no_missing":["5(0)","1(3)","2(2)","6(6)","14(4)","15(5)"],"object_location_counts":[{"shards":"1(3),2(2),5(0),6(6),14(4),15(5)","objects":6}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.9","version":"103'64","reported_seq":315,"reported_epoch":108,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:34:01.843421+0000","last_change":"2023-11-05T08:33:50.691854+0000","last_active":"2023-11-05T08:34:01.843421+0000","last_peered":"2023-11-05T08:34:01.843421+0000","last_clean":"2023-11-05T08:32:49.072716+0000","last_became_active":"2023-11-05T08:33:50.691854+0000","last_became_peered":"2023-11-05T08:33:50.691854+0000","last_unstale":"2023-11-05T08:34:01.843421+0000","last_undegraded":"2023-11-05T08:33:50.658262+0000","last_fullsized":"2023-11-05T08:33:50.657391+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":34,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015323719999999999,"stat_sum":{"num_bytes":14761984,"num_objects":13,"num_object_clones":9,"num_object_copies":91,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":44,"num_read_kb":36208,"num_write":100,"num_write_kb":14416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":10584064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,2147483647,3,15,8,10],"acting":[14,1,2147483647,3,15,8,10],"avail_no_missing":["14(0)","1(1)","3(3)","8(5)","10(6)","15(4)"],"object_location_counts":[{"shards":"1(1),3(3),8(5),10(6),14(0),15(4)","objects":13}],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.a","version":"108'47","reported_seq":407,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-05T08:34:01.823848+0000","last_change":"2023-11-05T08:34:01.818604+0000","last_active":"2023-11-05T08:34:01.823848+0000","last_peered":"2023-11-05T08:34:01.823848+0000","last_clean":"2023-11-05T08:34:01.823848+0000","last_became_active":"2023-11-05T08:33:50.684400+0000","last_became_peered":"2023-11-05T08:33:50.684400+0000","last_unstale":"2023-11-05T08:34:01.823848+0000","last_undegraded":"2023-11-05T08:34:01.823848+0000","last_fullsized":"2023-11-05T08:34:01.823848+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'44","last_scrub_stamp":"2023-11-05T08:34:01.079975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:34:01.079975+0000","objects_scrubbed":3,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:35:14.986189+0000","scrub_duration":0.0091389039999999998,"objects_trimmed":0,"snaptrim_duration":0.0080629699999999992,"stat_sum":{"num_bytes":3538944,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":2,"num_read":25,"num_read_kb":17912,"num_write":36,"num_write_kb":3696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":6406144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,9,14,6,11],"acting":[3,7,1,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.b","version":"108'58","reported_seq":253,"reported_epoch":108,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:34:02.347090+0000","last_change":"2023-11-05T08:32:51.095575+0000","last_active":"2023-11-05T08:34:02.347090+0000","last_peered":"2023-11-05T08:34:02.347090+0000","last_clean":"2023-11-05T08:32:49.072970+0000","last_became_active":"2023-11-05T08:32:51.095575+0000","last_became_peered":"2023-11-05T08:32:51.095575+0000","last_unstale":"2023-11-05T08:34:02.347090+0000","last_undegraded":"2023-11-05T08:32:51.085902+0000","last_fullsized":"2023-11-05T08:32:51.084936+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":34,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:12.100399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021279290000000002,"stat_sum":{"num_bytes":12435456,"num_objects":13,"num_object_clones":9,"num_object_copies":91,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":38,"num_read_kb":17439,"num_write":64,"num_write_kb":12768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2147483647,1,12,6,7,9],"acting":[4,2147483647,1,12,6,7,9],"avail_no_missing":["4(0)","1(2)","6(4)","7(5)","9(6)","12(3)"],"object_location_counts":[{"shards":"1(2),4(0),6(4),7(5),9(6),12(3)","objects":13}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.c","version":"79'6","reported_seq":186,"reported_epoch":108,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:34:01.820645+0000","last_change":"2023-11-05T08:32:51.099852+0000","last_active":"2023-11-05T08:34:01.820645+0000","last_peered":"2023-11-05T08:34:01.820645+0000","last_clean":"2023-11-05T08:32:49.087645+0000","last_became_active":"2023-11-05T08:32:51.099852+0000","last_became_peered":"2023-11-05T08:32:51.099852+0000","last_unstale":"2023-11-05T08:34:01.820645+0000","last_undegraded":"2023-11-05T08:32:51.089699+0000","last_fullsized":"2023-11-05T08:32:51.089041+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":34,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015630430000000001,"stat_sum":{"num_bytes":999424,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":7,"num_read_kb":914,"num_write":4,"num_write_kb":976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,2147483647,10,6,12,5],"acting":[11,8,2147483647,10,6,12,5],"avail_no_missing":["11(0)","5(6)","6(4)","8(1)","10(3)","12(5)"],"object_location_counts":[{"shards":"5(6),6(4),8(1),10(3),11(0),12(5)","objects":3}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.d","version":"103'36","reported_seq":257,"reported_epoch":108,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:34:01.810896+0000","last_change":"2023-11-05T08:33:50.690639+0000","last_active":"2023-11-05T08:34:01.810896+0000","last_peered":"2023-11-05T08:34:01.810896+0000","last_clean":"2023-11-05T08:32:49.088145+0000","last_became_active":"2023-11-05T08:33:50.690639+0000","last_became_peered":"2023-11-05T08:33:50.690639+0000","last_unstale":"2023-11-05T08:34:01.810896+0000","last_undegraded":"2023-11-05T08:33:50.664683+0000","last_fullsized":"2023-11-05T08:33:50.663817+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":34,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.024583713,"stat_sum":{"num_bytes":5013504,"num_objects":7,"num_object_clones":5,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":6009,"num_write":23,"num_write_kb":3536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3112960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,2147483647,8,4],"acting":[7,3,14,10,2147483647,8,4],"avail_no_missing":["7(0)","3(1)","4(6)","8(5)","10(3)","14(2)"],"object_location_counts":[{"shards":"3(1),4(6),7(0),8(5),10(3),14(2)","objects":7}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.e","version":"108'128","reported_seq":451,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-05T08:34:01.821498+0000","last_change":"2023-11-05T08:34:01.821498+0000","last_active":"2023-11-05T08:34:01.821498+0000","last_peered":"2023-11-05T08:34:01.821498+0000","last_clean":"2023-11-05T08:34:01.821498+0000","last_became_active":"2023-11-05T08:32:44.949185+0000","last_became_peered":"2023-11-05T08:32:44.949185+0000","last_unstale":"2023-11-05T08:34:01.821498+0000","last_undegraded":"2023-11-05T08:34:01.821498+0000","last_fullsized":"2023-11-05T08:34:01.821498+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'96","last_scrub_stamp":"2023-11-05T08:33:48.102281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:48.102281+0000","objects_scrubbed":12,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:35:04.316205+0000","scrub_duration":0.014167007000000001,"objects_trimmed":0,"snaptrim_duration":0.011717086999999999,"stat_sum":{"num_bytes":7028736,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":12450,"num_write":75,"num_write_kb":13632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1,11,8,6,12],"acting":[2,9,1,11,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.f","version":"108'85","reported_seq":284,"reported_epoch":108,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:34:01.938307+0000","last_change":"2023-11-05T08:32:51.104425+0000","last_active":"2023-11-05T08:34:01.938307+0000","last_peered":"2023-11-05T08:34:01.938307+0000","last_clean":"2023-11-05T08:32:49.098585+0000","last_became_active":"2023-11-05T08:32:51.104425+0000","last_became_peered":"2023-11-05T08:32:51.104425+0000","last_unstale":"2023-11-05T08:34:01.938307+0000","last_undegraded":"2023-11-05T08:32:51.091474+0000","last_fullsized":"2023-11-05T08:32:51.089784+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":34,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012093372,"stat_sum":{"num_bytes":22069248,"num_objects":20,"num_object_clones":16,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":50,"num_read_kb":25938,"num_write":122,"num_write_kb":23184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,14,13,8,10,2147483647],"acting":[11,2,14,13,8,10,2147483647],"avail_no_missing":["11(0)","2(1)","8(4)","10(5)","13(3)","14(2)"],"object_location_counts":[{"shards":"2(1),8(4),10(5),11(0),13(3),14(2)","objects":20}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.10","version":"103'68","reported_seq":332,"reported_epoch":104,"state":"active+clean+remapped","last_fresh":"2023-11-05T08:33:57.793801+0000","last_change":"2023-11-05T08:33:54.758708+0000","last_active":"2023-11-05T08:33:57.793801+0000","last_peered":"2023-11-05T08:33:57.793801+0000","last_clean":"2023-11-05T08:33:57.793801+0000","last_became_active":"2023-11-05T08:32:59.210399+0000","last_became_peered":"2023-11-05T08:32:59.210399+0000","last_unstale":"2023-11-05T08:33:57.793801+0000","last_undegraded":"2023-11-05T08:33:57.793801+0000","last_fullsized":"2023-11-05T08:33:57.793801+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":5,"last_scrub":"33'9","last_scrub_stamp":"2023-11-05T08:33:00.792248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:00.792248+0000","objects_scrubbed":3,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:19.462036+0000","scrub_duration":0.0073349119999999999,"objects_trimmed":0,"snaptrim_duration":0.017308477999999999,"stat_sum":{"num_bytes":6897664,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":5385,"num_write":48,"num_write_kb":8880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,7,4,8,2147483647,11],"acting":[13,6,7,4,8,6,11],"avail_no_missing":["13(0)","4(3)","6(1)","6(5)","7(2)","8(4)","11(6)"],"object_location_counts":[{"shards":"4(3),6(1),6(5),7(2),8(4),11(6),13(0)","objects":4}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"}]},{"pgid":"3.11","version":"97'31","reported_seq":382,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-05T08:34:01.802710+0000","last_change":"2023-11-05T08:34:01.803250+0000","last_active":"2023-11-05T08:34:01.802710+0000","last_peered":"2023-11-05T08:34:01.802710+0000","last_clean":"2023-11-05T08:34:01.802710+0000","last_became_active":"2023-11-05T08:32:59.214193+0000","last_became_peered":"2023-11-05T08:32:59.214193+0000","last_unstale":"2023-11-05T08:34:01.802710+0000","last_undegraded":"2023-11-05T08:34:01.802710+0000","last_fullsized":"2023-11-05T08:34:01.802710+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":5,"last_scrub":"46'17","last_scrub_stamp":"2023-11-05T08:33:00.070510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:00.070510+0000","objects_scrubbed":3,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:03.268732+0000","scrub_duration":0.0081678800000000006,"objects_trimmed":0,"snaptrim_duration":0.00078513400000000005,"stat_sum":{"num_bytes":3981312,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":16565,"num_write":5,"num_write_kb":1936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,15,9,5,6,1],"acting":[10,11,15,9,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.12","version":"37'7","reported_seq":297,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-05T08:34:01.804941+0000","last_change":"2023-11-05T08:34:01.806121+0000","last_active":"2023-11-05T08:34:01.804941+0000","last_peered":"2023-11-05T08:34:01.804941+0000","last_clean":"2023-11-05T08:34:01.804941+0000","last_became_active":"2023-11-05T08:33:50.685746+0000","last_became_peered":"2023-11-05T08:33:50.685746+0000","last_unstale":"2023-11-05T08:34:01.804941+0000","last_undegraded":"2023-11-05T08:34:01.804941+0000","last_fullsized":"2023-11-05T08:34:01.804941+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"37'7","last_scrub_stamp":"2023-11-05T08:33:02.650387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:02.650387+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:19.568123+0000","scrub_duration":0.0067664470000000001,"objects_trimmed":0,"snaptrim_duration":0.002076825,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8,6,4,3,15],"acting":[9,7,8,6,4,3,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.13","version":"97'25","reported_seq":357,"reported_epoch":108,"state":"active+clean+remapped","last_fresh":"2023-11-05T08:34:01.821656+0000","last_change":"2023-11-05T08:34:01.822172+0000","last_active":"2023-11-05T08:34:01.821656+0000","last_peered":"2023-11-05T08:34:01.821656+0000","last_clean":"2023-11-05T08:34:01.821656+0000","last_became_active":"2023-11-05T08:32:59.212969+0000","last_became_peered":"2023-11-05T08:32:59.212969+0000","last_unstale":"2023-11-05T08:34:01.821656+0000","last_undegraded":"2023-11-05T08:34:01.821656+0000","last_fullsized":"2023-11-05T08:34:01.821656+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'14","last_scrub_stamp":"2023-11-05T08:33:02.644028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:02.644028+0000","objects_scrubbed":1,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:10.863657+0000","scrub_duration":0.007386386,"objects_trimmed":0,"snaptrim_duration":0.001657958,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,9,13,2147483647,4,10],"acting":[11,7,9,13,2,4,10],"avail_no_missing":["11(0)","2(4)","4(5)","7(1)","9(2)","10(6)","13(3)"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"49","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":115941376,"num_objects":109,"num_object_clones":61,"num_object_copies":763,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":71,"num_objects_misplaced":8,"num_objects_unfound":0,"num_objects_dirty":109,"num_whiteouts":4,"num_read":344,"num_read_kb":151968,"num_write":653,"num_write_kb":115800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":23068672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":185552896,"data_stored":291856384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":990,"ondisk_log_size":990,"up":182,"acting":182,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":15,"acting":15,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":13898,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":22,"seq":94489280533,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45884,"kb_used_data":14696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325956,"statfs":{"total":96636764160,"available":96589778944,"internally_reserved":0,"allocated":15048704,"data_stored":23455151,"data_compressed":164710,"data_compressed_allocated":372736,"data_compressed_original":1191936,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":1,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.256,"15min":1.256},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":3.4590000000000001,"5min":3.4590000000000001,"15min":3.4590000000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":1.341,"5min":1.341,"15min":1.341},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":2.7280000000000002,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":1.1479999999999999}]},{"osd":2,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":2.2080000000000002,"5min":2.2080000000000002,"15min":2.2080000000000002},"last":0.85699999999999998},{"interface":"front","average":{"1min":1.218,"5min":1.218,"15min":1.218},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":3.0139999999999998,"5min":3.0139999999999998,"15min":3.0139999999999998},"last":1.0720000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0449999999999999}]},{"osd":4,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.222,"15min":1.222},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":3.5979999999999999,"5min":3.5979999999999999,"15min":3.5979999999999999},"last":0.82799999999999996},{"interface":"front","average":{"1min":1.321,"5min":1.321,"15min":1.321},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":3.383,"5min":3.383,"15min":3.383},"last":0.94299999999999995}]},{"osd":5,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.466,"15min":1.466},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":3.5059999999999998,"5min":3.5059999999999998,"15min":3.5059999999999998},"last":0.89800000000000002},{"interface":"front","average":{"1min":1.5029999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":3.1579999999999999,"5min":3.1579999999999999,"15min":3.1579999999999999},"last":1.109}]},{"osd":6,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.625,"5min":1.625,"15min":1.625},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":3.4820000000000002,"5min":3.4820000000000002,"15min":3.4820000000000002},"last":0.998},{"interface":"front","average":{"1min":1.579,"5min":1.579,"15min":1.579},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":3.0059999999999998,"5min":3.0059999999999998,"15min":3.0059999999999998},"last":0.75900000000000001}]},{"osd":7,"last update":"Sun Nov 5 08:33:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"min":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"max":{"1min":2.165,"5min":2.165,"15min":2.165},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.54,"5min":1.54,"15min":1.54},"min":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"max":{"1min":2.2759999999999998,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":1.3400000000000001}]},{"osd":8,"last update":"Sun Nov 5 08:33:52 2023","interfaces":[{"interface":"back","average":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.6850000000000001},"min":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":3.5739999999999998,"5min":3.5739999999999998,"15min":3.5739999999999998},"last":1.212},{"interface":"front","average":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"min":{"1min":1.044,"5min":1.044,"15min":1.044},"max":{"1min":2.9569999999999999,"5min":2.9569999999999999,"15min":2.9569999999999999},"last":0.97499999999999998}]},{"osd":9,"last update":"Sun Nov 5 08:33:52 2023","interfaces":[{"interface":"back","average":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":1.7849999999999999},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":4.6520000000000001,"5min":4.6520000000000001,"15min":4.6520000000000001},"last":1.0940000000000001},{"interface":"front","average":{"1min":2.0720000000000001,"5min":2.0720000000000001,"15min":2.0720000000000001},"min":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"max":{"1min":4.7450000000000001,"5min":4.7450000000000001,"15min":4.7450000000000001},"last":1.129}]},{"osd":11,"last update":"Sun Nov 5 08:33:48 2023","interfaces":[{"interface":"back","average":{"1min":1.6619999999999999,"5min":1.6619999999999999,"15min":1.6619999999999999},"min":{"1min":1.2,"5min":1.2,"15min":1.2},"max":{"1min":2.5129999999999999,"5min":2.5129999999999999,"15min":2.5129999999999999},"last":1.3939999999999999},{"interface":"front","average":{"1min":1.7909999999999999,"5min":1.7909999999999999,"15min":1.7909999999999999},"min":{"1min":1.056,"5min":1.056,"15min":1.056},"max":{"1min":3.6619999999999999,"5min":3.6619999999999999,"15min":3.6619999999999999},"last":1.3129999999999999}]},{"osd":12,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.962,"5min":1.962,"15min":1.962},"min":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"max":{"1min":5.7270000000000003,"5min":5.7270000000000003,"15min":5.7270000000000003},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.9239999999999999,"5min":1.9239999999999999,"15min":1.9239999999999999},"min":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"max":{"1min":6.093,"5min":6.093,"15min":6.093},"last":1.2849999999999999}]},{"osd":14,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.9470000000000001,"5min":1.9470000000000001,"15min":1.9470000000000001},"min":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"max":{"1min":3.7050000000000001,"5min":3.7050000000000001,"15min":3.7050000000000001},"last":1.264},{"interface":"front","average":{"1min":1.982,"5min":1.982,"15min":1.982},"min":{"1min":1.099,"5min":1.099,"15min":1.099},"max":{"1min":3.2349999999999999,"5min":3.2349999999999999,"15min":3.2349999999999999},"last":1.0329999999999999}]},{"osd":15,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.9370000000000001,"5min":1.9370000000000001,"15min":1.9370000000000001},"min":{"1min":1.155,"5min":1.155,"15min":1.155},"max":{"1min":3.048,"5min":3.048,"15min":3.048},"last":1.242},{"interface":"front","average":{"1min":2.129,"5min":2.129,"15min":2.129},"min":{"1min":1.135,"5min":1.135,"15min":1.135},"max":{"1min":5.3920000000000003,"5min":5.3920000000000003,"15min":5.3920000000000003},"last":1.3620000000000001}]}]},{"osd":15,"up_from":21,"seq":90194313238,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37980,"kb_used_data":10956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94333860,"statfs":{"total":96636764160,"available":96597872640,"internally_reserved":0,"allocated":11218944,"data_stored":16767293,"data_compressed":165545,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,4,5,6,7,8,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":1,"last update":"Sun Nov 5 08:33:55 2023","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.139,"15min":1.139},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":3.0329999999999999,"5min":3.0329999999999999,"15min":3.0329999999999999},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":8.6170000000000009,"5min":8.6170000000000009,"15min":8.6170000000000009},"last":0.435}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.131}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96399999999999997}]},{"osd":4,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.125,"15min":1.125},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":2.4649999999999999,"5min":2.4649999999999999,"15min":2.4649999999999999},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":2.331,"5min":2.331,"15min":2.331},"last":0.67200000000000004}]},{"osd":5,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.113,"15min":1.113},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":2.9460000000000002,"5min":2.9460000000000002,"15min":2.9460000000000002},"last":0.48399999999999999},{"interface":"front","average":{"1min":1.452,"5min":1.452,"15min":1.452},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":3.8849999999999998,"5min":3.8849999999999998,"15min":3.8849999999999998},"last":1.042}]},{"osd":6,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.119,"15min":1.119},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":2.4769999999999999,"5min":2.4769999999999999,"15min":2.4769999999999999},"last":0.84199999999999997},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":3.0880000000000001,"5min":3.0880000000000001,"15min":3.0880000000000001},"last":0.88800000000000001}]},{"osd":7,"last update":"Sun Nov 5 08:33:44 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.169,"15min":1.169},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.996,"5min":2.996,"15min":2.996},"last":1.0600000000000001},{"interface":"front","average":{"1min":1.389,"5min":1.389,"15min":1.389},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":5.444,"5min":5.444,"15min":5.444},"last":0.54300000000000004}]},{"osd":8,"last update":"Sun Nov 5 08:33:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":3.7389999999999999,"5min":3.7389999999999999,"15min":3.7389999999999999},"last":1.1970000000000001},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":4.2489999999999997,"5min":4.2489999999999997,"15min":4.2489999999999997},"last":0.97999999999999998}]},{"osd":9,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.5900000000000001,"5min":1.5900000000000001,"15min":1.5900000000000001},"min":{"1min":1.044,"5min":1.044,"15min":1.044},"max":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"last":1.2330000000000001},{"interface":"front","average":{"1min":1.883,"5min":1.883,"15min":1.883},"min":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"max":{"1min":13.029999999999999,"5min":13.029999999999999,"15min":13.029999999999999},"last":0.77600000000000002}]},{"osd":11,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"min":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":2.9590000000000001,"5min":2.9590000000000001,"15min":2.9590000000000001},"last":1.1599999999999999},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"min":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"max":{"1min":2.6469999999999998,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":0.72699999999999998}]},{"osd":13,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.591,"5min":1.591,"15min":1.591},"min":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"max":{"1min":3.3580000000000001,"5min":3.3580000000000001,"15min":3.3580000000000001},"last":1.2709999999999999},{"interface":"front","average":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":1.7849999999999999},"min":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"max":{"1min":5.1379999999999999,"5min":5.1379999999999999,"15min":5.1379999999999999},"last":1.018}]},{"osd":14,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.718,"5min":1.718,"15min":1.718},"min":{"1min":0.999,"5min":0.999,"15min":0.999},"max":{"1min":5.7720000000000002,"5min":5.7720000000000002,"15min":5.7720000000000002},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.696,"5min":1.696,"15min":1.696},"min":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"max":{"1min":7.5039999999999996,"5min":7.5039999999999996,"15min":7.5039999999999996},"last":0.94299999999999995}]}]},{"osd":14,"up_from":22,"seq":94489280533,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40056,"kb_used_data":13036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331784,"statfs":{"total":96636764160,"available":96595746816,"internally_reserved":0,"allocated":13348864,"data_stored":25578375,"data_compressed":165545,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":1,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":1.45},{"interface":"front","average":{"1min":1.131,"5min":1.131,"15min":1.131},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.732,"5min":1.732,"15min":1.732},"last":0.78500000000000003}]},{"osd":2,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.86099999999999999},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.839,"5min":1.839,"15min":1.839},"last":1.179}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3149999999999999}]},{"osd":4,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":1.0640000000000001},{"interface":"front","average":{"1min":0.997,"5min":0.997,"15min":0.997},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.391,"5min":1.391,"15min":1.391},"last":1.347}]},{"osd":5,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.101,"5min":1.101,"15min":1.101},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.605,"5min":1.605,"15min":1.605},"last":0.92400000000000004}]},{"osd":6,"last update":"Sun Nov 5 08:33:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.575,"5min":1.575,"15min":1.575},"last":0.59799999999999998},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.9610000000000001,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.97299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.133}]},{"osd":8,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.7050000000000001,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.22,"5min":1.22,"15min":1.22},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":1.242}]},{"osd":10,"last update":"Sun Nov 5 08:33:44 2023","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.29,"15min":1.29},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.802,"5min":1.802,"15min":1.802},"last":1.028},{"interface":"front","average":{"1min":1.325,"5min":1.325,"15min":1.325},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.8049999999999999,"5min":1.8049999999999999,"15min":1.8049999999999999},"last":1.528}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.429}]},{"osd":12,"last update":"Sun Nov 5 08:33:53 2023","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":1.4430000000000001},"min":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"max":{"1min":2.2389999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.153},{"interface":"front","average":{"1min":1.544,"5min":1.544,"15min":1.544},"min":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"max":{"1min":2.1320000000000001,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":1.478}]},{"osd":13,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.5900000000000001,"5min":1.5900000000000001,"15min":1.5900000000000001},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":2.4590000000000001,"5min":2.4590000000000001,"15min":2.4590000000000001},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"min":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":2.028,"5min":2.028,"15min":2.028},"last":1.214}]},{"osd":15,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":2.2429999999999999,"5min":2.2429999999999999,"15min":2.2429999999999999},"last":1.4139999999999999},{"interface":"front","average":{"1min":1.6399999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":2.3559999999999999,"5min":2.3559999999999999,"15min":2.3559999999999999},"last":1.548}]}]},{"osd":0,"up_from":21,"seq":90194313223,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":4392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":4497408,"data_stored":4339168,"data_compressed":21230,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,9,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]}]},{"osd":3,"up_from":93,"seq":399431958532,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38104,"kb_used_data":9924,"kb_used_omap":6,"kb_used_meta":28153,"kb_avail":94333736,"statfs":{"total":96636764160,"available":96597745664,"internally_reserved":0,"allocated":10162176,"data_stored":12466071,"data_compressed":161651,"data_compressed_allocated":348160,"data_compressed_original":696320,"omap_allocated":6448,"internal_metadata":28829392},"hb_peers":[2,4,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2589999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91100000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2250000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4350000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4790000000000001}]}]},{"osd":6,"up_from":21,"seq":90194313237,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47140,"kb_used_data":15960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324700,"statfs":{"total":96636764160,"available":96588492800,"internally_reserved":0,"allocated":16343040,"data_stored":21761825,"data_compressed":163600,"data_compressed_allocated":352256,"data_compressed_original":704512,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,3,4,5,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.02,"15min":1.02},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":0.84999999999999998},{"interface":"front","average":{"1min":1.153,"5min":1.153,"15min":1.153},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.899,"5min":1.899,"15min":1.899},"last":1.0569999999999999}]},{"osd":2,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.76000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4370000000000001}]},{"osd":4,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.038,"15min":1.038},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":0.874},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.617,"5min":1.617,"15min":1.617},"last":0.64200000000000002}]},{"osd":5,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.532,"5min":1.532,"15min":1.532},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.212,"5min":1.212,"15min":1.212},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.577,"5min":1.577,"15min":1.577},"last":0.92100000000000004}]},{"osd":7,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.72,"5min":1.72,"15min":1.72},"last":0.95399999999999996},{"interface":"front","average":{"1min":1.069,"5min":1.069,"15min":1.069},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.6859999999999999,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":1.292}]},{"osd":8,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.958,"5min":1.958,"15min":1.958},"last":0.59999999999999998},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.772,"5min":1.772,"15min":1.772},"last":1.724}]},{"osd":9,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.484},{"interface":"front","average":{"1min":1.306,"5min":1.306,"15min":1.306},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.7749999999999999,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":1.26}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3999999999999999}]},{"osd":11,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.379,"15min":1.379},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":2.0579999999999998,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":1.093},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":2.1930000000000001,"5min":2.1930000000000001,"15min":2.1930000000000001},"last":1.5249999999999999}]},{"osd":12,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.548,"5min":1.548,"15min":1.548},"min":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":2.1139999999999999,"5min":2.1139999999999999,"15min":2.1139999999999999},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.841,"5min":1.841,"15min":1.841},"last":1.1379999999999999}]},{"osd":13,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":1.6930000000000001,"5min":1.6930000000000001,"15min":1.6930000000000001},"min":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"max":{"1min":2.2759999999999998,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":1.589},{"interface":"front","average":{"1min":1.532,"5min":1.532,"15min":1.532},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.9710000000000001,"5min":1.9710000000000001,"15min":1.9710000000000001},"last":1.5660000000000001}]},{"osd":14,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":2.319,"5min":2.319,"15min":2.319},"last":1.327},{"interface":"front","average":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":2.4870000000000001,"5min":2.4870000000000001,"15min":2.4870000000000001},"last":1.6559999999999999}]},{"osd":15,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.679,"5min":1.679,"15min":1.679},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.4009999999999998,"5min":2.4009999999999998,"15min":2.4009999999999998},"last":1.1970000000000001},{"interface":"front","average":{"1min":1.482,"5min":1.482,"15min":1.482},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"last":1.623}]}]},{"osd":1,"up_from":22,"seq":94489280533,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46080,"kb_used_data":14900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325760,"statfs":{"total":96636764160,"available":96589578240,"internally_reserved":0,"allocated":15257600,"data_stored":21203847,"data_compressed":165545,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,5,6,7,8,9,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":2,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.272,"5min":1.272,"15min":1.272},"last":0.84899999999999998},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.1240000000000001,"5min":2.1240000000000001,"15min":2.1240000000000001},"last":0.77100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.873}]},{"osd":4,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.6100000000000001,"5min":1.6100000000000001,"15min":1.6100000000000001},"last":1.117}]},{"osd":5,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.071,"15min":1.071},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":2.6110000000000002,"5min":2.6110000000000002,"15min":2.6110000000000002},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.7310000000000001,"5min":1.7310000000000001,"15min":1.7310000000000001},"last":1.349}]},{"osd":6,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.135,"15min":1.135},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.61599999999999999},{"interface":"front","average":{"1min":1.246,"5min":1.246,"15min":1.246},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.1759999999999999}]},{"osd":7,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.282,"15min":1.282},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.708,"5min":1.708,"15min":1.708},"last":0.82599999999999996},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":2.3090000000000002,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":1.254}]},{"osd":8,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.288,"15min":1.288},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":2.5630000000000002,"5min":2.5630000000000002,"15min":2.5630000000000002},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.238,"5min":1.238,"15min":1.238},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":1.397}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5089999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.46}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3109999999999999}]},{"osd":14,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"min":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"max":{"1min":2.2320000000000002,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":1.4810000000000001},{"interface":"front","average":{"1min":1.498,"5min":1.498,"15min":1.498},"min":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"max":{"1min":2.7610000000000001,"5min":2.7610000000000001,"15min":2.7610000000000001},"last":1.038}]},{"osd":15,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.5640000000000001,"5min":1.5640000000000001,"15min":1.5640000000000001},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":2.7250000000000001,"5min":2.7250000000000001,"15min":2.7250000000000001},"last":1.55},{"interface":"front","average":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"min":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":1.655}]}]},{"osd":2,"up_from":22,"seq":94489280533,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42868,"kb_used_data":11688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328972,"statfs":{"total":96636764160,"available":96592867328,"internally_reserved":0,"allocated":11968512,"data_stored":20028839,"data_compressed":172494,"data_compressed_allocated":389120,"data_compressed_original":1224704,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,3,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.5529999999999999,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.482,"5min":1.482,"15min":1.482},"last":0.94699999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94199999999999995}]},{"osd":5,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.5029999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.79800000000000004}]},{"osd":6,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.5},{"interface":"front","average":{"1min":1.077,"5min":1.077,"15min":1.077},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":0.32700000000000001}]},{"osd":7,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.167,"15min":1.167},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":2.0880000000000001,"5min":2.0880000000000001,"15min":2.0880000000000001},"last":0.629},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.744,"5min":1.744,"15min":1.744},"last":0.81499999999999995}]},{"osd":8,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.831,"5min":1.831,"15min":1.831},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.129,"5min":1.129,"15min":1.129},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":2.1859999999999999,"5min":2.1859999999999999,"15min":2.1859999999999999},"last":0.40500000000000003}]},{"osd":9,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.7969999999999999,"5min":1.7969999999999999,"15min":1.7969999999999999},"last":0.84499999999999997},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":2.27,"5min":2.27,"15min":2.27},"last":0.70499999999999996}]},{"osd":10,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":2.976,"5min":2.976,"15min":2.976},"last":0.44800000000000001},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":2.5739999999999998,"5min":2.5739999999999998,"15min":2.5739999999999998},"last":0.91000000000000003}]},{"osd":11,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.9009999999999998,"5min":2.9009999999999998,"15min":2.9009999999999998},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":2.6859999999999999,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.73499999999999999}]},{"osd":12,"last update":"Sun Nov 5 08:33:45 2023","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.2,"15min":1.2},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.75600000000000001}]}]},{"osd":4,"up_from":22,"seq":94489280533,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44584,"kb_used_data":13400,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327256,"statfs":{"total":96636764160,"available":96591110144,"internally_reserved":0,"allocated":13721600,"data_stored":19343677,"data_compressed":165545,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,5,6,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":1,"last update":"Sun Nov 5 08:33:43 2023","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.6659999999999999,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":0.995}]},{"osd":2,"last update":"Sun Nov 5 08:33:43 2023","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":0.82399999999999995},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.9850000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":1.4790000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2849999999999999}]},{"osd":5,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.0299999999999998,"5min":2.0299999999999998,"15min":2.0299999999999998},"last":1.24},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.7,"5min":1.7,"15min":1.7},"last":1.4390000000000001}]},{"osd":6,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":0.995,"15min":0.995},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.7290000000000001,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":1.238,"5min":1.238,"15min":1.238},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":2.2789999999999999,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":1.101}]},{"osd":7,"last update":"Sun Nov 5 08:33:43 2023","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.889,"5min":1.889,"15min":1.889},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":1.514}]},{"osd":8,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.171,"15min":1.171},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":2.2490000000000001,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.7789999999999999,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":1.1839999999999999}]},{"osd":9,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":2.5249999999999999,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":1.359},{"interface":"front","average":{"1min":1.468,"5min":1.468,"15min":1.468},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.3290000000000002,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":1.647}]},{"osd":10,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":2.0459999999999998,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.401},{"interface":"front","average":{"1min":1.508,"5min":1.508,"15min":1.508},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.3919999999999999,"5min":2.3919999999999999,"15min":2.3919999999999999},"last":1.569}]},{"osd":11,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.431,"15min":1.431},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.1280000000000001,"5min":2.1280000000000001,"15min":2.1280000000000001},"last":1.3260000000000001},{"interface":"front","average":{"1min":1.3,"5min":1.3,"15min":1.3},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.9379999999999999,"5min":1.9379999999999999,"15min":1.9379999999999999},"last":2.0960000000000001}]},{"osd":12,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.3,"5min":1.3,"15min":1.3},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":2.161,"5min":2.161,"15min":2.161},"last":1.9219999999999999}]},{"osd":13,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.5640000000000001,"5min":1.5640000000000001,"15min":1.5640000000000001},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":2.3420000000000001,"5min":2.3420000000000001,"15min":2.3420000000000001},"last":2.036},{"interface":"front","average":{"1min":1.538,"5min":1.538,"15min":1.538},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":2.3140000000000001,"5min":2.3140000000000001,"15min":2.3140000000000001},"last":1.6890000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5109999999999999}]},{"osd":15,"last update":"Sun Nov 5 08:33:57 2023","interfaces":[{"interface":"back","average":{"1min":1.6990000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":2.5750000000000002,"5min":2.5750000000000002,"15min":2.5750000000000002},"last":1.6080000000000001},{"interface":"front","average":{"1min":1.6719999999999999,"5min":1.6719999999999999,"15min":1.6719999999999999},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":2.3940000000000001,"5min":2.3940000000000001,"15min":2.3940000000000001},"last":1.879}]}]},{"osd":5,"up_from":22,"seq":94489280533,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34200,"kb_used_data":7180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337640,"statfs":{"total":96636764160,"available":96601743360,"internally_reserved":0,"allocated":7352320,"data_stored":8196459,"data_compressed":163600,"data_compressed_allocated":352256,"data_compressed_original":704512,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,6,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":1,"last update":"Sun Nov 5 08:33:45 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.97599999999999998}]},{"osd":2,"last update":"Sun Nov 5 08:33:45 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.575,"5min":1.575,"15min":1.575},"last":1.3939999999999999},{"interface":"front","average":{"1min":1.04,"5min":1.04,"15min":1.04},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.546,"5min":1.546,"15min":1.546},"last":1.26}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":4,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.645,"5min":1.645,"15min":1.645},"last":1.831},{"interface":"front","average":{"1min":1.036,"5min":1.036,"15min":1.036},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.77,"5min":1.77,"15min":1.77},"last":1.47}]},{"osd":6,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":1.3620000000000001},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":1.512}]},{"osd":7,"last update":"Sun Nov 5 08:33:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.696,"5min":1.696,"15min":1.696},"last":1.3089999999999999},{"interface":"front","average":{"1min":1.177,"5min":1.177,"15min":1.177},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":1.212}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5760000000000001}]},{"osd":9,"last update":"Sun Nov 5 08:33:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.3,"15min":1.3},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.6990000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":1.4359999999999999},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"min":{"1min":0.88,"5min":0.88,"15min":0.88},"max":{"1min":1.8180000000000001,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":1.931}]},{"osd":10,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.7909999999999999},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":1.8799999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4990000000000001}]},{"osd":12,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":1.635,"15min":1.635},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":3.488,"5min":3.488,"15min":3.488},"last":2.0089999999999999},{"interface":"front","average":{"1min":1.46,"5min":1.46,"15min":1.46},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.1970000000000001,"5min":2.1970000000000001,"15min":2.1970000000000001},"last":1.9690000000000001}]},{"osd":13,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.6799999999999999,"5min":1.6799999999999999,"15min":1.6799999999999999},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":3.714,"5min":3.714,"15min":3.714},"last":2.1819999999999999},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.5649999999999999,"15min":1.5649999999999999},"min":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"max":{"1min":2.121,"5min":2.121,"15min":2.121},"last":2.0369999999999999}]},{"osd":14,"last update":"Sun Nov 5 08:33:45 2023","interfaces":[{"interface":"back","average":{"1min":1.718,"5min":1.718,"15min":1.718},"min":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"max":{"1min":2.1920000000000002,"5min":2.1920000000000002,"15min":2.1920000000000002},"last":2.1309999999999998},{"interface":"front","average":{"1min":1.647,"5min":1.647,"15min":1.647},"min":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"max":{"1min":2.2930000000000001,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":2.1059999999999999}]},{"osd":15,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.974,"5min":1.974,"15min":1.974},"last":1.637},{"interface":"front","average":{"1min":1.8140000000000001,"5min":1.8140000000000001,"15min":1.8140000000000001},"min":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"max":{"1min":3.5870000000000002,"5min":3.5870000000000002,"15min":3.5870000000000002},"last":2.0600000000000001}]}]},{"osd":7,"up_from":22,"seq":94489280534,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39656,"kb_used_data":12632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94332184,"statfs":{"total":96636764160,"available":96596156416,"internally_reserved":0,"allocated":12935168,"data_stored":17976199,"data_compressed":165545,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1930000000000001}]},{"osd":1,"last update":"Sun Nov 5 08:33:47 2023","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":2.1360000000000001,"5min":2.1360000000000001,"15min":2.1360000000000001},"last":1.258},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.85299999999999998}]},{"osd":2,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.401,"15min":1.401},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":1.889,"5min":1.889,"15min":1.889},"last":1.875},{"interface":"front","average":{"1min":1.238,"5min":1.238,"15min":1.238},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.663,"5min":1.663,"15min":1.663},"last":1.349}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4370000000000001}]},{"osd":4,"last update":"Sun Nov 5 08:33:30 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.095,"15min":1.095},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.7949999999999999,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":2.294,"5min":2.294,"15min":2.294},"last":1.0840000000000001}]},{"osd":5,"last update":"Sun Nov 5 08:33:30 2023","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.591,"5min":1.591,"15min":1.591},"last":1.5740000000000001},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.726,"5min":1.726,"15min":1.726},"last":1.302}]},{"osd":6,"last update":"Sun Nov 5 08:33:30 2023","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.232,"15min":1.232},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.6040000000000001,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":1.7689999999999999},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":2.0510000000000002,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.7170000000000001}]},{"osd":8,"last update":"Sun Nov 5 08:33:30 2023","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.381,"5min":1.381,"15min":1.381},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.48,"5min":2.48,"15min":2.48},"last":1.657}]},{"osd":9,"last update":"Sun Nov 5 08:33:47 2023","interfaces":[{"interface":"back","average":{"1min":1.6799999999999999,"5min":1.6799999999999999,"15min":1.6799999999999999},"min":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"max":{"1min":2.823,"5min":2.823,"15min":2.823},"last":2.0099999999999998},{"interface":"front","average":{"1min":1.48,"5min":1.48,"15min":1.48},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":2.3450000000000002,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":1.9350000000000001}]},{"osd":10,"last update":"Sun Nov 5 08:33:30 2023","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":1.478,"15min":1.478},"min":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"max":{"1min":1.931,"5min":1.931,"15min":1.931},"last":1.397},{"interface":"front","average":{"1min":1.671,"5min":1.671,"15min":1.671},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.5550000000000002,"5min":2.5550000000000002,"15min":2.5550000000000002},"last":2.1539999999999999}]},{"osd":11,"last update":"Sun Nov 5 08:33:47 2023","interfaces":[{"interface":"back","average":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":2.093,"5min":2.093,"15min":2.093},"last":1.6220000000000001},{"interface":"front","average":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":2.6600000000000001,"5min":2.6600000000000001,"15min":2.6600000000000001},"last":2.2069999999999999}]},{"osd":12,"last update":"Sun Nov 5 08:33:52 2023","interfaces":[{"interface":"back","average":{"1min":1.841,"5min":1.841,"15min":1.841},"min":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"max":{"1min":2.7250000000000001,"5min":2.7250000000000001,"15min":2.7250000000000001},"last":2.306},{"interface":"front","average":{"1min":1.639,"5min":1.639,"15min":1.639},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.3780000000000001,"5min":2.3780000000000001,"15min":2.3780000000000001},"last":2.1880000000000002}]},{"osd":13,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":1.667,"5min":1.667,"15min":1.667},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.589,"5min":2.589,"15min":2.589},"last":2.0430000000000001},{"interface":"front","average":{"1min":1.625,"5min":1.625,"15min":1.625},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":2.3050000000000002,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":1.9690000000000001}]},{"osd":14,"last update":"Sun Nov 5 08:33:30 2023","interfaces":[{"interface":"back","average":{"1min":1.75,"5min":1.75,"15min":1.75},"min":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":2.4910000000000001,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":2.1200000000000001},{"interface":"front","average":{"1min":1.7529999999999999,"5min":1.7529999999999999,"15min":1.7529999999999999},"min":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"max":{"1min":2.9750000000000001,"5min":2.9750000000000001,"15min":2.9750000000000001},"last":1.823}]},{"osd":15,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":1.7010000000000001,"5min":1.7010000000000001,"15min":1.7010000000000001},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":3.1760000000000002,"5min":3.1760000000000002,"15min":3.1760000000000002},"last":2.0859999999999999},{"interface":"front","average":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":2.73,"5min":2.73,"15min":2.73},"last":2.258}]}]},{"osd":8,"up_from":22,"seq":94489280533,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50488,"kb_used_data":19304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321352,"statfs":{"total":96636764160,"available":96585064448,"internally_reserved":0,"allocated":19767296,"data_stored":34982791,"data_compressed":165545,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":1,"last update":"Sun Nov 5 08:33:46 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":1.3140000000000001},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":1.22}]},{"osd":2,"last update":"Sun Nov 5 08:33:46 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":1.1559999999999999},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":1.4950000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4330000000000001}]},{"osd":4,"last update":"Sun Nov 5 08:33:46 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":1.585},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":1.524}]},{"osd":5,"last update":"Sun Nov 5 08:33:33 2023","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.786,"5min":1.786,"15min":1.786},"last":1.6950000000000001},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":1.8280000000000001}]},{"osd":6,"last update":"Sun Nov 5 08:33:33 2023","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.6799999999999999,"5min":1.6799999999999999,"15min":1.6799999999999999},"last":1.7889999999999999},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":1.875}]},{"osd":7,"last update":"Sun Nov 5 08:33:33 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.556,"5min":1.556,"15min":1.556},"last":1.7430000000000001},{"interface":"front","average":{"1min":1.214,"5min":1.214,"15min":1.214},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":2.157,"5min":2.157,"15min":2.157},"last":1.9510000000000001}]},{"osd":9,"last update":"Sun Nov 5 08:33:33 2023","interfaces":[{"interface":"back","average":{"1min":2.1509999999999998,"5min":2.1509999999999998,"15min":2.1509999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":19.849,"5min":19.849,"15min":19.849},"last":2.234},{"interface":"front","average":{"1min":2.2170000000000001,"5min":2.2170000000000001,"15min":2.2170000000000001},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":19.867999999999999,"5min":19.867999999999999,"15min":19.867999999999999},"last":1.917}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]},{"osd":11,"last update":"Sun Nov 5 08:33:33 2023","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.298,"15min":1.298},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":2.0710000000000002,"5min":2.0710000000000002,"15min":2.0710000000000002},"last":2.21},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":2.0960000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":13,"last update":"Sun Nov 5 08:33:33 2023","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":2.0670000000000002},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.8560000000000001,"5min":1.8560000000000001,"15min":1.8560000000000001},"last":2.0310000000000001}]},{"osd":14,"last update":"Sun Nov 5 08:33:33 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":2.226,"5min":2.226,"15min":2.226},"last":1.9930000000000001},{"interface":"front","average":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":1.4430000000000001},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.891,"5min":1.891,"15min":1.891},"last":2.266}]},{"osd":15,"last update":"Sun Nov 5 08:33:55 2023","interfaces":[{"interface":"back","average":{"1min":1.552,"5min":1.552,"15min":1.552},"min":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"max":{"1min":2.1629999999999998,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":2.1360000000000001},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.4490000000000001,"15min":1.4490000000000001},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.1970000000000001,"5min":2.1970000000000001,"15min":2.1970000000000001},"last":2.169}]}]},{"osd":9,"up_from":21,"seq":90194313237,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42728,"kb_used_data":11544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329112,"statfs":{"total":96636764160,"available":96593010688,"internally_reserved":0,"allocated":11821056,"data_stored":14945159,"data_compressed":165545,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,6,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":1,"last update":"Sun Nov 5 08:34:00 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.506,"5min":1.506,"15min":1.506},"last":0.75},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.734,"5min":1.734,"15min":1.734},"last":1.0640000000000001}]},{"osd":2,"last update":"Sun Nov 5 08:34:00 2023","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.81200000000000006},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.5620000000000001,"5min":1.5620000000000001,"15min":1.5620000000000001},"last":0.94499999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.23}]},{"osd":4,"last update":"Sun Nov 5 08:34:00 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.095,"15min":1.095},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.478,"5min":1.478,"15min":1.478},"last":0.86499999999999999},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.595,"5min":1.595,"15min":1.595},"last":1.52}]},{"osd":6,"last update":"Sun Nov 5 08:33:32 2023","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.101,"15min":1.101},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":1.641,"5min":1.641,"15min":1.641},"last":1.1399999999999999},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.486,"5min":1.486,"15min":1.486},"last":1.3740000000000001}]},{"osd":7,"last update":"Sun Nov 5 08:34:00 2023","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"min":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"max":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.482},{"interface":"front","average":{"1min":1.369,"5min":1.369,"15min":1.369},"min":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":1.925,"5min":1.925,"15min":1.925},"last":1.6599999999999999}]},{"osd":8,"last update":"Sun Nov 5 08:33:32 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.202,"15min":1.202},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":1.4179999999999999},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.944,"5min":1.944,"15min":1.944},"last":1.617}]},{"osd":10,"last update":"Sun Nov 5 08:33:32 2023","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.752,"5min":1.752,"15min":1.752},"last":1.1870000000000001},{"interface":"front","average":{"1min":1.319,"5min":1.319,"15min":1.319},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":2.1240000000000001,"5min":2.1240000000000001,"15min":2.1240000000000001},"last":1.002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2669999999999999}]},{"osd":12,"last update":"Sun Nov 5 08:33:32 2023","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.381,"15min":1.381},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":2.0710000000000002,"5min":2.0710000000000002,"15min":2.0710000000000002},"last":1.2709999999999999},{"interface":"front","average":{"1min":1.462,"5min":1.462,"15min":1.462},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":2.0350000000000001,"5min":2.0350000000000001,"15min":2.0350000000000001},"last":1.4550000000000001}]},{"osd":13,"last update":"Sun Nov 5 08:33:32 2023","interfaces":[{"interface":"back","average":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"min":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"max":{"1min":2.8010000000000002,"5min":2.8010000000000002,"15min":2.8010000000000002},"last":1.748},{"interface":"front","average":{"1min":1.478,"5min":1.478,"15min":1.478},"min":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"max":{"1min":2.177,"5min":2.177,"15min":2.177},"last":1.7150000000000001}]},{"osd":14,"last update":"Sun Nov 5 08:33:32 2023","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.506,"15min":1.506},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.849,"5min":1.849,"15min":1.849},"last":1.772},{"interface":"front","average":{"1min":1.6699999999999999,"5min":1.6699999999999999,"15min":1.6699999999999999},"min":{"1min":1.294,"5min":1.294,"15min":1.294},"max":{"1min":2.9710000000000001,"5min":2.9710000000000001,"15min":2.9710000000000001},"last":1.591}]},{"osd":15,"last update":"Sun Nov 5 08:33:32 2023","interfaces":[{"interface":"back","average":{"1min":1.603,"5min":1.603,"15min":1.603},"min":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"max":{"1min":1.9670000000000001,"5min":1.9670000000000001,"15min":1.9670000000000001},"last":1.821},{"interface":"front","average":{"1min":1.716,"5min":1.716,"15min":1.716},"min":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"max":{"1min":3.0950000000000002,"5min":3.0950000000000002,"15min":3.0950000000000002},"last":1.6919999999999999}]}]},{"osd":10,"up_from":22,"seq":94489280533,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41780,"kb_used_data":14760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330060,"statfs":{"total":96636764160,"available":96593981440,"internally_reserved":0,"allocated":15114240,"data_stored":26340231,"data_compressed":165545,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,4,5,6,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":1,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.86399999999999999}]},{"osd":2,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.768,"5min":1.768,"15min":1.768},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.01,"5min":1.01,"15min":1.01},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":2.9729999999999999,"5min":2.9729999999999999,"15min":2.9729999999999999},"last":0.81699999999999995}]},{"osd":4,"last update":"Sun Nov 5 08:33:57 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":1.0600000000000001},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.482,"5min":1.482,"15min":1.482},"last":1.177}]},{"osd":5,"last update":"Sun Nov 5 08:33:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":3.4790000000000001,"5min":3.4790000000000001,"15min":3.4790000000000001},"last":1.151},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":3.5339999999999998,"5min":3.5339999999999998,"15min":3.5339999999999998},"last":0.90600000000000003}]},{"osd":6,"last update":"Sun Nov 5 08:33:57 2023","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.55,"5min":1.55,"15min":1.55},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.5469999999999999,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":1.0269999999999999}]},{"osd":8,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.105,"15min":1.105},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.661,"5min":1.661,"15min":1.661},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.673,"5min":1.673,"15min":1.673},"last":1.0820000000000001}]},{"osd":9,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.264,"15min":1.264},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":2.8199999999999998,"5min":2.8199999999999998,"15min":2.8199999999999998},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.319,"5min":1.319,"15min":1.319},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":1.827,"5min":1.827,"15min":1.827},"last":1.28}]},{"osd":11,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":1.877,"5min":1.877,"15min":1.877},"last":1.234},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":2.8889999999999998,"5min":2.8889999999999998,"15min":2.8889999999999998},"last":1.256}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":13,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.536,"5min":1.536,"15min":1.536},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":2.778,"5min":2.778,"15min":2.778},"last":1.355},{"interface":"front","average":{"1min":1.6200000000000001,"5min":1.6200000000000001,"15min":1.6200000000000001},"min":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"max":{"1min":3.681,"5min":3.681,"15min":3.681},"last":1.1279999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]},{"osd":15,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":2.0550000000000002,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":1.331},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.5289999999999999,"15min":1.5289999999999999},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.4620000000000002,"5min":2.4620000000000002,"15min":2.4620000000000002},"last":1.3080000000000001}]}]},{"osd":11,"up_from":21,"seq":90194313238,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47192,"kb_used_data":16008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324648,"statfs":{"total":96636764160,"available":96588439552,"internally_reserved":0,"allocated":16392192,"data_stored":25136007,"data_compressed":165545,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,4,5,6,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":1,"last update":"Sun Nov 5 08:33:53 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.044,"15min":1.044},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":1.069,"5min":1.069,"15min":1.069},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.85199999999999998}]},{"osd":2,"last update":"Sun Nov 5 08:33:48 2023","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.216,"15min":1.216},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":2.298,"5min":2.298,"15min":2.298},"last":1.117},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":2.9020000000000001,"5min":2.9020000000000001,"15min":2.9020000000000001},"last":1.1499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":4,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.167,"15min":1.167},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.7390000000000001,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":1.048},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.91900000000000004}]},{"osd":5,"last update":"Sun Nov 5 08:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.244,"15min":1.244},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":3.0459999999999998,"5min":3.0459999999999998,"15min":3.0459999999999998},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":3.3799999999999999,"5min":3.3799999999999999,"15min":3.3799999999999999},"last":1.3200000000000001}]},{"osd":6,"last update":"Sun Nov 5 08:33:30 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.244,"15min":1.244},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":2.0630000000000002,"5min":2.0630000000000002,"15min":2.0630000000000002},"last":1.4319999999999999},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":2.7770000000000001,"5min":2.7770000000000001,"15min":2.7770000000000001},"last":1.532}]},{"osd":7,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.76,"5min":1.76,"15min":1.76},"last":1.508},{"interface":"front","average":{"1min":1.395,"5min":1.395,"15min":1.395},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.909,"5min":1.909,"15min":1.909},"last":1.6850000000000001}]},{"osd":8,"last update":"Sun Nov 5 08:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.9079999999999999,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.948,"5min":1.948,"15min":1.948},"last":1.774}]},{"osd":9,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":2.089,"5min":2.089,"15min":2.089},"last":1.472},{"interface":"front","average":{"1min":1.5660000000000001,"5min":1.5660000000000001,"15min":1.5660000000000001},"min":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"max":{"1min":2.1720000000000002,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":1.6499999999999999}]},{"osd":10,"last update":"Sun Nov 5 08:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.599,"5min":1.599,"15min":1.599},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.6890000000000001,"5min":2.6890000000000001,"15min":2.6890000000000001},"last":1.7969999999999999},{"interface":"front","average":{"1min":1.371,"5min":1.371,"15min":1.371},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":2.3679999999999999,"5min":2.3679999999999999,"15min":2.3679999999999999},"last":1.569}]},{"osd":12,"last update":"Sun Nov 5 08:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"max":{"1min":2.423,"5min":2.423,"15min":2.423},"last":1.8660000000000001},{"interface":"front","average":{"1min":1.702,"5min":1.702,"15min":1.702},"min":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"max":{"1min":2.6160000000000001,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":1.75}]},{"osd":13,"last update":"Sun Nov 5 08:33:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.4450000000000001,"15min":1.4450000000000001},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":2.48,"5min":2.48,"15min":2.48},"last":1.623},{"interface":"front","average":{"1min":1.714,"5min":1.714,"15min":1.714},"min":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"max":{"1min":2.6219999999999999,"5min":2.6219999999999999,"15min":2.6219999999999999},"last":1.595}]},{"osd":14,"last update":"Sun Nov 5 08:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.8580000000000001,"5min":1.8580000000000001,"15min":1.8580000000000001},"min":{"1min":1.137,"5min":1.137,"15min":1.137},"max":{"1min":5.375,"5min":5.375,"15min":5.375},"last":1.724},{"interface":"front","average":{"1min":1.754,"5min":1.754,"15min":1.754},"min":{"1min":1.038,"5min":1.038,"15min":1.038},"max":{"1min":2.5369999999999999,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.9179999999999999}]},{"osd":15,"last update":"Sun Nov 5 08:33:30 2023","interfaces":[{"interface":"back","average":{"1min":2.125,"5min":2.125,"15min":2.125},"min":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":1.843},{"interface":"front","average":{"1min":1.762,"5min":1.762,"15min":1.762},"min":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"max":{"1min":5.2549999999999999,"5min":5.2549999999999999,"15min":5.2549999999999999},"last":1.819}]}]},{"osd":12,"up_from":22,"seq":94489280533,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34744,"kb_used_data":7724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337096,"statfs":{"total":96636764160,"available":96601186304,"internally_reserved":0,"allocated":7909376,"data_stored":10623293,"data_compressed":165545,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[2,3,4,5,6,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":1,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.603,"5min":1.603,"15min":1.603},"last":1.2909999999999999}]},{"osd":2,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":1.0309999999999999},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.64400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0980000000000001}]},{"osd":4,"last update":"Sun Nov 5 08:34:02 2023","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.429,"5min":1.429,"15min":1.429},"last":1.0680000000000001}]},{"osd":5,"last update":"Sun Nov 5 08:34:02 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":1.123},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":1.2809999999999999}]},{"osd":6,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.73299999999999998},{"interface":"front","average":{"1min":1.073,"5min":1.073,"15min":1.073},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":1.343}]},{"osd":9,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.208,"15min":1.208},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":2.2109999999999999,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":0.99099999999999999},{"interface":"front","average":{"1min":1.206,"5min":1.206,"15min":1.206},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.964,"5min":1.964,"15min":1.964},"last":0.96399999999999997}]},{"osd":11,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":2.1659999999999999,"5min":2.1659999999999999,"15min":2.1659999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.8680000000000001,"5min":1.8680000000000001,"15min":1.8680000000000001},"last":1.149}]},{"osd":13,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.25,"5min":2.25,"15min":2.25},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":2.399,"5min":2.399,"15min":2.399},"last":1.391}]},{"osd":14,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":2.1070000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.218},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":2.3420000000000001,"5min":2.3420000000000001,"15min":2.3420000000000001},"last":1.2450000000000001}]},{"osd":15,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.2959999999999998,"5min":2.2959999999999998,"15min":2.2959999999999998},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.4099999999999999,"15min":1.4099999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.873,"5min":1.873,"15min":1.873},"last":1.3169999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6949,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6949,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4165632,"data_stored":4165632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14594048,"data_stored":21921792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11247616,"data_stored":20656128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8929280,"data_stored":11505664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12468224,"data_stored":18362368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":6107136,"data_stored":7225344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15101952,"data_stored":20791296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12271616,"data_stored":18165760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":18612224,"data_stored":33574912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":10563584,"data_stored":13963264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":13856768,"data_stored":25358336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":15134720,"data_stored":24154112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":6656000,"data_stored":9641984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":13787136,"data_stored":21987328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":12091392,"data_stored":24596480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":9965568,"data_stored":15785984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-05T08:34:03.613 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2023-11-05T08:34:03.613 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-11-05T08:34:03.614 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2023-11-05T08:34:03.614 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2023-11-05T08:34:03.640 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:34:03.638+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:34:03.710 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:34:03.711 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:34:03.723 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-05T08:34:03.727 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-11-05T08:34:03.826 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:34:03.821+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:34:03.829 INFO:tasks.rados.rados.0.smithi090.stdout:384: done (4 left) 2023-11-05T08:34:03.829 INFO:tasks.rados.rados.0.smithi090.stdout:385: snap_create 2023-11-05T08:34:03.830 INFO:tasks.rados.rados.0.smithi090.stdout:376: finishing write tid 7 to smithi09039252-20 2023-11-05T08:34:03.830 INFO:tasks.rados.rados.0.smithi090.stdout:376: oid 20 updating version 0 to 54 2023-11-05T08:34:03.830 INFO:tasks.rados.rados.0.smithi090.stdout:376: oid 20 updating version 54 to 58 2023-11-05T08:34:03.831 INFO:tasks.rados.rados.0.smithi090.stdout:376: oid 20 version 58 is already newer than 55 2023-11-05T08:34:03.831 INFO:tasks.rados.rados.0.smithi090.stdout:376: oid 20 version 58 is already newer than 56 2023-11-05T08:34:03.831 INFO:tasks.rados.rados.0.smithi090.stdout:376: oid 20 version 58 is already newer than 53 2023-11-05T08:34:03.832 INFO:tasks.rados.rados.0.smithi090.stdout:376: oid 20 version 58 is already newer than 57 2023-11-05T08:34:03.832 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 20 v 58 (ObjNum 141 snap 38 seq_num 141) dirty exists 2023-11-05T08:34:03.832 INFO:tasks.rados.rados.0.smithi090.stdout:376: left oid 20 (ObjNum 141 snap 38 seq_num 141) 2023-11-05T08:34:03.832 INFO:tasks.rados.rados.0.smithi090.stdout:377: expect (ObjNum 93 snap 16 seq_num 93) 2023-11-05T08:34:03.897 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:34:03.898 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:34:03.913 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-11-05T08:34:03.927 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:34:03.920+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:34:03.944 INFO:tasks.rados.rados.0.smithi090.stdout:381: expect (ObjNum 114 snap 29 seq_num 114) 2023-11-05T08:34:03.962 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:34:03.962 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:34:03.963 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 110, 2023-11-05T08:34:03.963 INFO:teuthology.orchestra.run.smithi002.stdout: "fsid": "8f1fa9b5-06de-4da7-985f-30d4ad103e2d", 2023-11-05T08:34:03.963 INFO:teuthology.orchestra.run.smithi002.stdout: "created": "2023-11-05T08:32:15.953286+0000", 2023-11-05T08:34:03.963 INFO:teuthology.orchestra.run.smithi002.stdout: "modified": "2023-11-05T08:34:03.819986+0000", 2023-11-05T08:34:03.963 INFO:teuthology.orchestra.run.smithi002.stdout: "last_up_change": "2023-11-05T08:33:46.664317+0000", 2023-11-05T08:34:03.964 INFO:teuthology.orchestra.run.smithi002.stdout: "last_in_change": "2023-11-05T08:32:20.284340+0000", 2023-11-05T08:34:03.964 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-11-05T08:34:03.964 INFO:teuthology.orchestra.run.smithi002.stdout: "flags_num": 5799936, 2023-11-05T08:34:03.964 INFO:teuthology.orchestra.run.smithi002.stdout: "flags_set": [ 2023-11-05T08:34:03.965 INFO:teuthology.orchestra.run.smithi002.stdout: "pglog_hardlimit", 2023-11-05T08:34:03.965 INFO:teuthology.orchestra.run.smithi002.stdout: "purged_snapdirs", 2023-11-05T08:34:03.965 INFO:teuthology.orchestra.run.smithi002.stdout: "recovery_deletes", 2023-11-05T08:34:03.965 INFO:teuthology.orchestra.run.smithi002.stdout: "sortbitwise" 2023-11-05T08:34:03.965 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-05T08:34:03.966 INFO:teuthology.orchestra.run.smithi002.stdout: "crush_version": 7, 2023-11-05T08:34:03.966 INFO:teuthology.orchestra.run.smithi002.stdout: "full_ratio": 0.94999998807907104, 2023-11-05T08:34:03.966 INFO:teuthology.orchestra.run.smithi002.stdout: "backfillfull_ratio": 0.89999997615814209, 2023-11-05T08:34:03.966 INFO:teuthology.orchestra.run.smithi002.stdout: "nearfull_ratio": 0.85000002384185791, 2023-11-05T08:34:03.967 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_snapshot": "", 2023-11-05T08:34:03.967 INFO:teuthology.orchestra.run.smithi002.stdout: "pool_max": 3, 2023-11-05T08:34:03.967 INFO:teuthology.orchestra.run.smithi002.stdout: "max_osd": 16, 2023-11-05T08:34:03.967 INFO:teuthology.orchestra.run.smithi002.stdout: "require_min_compat_client": "luminous", 2023-11-05T08:34:03.967 INFO:teuthology.orchestra.run.smithi002.stdout: "min_compat_client": "jewel", 2023-11-05T08:34:03.968 INFO:teuthology.orchestra.run.smithi002.stdout: "require_osd_release": "reef", 2023-11-05T08:34:03.968 INFO:teuthology.orchestra.run.smithi002.stdout: "allow_crimson": false, 2023-11-05T08:34:03.968 INFO:teuthology.orchestra.run.smithi002.stdout: "pools": [ 2023-11-05T08:34:03.968 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:03.968 INFO:teuthology.orchestra.run.smithi002.stdout: "pool": 1, 2023-11-05T08:34:03.969 INFO:teuthology.orchestra.run.smithi002.stdout: "pool_name": ".mgr", 2023-11-05T08:34:03.969 INFO:teuthology.orchestra.run.smithi002.stdout: "create_time": "2023-11-05T08:32:26.748874+0000", 2023-11-05T08:34:03.969 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-05T08:34:03.969 INFO:teuthology.orchestra.run.smithi002.stdout: "flags_names": "hashpspool", 2023-11-05T08:34:03.969 INFO:teuthology.orchestra.run.smithi002.stdout: "type": 1, 2023-11-05T08:34:03.970 INFO:teuthology.orchestra.run.smithi002.stdout: "size": 2, 2023-11-05T08:34:03.970 INFO:teuthology.orchestra.run.smithi002.stdout: "min_size": 1, 2023-11-05T08:34:03.970 INFO:teuthology.orchestra.run.smithi002.stdout: "crush_rule": 0, 2023-11-05T08:34:03.970 INFO:teuthology.orchestra.run.smithi002.stdout: "peering_crush_bucket_count": 0, 2023-11-05T08:34:03.970 INFO:teuthology.orchestra.run.smithi002.stdout: "peering_crush_bucket_target": 0, 2023-11-05T08:34:03.971 INFO:teuthology.orchestra.run.smithi002.stdout: "peering_crush_bucket_barrier": 0, 2023-11-05T08:34:03.971 INFO:teuthology.orchestra.run.smithi002.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-11-05T08:34:03.971 INFO:teuthology.orchestra.run.smithi002.stdout: "object_hash": 2, 2023-11-05T08:34:03.971 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_autoscale_mode": "off", 2023-11-05T08:34:03.972 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_num": 1, 2023-11-05T08:34:03.972 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_placement_num": 1, 2023-11-05T08:34:03.972 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_placement_num_target": 1, 2023-11-05T08:34:03.972 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_num_target": 1, 2023-11-05T08:34:03.973 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_num_pending": 1, 2023-11-05T08:34:03.973 INFO:teuthology.orchestra.run.smithi002.stdout: "last_pg_merge_meta": { 2023-11-05T08:34:03.973 INFO:teuthology.orchestra.run.smithi002.stdout: "source_pgid": "0.0", 2023-11-05T08:34:03.973 INFO:teuthology.orchestra.run.smithi002.stdout: "ready_epoch": 0, 2023-11-05T08:34:03.973 INFO:teuthology.orchestra.run.smithi002.stdout: "last_epoch_started": 0, 2023-11-05T08:34:03.974 INFO:teuthology.orchestra.run.smithi002.stdout: "last_epoch_clean": 0, 2023-11-05T08:34:03.974 INFO:teuthology.orchestra.run.smithi002.stdout: "source_version": "0'0", 2023-11-05T08:34:03.974 INFO:teuthology.orchestra.run.smithi002.stdout: "target_version": "0'0" 2023-11-05T08:34:03.974 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:03.974 INFO:teuthology.orchestra.run.smithi002.stdout: "last_change": "24", 2023-11-05T08:34:03.975 INFO:teuthology.orchestra.run.smithi002.stdout: "last_force_op_resend": "0", 2023-11-05T08:34:03.975 INFO:teuthology.orchestra.run.smithi002.stdout: "last_force_op_resend_prenautilus": "0", 2023-11-05T08:34:03.975 INFO:teuthology.orchestra.run.smithi002.stdout: "last_force_op_resend_preluminous": "0", 2023-11-05T08:34:03.975 INFO:teuthology.orchestra.run.smithi002.stdout: "auid": 0, 2023-11-05T08:34:03.975 INFO:teuthology.orchestra.run.smithi002.stdout: "snap_mode": "selfmanaged", 2023-11-05T08:34:03.976 INFO:teuthology.orchestra.run.smithi002.stdout: "snap_seq": 0, 2023-11-05T08:34:03.976 INFO:teuthology.orchestra.run.smithi002.stdout: "snap_epoch": 0, 2023-11-05T08:34:03.976 INFO:teuthology.orchestra.run.smithi002.stdout: "pool_snaps": [], 2023-11-05T08:34:03.976 INFO:teuthology.orchestra.run.smithi002.stdout: "removed_snaps": "[]", 2023-11-05T08:34:03.976 INFO:teuthology.orchestra.run.smithi002.stdout: "quota_max_bytes": 0, 2023-11-05T08:34:03.977 INFO:teuthology.orchestra.run.smithi002.stdout: "quota_max_objects": 0, 2023-11-05T08:34:03.977 INFO:teuthology.orchestra.run.smithi002.stdout: "tiers": [], 2023-11-05T08:34:03.977 INFO:teuthology.orchestra.run.smithi002.stdout: "tier_of": -1, 2023-11-05T08:34:03.977 INFO:teuthology.orchestra.run.smithi002.stdout: "read_tier": -1, 2023-11-05T08:34:03.977 INFO:teuthology.orchestra.run.smithi002.stdout: "write_tier": -1, 2023-11-05T08:34:03.978 INFO:teuthology.orchestra.run.smithi002.stdout: "cache_mode": "none", 2023-11-05T08:34:03.978 INFO:teuthology.orchestra.run.smithi002.stdout: "target_max_bytes": 0, 2023-11-05T08:34:03.978 INFO:teuthology.orchestra.run.smithi002.stdout: "target_max_objects": 0, 2023-11-05T08:34:03.978 INFO:teuthology.orchestra.run.smithi002.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-11-05T08:34:03.978 INFO:teuthology.orchestra.run.smithi002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-11-05T08:34:03.979 INFO:teuthology.orchestra.run.smithi002.stdout: "cache_target_full_ratio_micro": 800000, 2023-11-05T08:34:03.979 INFO:teuthology.orchestra.run.smithi002.stdout: "cache_min_flush_age": 0, 2023-11-05T08:34:03.979 INFO:teuthology.orchestra.run.smithi002.stdout: "cache_min_evict_age": 0, 2023-11-05T08:34:03.979 INFO:teuthology.orchestra.run.smithi002.stdout: "erasure_code_profile": "", 2023-11-05T08:34:03.979 INFO:teuthology.orchestra.run.smithi002.stdout: "hit_set_params": { 2023-11-05T08:34:03.980 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "none" 2023-11-05T08:34:03.980 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:03.980 INFO:teuthology.orchestra.run.smithi002.stdout: "hit_set_period": 0, 2023-11-05T08:34:03.980 INFO:teuthology.orchestra.run.smithi002.stdout: "hit_set_count": 0, 2023-11-05T08:34:03.980 INFO:teuthology.orchestra.run.smithi002.stdout: "use_gmt_hitset": true, 2023-11-05T08:34:03.981 INFO:teuthology.orchestra.run.smithi002.stdout: "min_read_recency_for_promote": 0, 2023-11-05T08:34:03.981 INFO:teuthology.orchestra.run.smithi002.stdout: "min_write_recency_for_promote": 0, 2023-11-05T08:34:03.981 INFO:teuthology.orchestra.run.smithi002.stdout: "hit_set_grade_decay_rate": 0, 2023-11-05T08:34:03.981 INFO:teuthology.orchestra.run.smithi002.stdout: "hit_set_search_last_n": 0, 2023-11-05T08:34:03.981 INFO:teuthology.orchestra.run.smithi002.stdout: "grade_table": [], 2023-11-05T08:34:03.982 INFO:teuthology.orchestra.run.smithi002.stdout: "stripe_width": 0, 2023-11-05T08:34:03.982 INFO:teuthology.orchestra.run.smithi002.stdout: "expected_num_objects": 0, 2023-11-05T08:34:03.982 INFO:teuthology.orchestra.run.smithi002.stdout: "fast_read": false, 2023-11-05T08:34:03.982 INFO:teuthology.orchestra.run.smithi002.stdout: "options": { 2023-11-05T08:34:03.982 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_num_max": 32, 2023-11-05T08:34:03.983 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_num_min": 1 2023-11-05T08:34:03.983 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:03.983 INFO:teuthology.orchestra.run.smithi002.stdout: "application_metadata": { 2023-11-05T08:34:03.983 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": {} 2023-11-05T08:34:03.983 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:03.984 INFO:teuthology.orchestra.run.smithi002.stdout: "read_balance": { 2023-11-05T08:34:03.984 INFO:teuthology.orchestra.run.smithi002.stdout: "score_acting": 15.380000114440918, 2023-11-05T08:34:03.984 INFO:teuthology.orchestra.run.smithi002.stdout: "score_stable": 15.380000114440918, 2023-11-05T08:34:03.984 INFO:teuthology.orchestra.run.smithi002.stdout: "optimal_score": 0.12999999523162842, 2023-11-05T08:34:03.984 INFO:teuthology.orchestra.run.smithi002.stdout: "raw_score_acting": 2, 2023-11-05T08:34:03.985 INFO:teuthology.orchestra.run.smithi002.stdout: "raw_score_stable": 2, 2023-11-05T08:34:03.985 INFO:teuthology.orchestra.run.smithi002.stdout: "primary_affinity_weighted": 0.94039440155029297, 2023-11-05T08:34:03.985 INFO:teuthology.orchestra.run.smithi002.stdout: "average_primary_affinity": 0.93999999761581421, 2023-11-05T08:34:03.985 INFO:teuthology.orchestra.run.smithi002.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2023-11-05T08:34:03.985 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:03.986 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:03.986 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:03.986 INFO:teuthology.orchestra.run.smithi002.stdout: "pool": 2, 2023-11-05T08:34:03.986 INFO:teuthology.orchestra.run.smithi002.stdout: "pool_name": "rbd", 2023-11-05T08:34:03.986 INFO:teuthology.orchestra.run.smithi002.stdout: "create_time": "2023-11-05T08:32:30.772862+0000", 2023-11-05T08:34:03.986 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-05T08:34:03.987 INFO:teuthology.orchestra.run.smithi002.stdout: "flags_names": "hashpspool", 2023-11-05T08:34:03.987 INFO:teuthology.orchestra.run.smithi002.stdout: "type": 1, 2023-11-05T08:34:03.987 INFO:teuthology.orchestra.run.smithi002.stdout: "size": 2, 2023-11-05T08:34:03.987 INFO:teuthology.orchestra.run.smithi002.stdout: "min_size": 1, 2023-11-05T08:34:03.988 INFO:teuthology.orchestra.run.smithi002.stdout: "crush_rule": 0, 2023-11-05T08:34:03.988 INFO:teuthology.orchestra.run.smithi002.stdout: "peering_crush_bucket_count": 0, 2023-11-05T08:34:03.988 INFO:teuthology.orchestra.run.smithi002.stdout: "peering_crush_bucket_target": 0, 2023-11-05T08:34:03.988 INFO:teuthology.orchestra.run.smithi002.stdout: "peering_crush_bucket_barrier": 0, 2023-11-05T08:34:03.988 INFO:teuthology.orchestra.run.smithi002.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-11-05T08:34:03.988 INFO:teuthology.orchestra.run.smithi002.stdout: "object_hash": 2, 2023-11-05T08:34:03.989 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_autoscale_mode": "off", 2023-11-05T08:34:03.989 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_num": 8, 2023-11-05T08:34:03.989 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_placement_num": 8, 2023-11-05T08:34:03.989 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_placement_num_target": 8, 2023-11-05T08:34:03.989 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_num_target": 8, 2023-11-05T08:34:03.990 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_num_pending": 8, 2023-11-05T08:34:03.990 INFO:teuthology.orchestra.run.smithi002.stdout: "last_pg_merge_meta": { 2023-11-05T08:34:03.990 INFO:teuthology.orchestra.run.smithi002.stdout: "source_pgid": "0.0", 2023-11-05T08:34:03.990 INFO:teuthology.orchestra.run.smithi002.stdout: "ready_epoch": 0, 2023-11-05T08:34:03.991 INFO:teuthology.orchestra.run.smithi002.stdout: "last_epoch_started": 0, 2023-11-05T08:34:03.991 INFO:teuthology.orchestra.run.smithi002.stdout: "last_epoch_clean": 0, 2023-11-05T08:34:03.991 INFO:teuthology.orchestra.run.smithi002.stdout: "source_version": "0'0", 2023-11-05T08:34:03.991 INFO:teuthology.orchestra.run.smithi002.stdout: "target_version": "0'0" 2023-11-05T08:34:03.991 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:03.992 INFO:teuthology.orchestra.run.smithi002.stdout: "last_change": "27", 2023-11-05T08:34:03.992 INFO:teuthology.orchestra.run.smithi002.stdout: "last_force_op_resend": "0", 2023-11-05T08:34:03.992 INFO:teuthology.orchestra.run.smithi002.stdout: "last_force_op_resend_prenautilus": "0", 2023-11-05T08:34:03.992 INFO:teuthology.orchestra.run.smithi002.stdout: "last_force_op_resend_preluminous": "0", 2023-11-05T08:34:03.992 INFO:teuthology.orchestra.run.smithi002.stdout: "auid": 0, 2023-11-05T08:34:03.993 INFO:teuthology.orchestra.run.smithi002.stdout: "snap_mode": "selfmanaged", 2023-11-05T08:34:03.993 INFO:teuthology.orchestra.run.smithi002.stdout: "snap_seq": 0, 2023-11-05T08:34:03.993 INFO:teuthology.orchestra.run.smithi002.stdout: "snap_epoch": 0, 2023-11-05T08:34:03.993 INFO:teuthology.orchestra.run.smithi002.stdout: "pool_snaps": [], 2023-11-05T08:34:03.993 INFO:teuthology.orchestra.run.smithi002.stdout: "removed_snaps": "[]", 2023-11-05T08:34:03.994 INFO:teuthology.orchestra.run.smithi002.stdout: "quota_max_bytes": 0, 2023-11-05T08:34:03.994 INFO:teuthology.orchestra.run.smithi002.stdout: "quota_max_objects": 0, 2023-11-05T08:34:03.994 INFO:teuthology.orchestra.run.smithi002.stdout: "tiers": [], 2023-11-05T08:34:03.994 INFO:teuthology.orchestra.run.smithi002.stdout: "tier_of": -1, 2023-11-05T08:34:03.994 INFO:teuthology.orchestra.run.smithi002.stdout: "read_tier": -1, 2023-11-05T08:34:03.995 INFO:teuthology.orchestra.run.smithi002.stdout: "write_tier": -1, 2023-11-05T08:34:03.995 INFO:teuthology.orchestra.run.smithi002.stdout: "cache_mode": "none", 2023-11-05T08:34:03.995 INFO:teuthology.orchestra.run.smithi002.stdout: "target_max_bytes": 0, 2023-11-05T08:34:03.995 INFO:teuthology.orchestra.run.smithi002.stdout: "target_max_objects": 0, 2023-11-05T08:34:03.995 INFO:teuthology.orchestra.run.smithi002.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-11-05T08:34:03.995 INFO:teuthology.orchestra.run.smithi002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-11-05T08:34:03.996 INFO:teuthology.orchestra.run.smithi002.stdout: "cache_target_full_ratio_micro": 800000, 2023-11-05T08:34:03.996 INFO:teuthology.orchestra.run.smithi002.stdout: "cache_min_flush_age": 0, 2023-11-05T08:34:03.996 INFO:teuthology.orchestra.run.smithi002.stdout: "cache_min_evict_age": 0, 2023-11-05T08:34:03.996 INFO:teuthology.orchestra.run.smithi002.stdout: "erasure_code_profile": "", 2023-11-05T08:34:03.996 INFO:teuthology.orchestra.run.smithi002.stdout: "hit_set_params": { 2023-11-05T08:34:03.997 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "none" 2023-11-05T08:34:03.997 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:03.997 INFO:teuthology.orchestra.run.smithi002.stdout: "hit_set_period": 0, 2023-11-05T08:34:03.997 INFO:teuthology.orchestra.run.smithi002.stdout: "hit_set_count": 0, 2023-11-05T08:34:03.997 INFO:teuthology.orchestra.run.smithi002.stdout: "use_gmt_hitset": true, 2023-11-05T08:34:03.997 INFO:teuthology.orchestra.run.smithi002.stdout: "min_read_recency_for_promote": 0, 2023-11-05T08:34:03.998 INFO:teuthology.orchestra.run.smithi002.stdout: "min_write_recency_for_promote": 0, 2023-11-05T08:34:03.998 INFO:teuthology.orchestra.run.smithi002.stdout: "hit_set_grade_decay_rate": 0, 2023-11-05T08:34:03.998 INFO:teuthology.orchestra.run.smithi002.stdout: "hit_set_search_last_n": 0, 2023-11-05T08:34:03.998 INFO:teuthology.orchestra.run.smithi002.stdout: "grade_table": [], 2023-11-05T08:34:03.998 INFO:teuthology.orchestra.run.smithi002.stdout: "stripe_width": 0, 2023-11-05T08:34:03.999 INFO:teuthology.orchestra.run.smithi002.stdout: "expected_num_objects": 0, 2023-11-05T08:34:03.999 INFO:teuthology.orchestra.run.smithi002.stdout: "fast_read": false, 2023-11-05T08:34:03.999 INFO:teuthology.orchestra.run.smithi002.stdout: "options": {}, 2023-11-05T08:34:03.999 INFO:teuthology.orchestra.run.smithi002.stdout: "application_metadata": { 2023-11-05T08:34:03.999 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd": {} 2023-11-05T08:34:04.000 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.000 INFO:teuthology.orchestra.run.smithi002.stdout: "read_balance": { 2023-11-05T08:34:04.000 INFO:teuthology.orchestra.run.smithi002.stdout: "score_acting": 4, 2023-11-05T08:34:04.000 INFO:teuthology.orchestra.run.smithi002.stdout: "score_stable": 4, 2023-11-05T08:34:04.000 INFO:teuthology.orchestra.run.smithi002.stdout: "optimal_score": 0.5, 2023-11-05T08:34:04.000 INFO:teuthology.orchestra.run.smithi002.stdout: "raw_score_acting": 2, 2023-11-05T08:34:04.001 INFO:teuthology.orchestra.run.smithi002.stdout: "raw_score_stable": 2, 2023-11-05T08:34:04.001 INFO:teuthology.orchestra.run.smithi002.stdout: "primary_affinity_weighted": 0.94039440155029297, 2023-11-05T08:34:04.001 INFO:teuthology.orchestra.run.smithi002.stdout: "average_primary_affinity": 0.93999999761581421, 2023-11-05T08:34:04.001 INFO:teuthology.orchestra.run.smithi002.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2023-11-05T08:34:04.001 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.002 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.002 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.002 INFO:teuthology.orchestra.run.smithi002.stdout: "pool": 3, 2023-11-05T08:34:04.002 INFO:teuthology.orchestra.run.smithi002.stdout: "pool_name": "unique_pool_0", 2023-11-05T08:34:04.002 INFO:teuthology.orchestra.run.smithi002.stdout: "create_time": "2023-11-05T08:32:42.050417+0000", 2023-11-05T08:34:04.003 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 8193, 2023-11-05T08:34:04.003 INFO:teuthology.orchestra.run.smithi002.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2023-11-05T08:34:04.003 INFO:teuthology.orchestra.run.smithi002.stdout: "type": 3, 2023-11-05T08:34:04.003 INFO:teuthology.orchestra.run.smithi002.stdout: "size": 7, 2023-11-05T08:34:04.003 INFO:teuthology.orchestra.run.smithi002.stdout: "min_size": 5, 2023-11-05T08:34:04.003 INFO:teuthology.orchestra.run.smithi002.stdout: "crush_rule": 1, 2023-11-05T08:34:04.004 INFO:teuthology.orchestra.run.smithi002.stdout: "peering_crush_bucket_count": 0, 2023-11-05T08:34:04.004 INFO:teuthology.orchestra.run.smithi002.stdout: "peering_crush_bucket_target": 0, 2023-11-05T08:34:04.004 INFO:teuthology.orchestra.run.smithi002.stdout: "peering_crush_bucket_barrier": 0, 2023-11-05T08:34:04.004 INFO:teuthology.orchestra.run.smithi002.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-11-05T08:34:04.004 INFO:teuthology.orchestra.run.smithi002.stdout: "object_hash": 2, 2023-11-05T08:34:04.005 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_autoscale_mode": "off", 2023-11-05T08:34:04.005 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_num": 26, 2023-11-05T08:34:04.005 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_placement_num": 26, 2023-11-05T08:34:04.005 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_placement_num_target": 26, 2023-11-05T08:34:04.005 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_num_target": 26, 2023-11-05T08:34:04.006 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_num_pending": 26, 2023-11-05T08:34:04.006 INFO:teuthology.orchestra.run.smithi002.stdout: "last_pg_merge_meta": { 2023-11-05T08:34:04.006 INFO:teuthology.orchestra.run.smithi002.stdout: "source_pgid": "0.0", 2023-11-05T08:34:04.006 INFO:teuthology.orchestra.run.smithi002.stdout: "ready_epoch": 0, 2023-11-05T08:34:04.006 INFO:teuthology.orchestra.run.smithi002.stdout: "last_epoch_started": 0, 2023-11-05T08:34:04.007 INFO:teuthology.orchestra.run.smithi002.stdout: "last_epoch_clean": 0, 2023-11-05T08:34:04.007 INFO:teuthology.orchestra.run.smithi002.stdout: "source_version": "0'0", 2023-11-05T08:34:04.007 INFO:teuthology.orchestra.run.smithi002.stdout: "target_version": "0'0" 2023-11-05T08:34:04.007 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.007 INFO:teuthology.orchestra.run.smithi002.stdout: "last_change": "110", 2023-11-05T08:34:04.008 INFO:teuthology.orchestra.run.smithi002.stdout: "last_force_op_resend": "0", 2023-11-05T08:34:04.008 INFO:teuthology.orchestra.run.smithi002.stdout: "last_force_op_resend_prenautilus": "0", 2023-11-05T08:34:04.008 INFO:teuthology.orchestra.run.smithi002.stdout: "last_force_op_resend_preluminous": "40", 2023-11-05T08:34:04.008 INFO:teuthology.orchestra.run.smithi002.stdout: "auid": 0, 2023-11-05T08:34:04.008 INFO:teuthology.orchestra.run.smithi002.stdout: "snap_mode": "selfmanaged", 2023-11-05T08:34:04.009 INFO:teuthology.orchestra.run.smithi002.stdout: "snap_seq": 76, 2023-11-05T08:34:04.009 INFO:teuthology.orchestra.run.smithi002.stdout: "snap_epoch": 110, 2023-11-05T08:34:04.009 INFO:teuthology.orchestra.run.smithi002.stdout: "pool_snaps": [], 2023-11-05T08:34:04.009 INFO:teuthology.orchestra.run.smithi002.stdout: "removed_snaps": "[]", 2023-11-05T08:34:04.009 INFO:teuthology.orchestra.run.smithi002.stdout: "quota_max_bytes": 0, 2023-11-05T08:34:04.010 INFO:teuthology.orchestra.run.smithi002.stdout: "quota_max_objects": 0, 2023-11-05T08:34:04.010 INFO:teuthology.orchestra.run.smithi002.stdout: "tiers": [], 2023-11-05T08:34:04.010 INFO:teuthology.orchestra.run.smithi002.stdout: "tier_of": -1, 2023-11-05T08:34:04.010 INFO:teuthology.orchestra.run.smithi002.stdout: "read_tier": -1, 2023-11-05T08:34:04.010 INFO:teuthology.orchestra.run.smithi002.stdout: "write_tier": -1, 2023-11-05T08:34:04.011 INFO:teuthology.orchestra.run.smithi002.stdout: "cache_mode": "none", 2023-11-05T08:34:04.011 INFO:teuthology.orchestra.run.smithi002.stdout: "target_max_bytes": 0, 2023-11-05T08:34:04.011 INFO:teuthology.orchestra.run.smithi002.stdout: "target_max_objects": 0, 2023-11-05T08:34:04.011 INFO:teuthology.orchestra.run.smithi002.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-11-05T08:34:04.011 INFO:teuthology.orchestra.run.smithi002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-11-05T08:34:04.012 INFO:teuthology.orchestra.run.smithi002.stdout: "cache_target_full_ratio_micro": 800000, 2023-11-05T08:34:04.012 INFO:teuthology.orchestra.run.smithi002.stdout: "cache_min_flush_age": 0, 2023-11-05T08:34:04.012 INFO:teuthology.orchestra.run.smithi002.stdout: "cache_min_evict_age": 0, 2023-11-05T08:34:04.012 INFO:teuthology.orchestra.run.smithi002.stdout: "erasure_code_profile": "shecprofile", 2023-11-05T08:34:04.012 INFO:teuthology.orchestra.run.smithi002.stdout: "hit_set_params": { 2023-11-05T08:34:04.013 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "none" 2023-11-05T08:34:04.013 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.013 INFO:teuthology.orchestra.run.smithi002.stdout: "hit_set_period": 0, 2023-11-05T08:34:04.013 INFO:teuthology.orchestra.run.smithi002.stdout: "hit_set_count": 0, 2023-11-05T08:34:04.013 INFO:teuthology.orchestra.run.smithi002.stdout: "use_gmt_hitset": true, 2023-11-05T08:34:04.014 INFO:teuthology.orchestra.run.smithi002.stdout: "min_read_recency_for_promote": 0, 2023-11-05T08:34:04.014 INFO:teuthology.orchestra.run.smithi002.stdout: "min_write_recency_for_promote": 0, 2023-11-05T08:34:04.014 INFO:teuthology.orchestra.run.smithi002.stdout: "hit_set_grade_decay_rate": 0, 2023-11-05T08:34:04.014 INFO:teuthology.orchestra.run.smithi002.stdout: "hit_set_search_last_n": 0, 2023-11-05T08:34:04.014 INFO:teuthology.orchestra.run.smithi002.stdout: "grade_table": [], 2023-11-05T08:34:04.015 INFO:teuthology.orchestra.run.smithi002.stdout: "stripe_width": 16384, 2023-11-05T08:34:04.015 INFO:teuthology.orchestra.run.smithi002.stdout: "expected_num_objects": 0, 2023-11-05T08:34:04.015 INFO:teuthology.orchestra.run.smithi002.stdout: "fast_read": false, 2023-11-05T08:34:04.015 INFO:teuthology.orchestra.run.smithi002.stdout: "options": {}, 2023-11-05T08:34:04.015 INFO:teuthology.orchestra.run.smithi002.stdout: "application_metadata": { 2023-11-05T08:34:04.016 INFO:teuthology.orchestra.run.smithi002.stdout: "rados": {} 2023-11-05T08:34:04.016 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.016 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.016 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-05T08:34:04.016 INFO:teuthology.orchestra.run.smithi002.stdout: "osds": [ 2023-11-05T08:34:04.017 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.017 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": 0, 2023-11-05T08:34:04.017 INFO:teuthology.orchestra.run.smithi002.stdout: "uuid": "3626fa17-5bde-458d-96fb-f5e70ac6edd6", 2023-11-05T08:34:04.017 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 0, 2023-11-05T08:34:04.017 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2023-11-05T08:34:04.018 INFO:teuthology.orchestra.run.smithi002.stdout: "weight": 1, 2023-11-05T08:34:04.018 INFO:teuthology.orchestra.run.smithi002.stdout: "primary_affinity": 1, 2023-11-05T08:34:04.018 INFO:teuthology.orchestra.run.smithi002.stdout: "last_clean_begin": 0, 2023-11-05T08:34:04.018 INFO:teuthology.orchestra.run.smithi002.stdout: "last_clean_end": 0, 2023-11-05T08:34:04.018 INFO:teuthology.orchestra.run.smithi002.stdout: "up_from": 21, 2023-11-05T08:34:04.019 INFO:teuthology.orchestra.run.smithi002.stdout: "up_thru": 30, 2023-11-05T08:34:04.019 INFO:teuthology.orchestra.run.smithi002.stdout: "down_at": 37, 2023-11-05T08:34:04.019 INFO:teuthology.orchestra.run.smithi002.stdout: "lost_at": 0, 2023-11-05T08:34:04.019 INFO:teuthology.orchestra.run.smithi002.stdout: "public_addrs": { 2023-11-05T08:34:04.019 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.020 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.020 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.020 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:6802", 2023-11-05T08:34:04.020 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1948090427 2023-11-05T08:34:04.020 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.021 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.021 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.021 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:6803", 2023-11-05T08:34:04.021 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1948090427 2023-11-05T08:34:04.021 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.022 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.022 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.022 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_addrs": { 2023-11-05T08:34:04.022 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.023 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.023 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.023 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:6804", 2023-11-05T08:34:04.023 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1948090427 2023-11-05T08:34:04.023 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.024 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.024 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.024 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:6805", 2023-11-05T08:34:04.024 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1948090427 2023-11-05T08:34:04.024 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.024 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.025 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.025 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_back_addrs": { 2023-11-05T08:34:04.025 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.025 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.025 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.026 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:6808", 2023-11-05T08:34:04.026 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1948090427 2023-11-05T08:34:04.026 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.027 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.027 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.027 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:6809", 2023-11-05T08:34:04.028 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1948090427 2023-11-05T08:34:04.028 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.028 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.028 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.029 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_front_addrs": { 2023-11-05T08:34:04.029 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.029 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.030 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.030 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:6806", 2023-11-05T08:34:04.030 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1948090427 2023-11-05T08:34:04.031 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.031 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.031 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.032 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:6807", 2023-11-05T08:34:04.032 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1948090427 2023-11-05T08:34:04.032 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.033 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.033 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.033 INFO:teuthology.orchestra.run.smithi002.stdout: "public_addr": "172.21.15.2:6803/1948090427", 2023-11-05T08:34:04.033 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_addr": "172.21.15.2:6805/1948090427", 2023-11-05T08:34:04.034 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_back_addr": "172.21.15.2:6809/1948090427", 2023-11-05T08:34:04.034 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_front_addr": "172.21.15.2:6807/1948090427", 2023-11-05T08:34:04.034 INFO:teuthology.orchestra.run.smithi002.stdout: "state": [ 2023-11-05T08:34:04.034 INFO:teuthology.orchestra.run.smithi002.stdout: "exists" 2023-11-05T08:34:04.035 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.035 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.035 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.035 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": 1, 2023-11-05T08:34:04.035 INFO:teuthology.orchestra.run.smithi002.stdout: "uuid": "8f524504-91f5-44f3-afbd-69dfc19f03d3", 2023-11-05T08:34:04.035 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2023-11-05T08:34:04.036 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2023-11-05T08:34:04.036 INFO:teuthology.orchestra.run.smithi002.stdout: "weight": 1, 2023-11-05T08:34:04.036 INFO:teuthology.orchestra.run.smithi002.stdout: "primary_affinity": 1, 2023-11-05T08:34:04.036 INFO:teuthology.orchestra.run.smithi002.stdout: "last_clean_begin": 0, 2023-11-05T08:34:04.036 INFO:teuthology.orchestra.run.smithi002.stdout: "last_clean_end": 0, 2023-11-05T08:34:04.037 INFO:teuthology.orchestra.run.smithi002.stdout: "up_from": 22, 2023-11-05T08:34:04.037 INFO:teuthology.orchestra.run.smithi002.stdout: "up_thru": 0, 2023-11-05T08:34:04.037 INFO:teuthology.orchestra.run.smithi002.stdout: "down_at": 0, 2023-11-05T08:34:04.037 INFO:teuthology.orchestra.run.smithi002.stdout: "lost_at": 0, 2023-11-05T08:34:04.037 INFO:teuthology.orchestra.run.smithi002.stdout: "public_addrs": { 2023-11-05T08:34:04.038 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.038 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.038 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.038 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.53:6816", 2023-11-05T08:34:04.038 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 121537835 2023-11-05T08:34:04.039 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.039 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.039 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.039 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.53:6818", 2023-11-05T08:34:04.039 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 121537835 2023-11-05T08:34:04.039 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.040 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.040 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.040 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_addrs": { 2023-11-05T08:34:04.040 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.040 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.041 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.041 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.53:6820", 2023-11-05T08:34:04.041 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 121537835 2023-11-05T08:34:04.041 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.041 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.041 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.042 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.53:6822", 2023-11-05T08:34:04.042 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 121537835 2023-11-05T08:34:04.042 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.042 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.042 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.042 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_back_addrs": { 2023-11-05T08:34:04.043 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.043 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.043 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.043 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.53:6827", 2023-11-05T08:34:04.043 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 121537835 2023-11-05T08:34:04.043 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.044 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.044 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.044 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.53:6829", 2023-11-05T08:34:04.044 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 121537835 2023-11-05T08:34:04.044 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.045 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.045 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.045 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_front_addrs": { 2023-11-05T08:34:04.045 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.045 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.045 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.046 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.53:6824", 2023-11-05T08:34:04.046 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 121537835 2023-11-05T08:34:04.046 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.046 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.047 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.047 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.53:6826", 2023-11-05T08:34:04.047 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 121537835 2023-11-05T08:34:04.047 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.047 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.048 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.048 INFO:teuthology.orchestra.run.smithi002.stdout: "public_addr": "172.21.15.53:6818/121537835", 2023-11-05T08:34:04.048 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_addr": "172.21.15.53:6822/121537835", 2023-11-05T08:34:04.048 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_back_addr": "172.21.15.53:6829/121537835", 2023-11-05T08:34:04.048 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_front_addr": "172.21.15.53:6826/121537835", 2023-11-05T08:34:04.048 INFO:teuthology.orchestra.run.smithi002.stdout: "state": [ 2023-11-05T08:34:04.049 INFO:teuthology.orchestra.run.smithi002.stdout: "exists", 2023-11-05T08:34:04.049 INFO:teuthology.orchestra.run.smithi002.stdout: "up" 2023-11-05T08:34:04.049 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.049 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.049 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.050 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": 2, 2023-11-05T08:34:04.050 INFO:teuthology.orchestra.run.smithi002.stdout: "uuid": "696f71d0-a98b-46b5-bdc0-0da3d6065469", 2023-11-05T08:34:04.050 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2023-11-05T08:34:04.050 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2023-11-05T08:34:04.050 INFO:teuthology.orchestra.run.smithi002.stdout: "weight": 1, 2023-11-05T08:34:04.051 INFO:teuthology.orchestra.run.smithi002.stdout: "primary_affinity": 1, 2023-11-05T08:34:04.051 INFO:teuthology.orchestra.run.smithi002.stdout: "last_clean_begin": 0, 2023-11-05T08:34:04.051 INFO:teuthology.orchestra.run.smithi002.stdout: "last_clean_end": 0, 2023-11-05T08:34:04.051 INFO:teuthology.orchestra.run.smithi002.stdout: "up_from": 22, 2023-11-05T08:34:04.051 INFO:teuthology.orchestra.run.smithi002.stdout: "up_thru": 30, 2023-11-05T08:34:04.051 INFO:teuthology.orchestra.run.smithi002.stdout: "down_at": 0, 2023-11-05T08:34:04.052 INFO:teuthology.orchestra.run.smithi002.stdout: "lost_at": 0, 2023-11-05T08:34:04.052 INFO:teuthology.orchestra.run.smithi002.stdout: "public_addrs": { 2023-11-05T08:34:04.052 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.052 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.052 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.053 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.69:6800", 2023-11-05T08:34:04.053 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1463424062 2023-11-05T08:34:04.053 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.053 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.053 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.053 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.69:6801", 2023-11-05T08:34:04.054 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1463424062 2023-11-05T08:34:04.054 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.054 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.054 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.054 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_addrs": { 2023-11-05T08:34:04.055 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.055 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.055 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.055 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.69:6802", 2023-11-05T08:34:04.055 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1463424062 2023-11-05T08:34:04.055 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.056 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.056 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.056 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.69:6803", 2023-11-05T08:34:04.056 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1463424062 2023-11-05T08:34:04.056 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.056 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.057 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.057 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_back_addrs": { 2023-11-05T08:34:04.057 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.057 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.057 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.058 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.69:6806", 2023-11-05T08:34:04.058 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1463424062 2023-11-05T08:34:04.058 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.058 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.058 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.058 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.69:6807", 2023-11-05T08:34:04.059 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1463424062 2023-11-05T08:34:04.059 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.059 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.059 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.059 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_front_addrs": { 2023-11-05T08:34:04.059 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.060 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.060 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.060 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.69:6804", 2023-11-05T08:34:04.060 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1463424062 2023-11-05T08:34:04.060 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.060 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.061 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.061 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.69:6805", 2023-11-05T08:34:04.061 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1463424062 2023-11-05T08:34:04.061 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.061 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.061 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.062 INFO:teuthology.orchestra.run.smithi002.stdout: "public_addr": "172.21.15.69:6801/1463424062", 2023-11-05T08:34:04.062 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_addr": "172.21.15.69:6803/1463424062", 2023-11-05T08:34:04.062 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_back_addr": "172.21.15.69:6807/1463424062", 2023-11-05T08:34:04.062 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_front_addr": "172.21.15.69:6805/1463424062", 2023-11-05T08:34:04.062 INFO:teuthology.orchestra.run.smithi002.stdout: "state": [ 2023-11-05T08:34:04.063 INFO:teuthology.orchestra.run.smithi002.stdout: "exists", 2023-11-05T08:34:04.063 INFO:teuthology.orchestra.run.smithi002.stdout: "up" 2023-11-05T08:34:04.063 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.063 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.063 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.064 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": 3, 2023-11-05T08:34:04.064 INFO:teuthology.orchestra.run.smithi002.stdout: "uuid": "fc979d30-659b-47f4-b06b-97a4d41c2e90", 2023-11-05T08:34:04.064 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2023-11-05T08:34:04.064 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2023-11-05T08:34:04.064 INFO:teuthology.orchestra.run.smithi002.stdout: "weight": 1, 2023-11-05T08:34:04.064 INFO:teuthology.orchestra.run.smithi002.stdout: "primary_affinity": 0.0463104248046875, 2023-11-05T08:34:04.065 INFO:teuthology.orchestra.run.smithi002.stdout: "last_clean_begin": 21, 2023-11-05T08:34:04.065 INFO:teuthology.orchestra.run.smithi002.stdout: "last_clean_end": 43, 2023-11-05T08:34:04.065 INFO:teuthology.orchestra.run.smithi002.stdout: "up_from": 93, 2023-11-05T08:34:04.065 INFO:teuthology.orchestra.run.smithi002.stdout: "up_thru": 43, 2023-11-05T08:34:04.065 INFO:teuthology.orchestra.run.smithi002.stdout: "down_at": 45, 2023-11-05T08:34:04.065 INFO:teuthology.orchestra.run.smithi002.stdout: "lost_at": 0, 2023-11-05T08:34:04.066 INFO:teuthology.orchestra.run.smithi002.stdout: "public_addrs": { 2023-11-05T08:34:04.066 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.066 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.066 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.066 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.90:6800", 2023-11-05T08:34:04.067 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2085535590 2023-11-05T08:34:04.067 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.067 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.067 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.067 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.90:6801", 2023-11-05T08:34:04.068 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2085535590 2023-11-05T08:34:04.068 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.068 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.068 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.068 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_addrs": { 2023-11-05T08:34:04.068 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.069 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.069 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.069 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.90:6802", 2023-11-05T08:34:04.069 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2085535590 2023-11-05T08:34:04.069 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.069 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.070 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.070 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.90:6803", 2023-11-05T08:34:04.070 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2085535590 2023-11-05T08:34:04.070 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.070 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.071 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.071 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_back_addrs": { 2023-11-05T08:34:04.071 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.071 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.071 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.071 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.90:6806", 2023-11-05T08:34:04.072 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2085535590 2023-11-05T08:34:04.072 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.072 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.072 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.072 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.90:6807", 2023-11-05T08:34:04.072 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2085535590 2023-11-05T08:34:04.073 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.073 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.073 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.073 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_front_addrs": { 2023-11-05T08:34:04.073 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.073 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.074 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.074 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.90:6804", 2023-11-05T08:34:04.074 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2085535590 2023-11-05T08:34:04.074 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.074 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.074 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.075 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.90:6805", 2023-11-05T08:34:04.075 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2085535590 2023-11-05T08:34:04.075 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.075 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.075 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.075 INFO:teuthology.orchestra.run.smithi002.stdout: "public_addr": "172.21.15.90:6801/2085535590", 2023-11-05T08:34:04.076 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_addr": "172.21.15.90:6803/2085535590", 2023-11-05T08:34:04.076 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_back_addr": "172.21.15.90:6807/2085535590", 2023-11-05T08:34:04.076 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_front_addr": "172.21.15.90:6805/2085535590", 2023-11-05T08:34:04.076 INFO:teuthology.orchestra.run.smithi002.stdout: "state": [ 2023-11-05T08:34:04.076 INFO:teuthology.orchestra.run.smithi002.stdout: "exists", 2023-11-05T08:34:04.076 INFO:teuthology.orchestra.run.smithi002.stdout: "up" 2023-11-05T08:34:04.077 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.077 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.077 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.077 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": 4, 2023-11-05T08:34:04.077 INFO:teuthology.orchestra.run.smithi002.stdout: "uuid": "2ed87cb9-9ced-46fd-8346-3f47709bb69e", 2023-11-05T08:34:04.077 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2023-11-05T08:34:04.077 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2023-11-05T08:34:04.078 INFO:teuthology.orchestra.run.smithi002.stdout: "weight": 1, 2023-11-05T08:34:04.078 INFO:teuthology.orchestra.run.smithi002.stdout: "primary_affinity": 1, 2023-11-05T08:34:04.078 INFO:teuthology.orchestra.run.smithi002.stdout: "last_clean_begin": 0, 2023-11-05T08:34:04.078 INFO:teuthology.orchestra.run.smithi002.stdout: "last_clean_end": 0, 2023-11-05T08:34:04.078 INFO:teuthology.orchestra.run.smithi002.stdout: "up_from": 22, 2023-11-05T08:34:04.079 INFO:teuthology.orchestra.run.smithi002.stdout: "up_thru": 44, 2023-11-05T08:34:04.079 INFO:teuthology.orchestra.run.smithi002.stdout: "down_at": 0, 2023-11-05T08:34:04.079 INFO:teuthology.orchestra.run.smithi002.stdout: "lost_at": 0, 2023-11-05T08:34:04.079 INFO:teuthology.orchestra.run.smithi002.stdout: "public_addrs": { 2023-11-05T08:34:04.079 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.079 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.079 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.080 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:6810", 2023-11-05T08:34:04.080 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2916800887 2023-11-05T08:34:04.080 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.080 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.080 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.081 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:6811", 2023-11-05T08:34:04.081 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2916800887 2023-11-05T08:34:04.081 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.081 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.081 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.081 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_addrs": { 2023-11-05T08:34:04.082 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.082 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.082 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.082 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:6812", 2023-11-05T08:34:04.082 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2916800887 2023-11-05T08:34:04.082 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.083 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.083 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.083 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:6813", 2023-11-05T08:34:04.083 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2916800887 2023-11-05T08:34:04.083 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.083 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.084 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.084 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_back_addrs": { 2023-11-05T08:34:04.084 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.084 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.084 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.084 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:6816", 2023-11-05T08:34:04.084 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2916800887 2023-11-05T08:34:04.085 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.085 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.085 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.085 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:6817", 2023-11-05T08:34:04.085 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2916800887 2023-11-05T08:34:04.085 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.086 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.086 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.086 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_front_addrs": { 2023-11-05T08:34:04.086 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.086 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.087 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.087 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:6814", 2023-11-05T08:34:04.087 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2916800887 2023-11-05T08:34:04.087 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.087 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.087 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.088 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:6815", 2023-11-05T08:34:04.088 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2916800887 2023-11-05T08:34:04.088 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.088 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.088 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.088 INFO:teuthology.orchestra.run.smithi002.stdout: "public_addr": "172.21.15.2:6811/2916800887", 2023-11-05T08:34:04.089 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_addr": "172.21.15.2:6813/2916800887", 2023-11-05T08:34:04.089 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_back_addr": "172.21.15.2:6817/2916800887", 2023-11-05T08:34:04.089 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_front_addr": "172.21.15.2:6815/2916800887", 2023-11-05T08:34:04.089 INFO:teuthology.orchestra.run.smithi002.stdout: "state": [ 2023-11-05T08:34:04.089 INFO:teuthology.orchestra.run.smithi002.stdout: "exists", 2023-11-05T08:34:04.089 INFO:teuthology.orchestra.run.smithi002.stdout: "up" 2023-11-05T08:34:04.090 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.090 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.090 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.090 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": 5, 2023-11-05T08:34:04.090 INFO:teuthology.orchestra.run.smithi002.stdout: "uuid": "5bb59d8d-9b93-48b2-a5ee-13646b19ce50", 2023-11-05T08:34:04.091 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2023-11-05T08:34:04.091 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2023-11-05T08:34:04.091 INFO:teuthology.orchestra.run.smithi002.stdout: "weight": 1, 2023-11-05T08:34:04.091 INFO:teuthology.orchestra.run.smithi002.stdout: "primary_affinity": 1, 2023-11-05T08:34:04.091 INFO:teuthology.orchestra.run.smithi002.stdout: "last_clean_begin": 0, 2023-11-05T08:34:04.091 INFO:teuthology.orchestra.run.smithi002.stdout: "last_clean_end": 0, 2023-11-05T08:34:04.092 INFO:teuthology.orchestra.run.smithi002.stdout: "up_from": 22, 2023-11-05T08:34:04.092 INFO:teuthology.orchestra.run.smithi002.stdout: "up_thru": 96, 2023-11-05T08:34:04.092 INFO:teuthology.orchestra.run.smithi002.stdout: "down_at": 0, 2023-11-05T08:34:04.092 INFO:teuthology.orchestra.run.smithi002.stdout: "lost_at": 0, 2023-11-05T08:34:04.092 INFO:teuthology.orchestra.run.smithi002.stdout: "public_addrs": { 2023-11-05T08:34:04.092 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.093 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.093 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.093 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.53:6808", 2023-11-05T08:34:04.093 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 588137330 2023-11-05T08:34:04.093 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.093 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.094 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.094 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.53:6809", 2023-11-05T08:34:04.094 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 588137330 2023-11-05T08:34:04.094 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.094 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.094 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.095 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_addrs": { 2023-11-05T08:34:04.095 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.095 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.095 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.095 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.53:6810", 2023-11-05T08:34:04.095 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 588137330 2023-11-05T08:34:04.096 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.096 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.096 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.096 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.53:6811", 2023-11-05T08:34:04.096 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 588137330 2023-11-05T08:34:04.096 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.097 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.097 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.097 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_back_addrs": { 2023-11-05T08:34:04.097 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.097 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.097 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.098 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.53:6814", 2023-11-05T08:34:04.098 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 588137330 2023-11-05T08:34:04.098 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.098 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.098 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.099 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.53:6815", 2023-11-05T08:34:04.099 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 588137330 2023-11-05T08:34:04.099 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.099 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.099 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.099 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_front_addrs": { 2023-11-05T08:34:04.100 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.100 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.100 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.100 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.53:6812", 2023-11-05T08:34:04.100 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 588137330 2023-11-05T08:34:04.100 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.101 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.101 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.101 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.53:6813", 2023-11-05T08:34:04.101 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 588137330 2023-11-05T08:34:04.101 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.101 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.102 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.102 INFO:teuthology.orchestra.run.smithi002.stdout: "public_addr": "172.21.15.53:6809/588137330", 2023-11-05T08:34:04.102 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_addr": "172.21.15.53:6811/588137330", 2023-11-05T08:34:04.102 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_back_addr": "172.21.15.53:6815/588137330", 2023-11-05T08:34:04.102 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_front_addr": "172.21.15.53:6813/588137330", 2023-11-05T08:34:04.102 INFO:teuthology.orchestra.run.smithi002.stdout: "state": [ 2023-11-05T08:34:04.103 INFO:teuthology.orchestra.run.smithi002.stdout: "exists", 2023-11-05T08:34:04.103 INFO:teuthology.orchestra.run.smithi002.stdout: "up" 2023-11-05T08:34:04.103 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.103 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.103 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.103 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": 6, 2023-11-05T08:34:04.104 INFO:teuthology.orchestra.run.smithi002.stdout: "uuid": "46cd8f39-841d-4b7b-a4ef-6ec6493eea5c", 2023-11-05T08:34:04.104 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2023-11-05T08:34:04.104 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2023-11-05T08:34:04.104 INFO:teuthology.orchestra.run.smithi002.stdout: "weight": 1, 2023-11-05T08:34:04.104 INFO:teuthology.orchestra.run.smithi002.stdout: "primary_affinity": 1, 2023-11-05T08:34:04.104 INFO:teuthology.orchestra.run.smithi002.stdout: "last_clean_begin": 0, 2023-11-05T08:34:04.105 INFO:teuthology.orchestra.run.smithi002.stdout: "last_clean_end": 0, 2023-11-05T08:34:04.105 INFO:teuthology.orchestra.run.smithi002.stdout: "up_from": 21, 2023-11-05T08:34:04.105 INFO:teuthology.orchestra.run.smithi002.stdout: "up_thru": 93, 2023-11-05T08:34:04.105 INFO:teuthology.orchestra.run.smithi002.stdout: "down_at": 0, 2023-11-05T08:34:04.105 INFO:teuthology.orchestra.run.smithi002.stdout: "lost_at": 0, 2023-11-05T08:34:04.105 INFO:teuthology.orchestra.run.smithi002.stdout: "public_addrs": { 2023-11-05T08:34:04.106 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.106 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.106 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.106 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.69:6815", 2023-11-05T08:34:04.106 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 3720624375 2023-11-05T08:34:04.106 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.107 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.107 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.107 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.69:6818", 2023-11-05T08:34:04.107 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 3720624375 2023-11-05T08:34:04.107 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.108 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.108 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.108 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_addrs": { 2023-11-05T08:34:04.108 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.108 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.108 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.109 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.69:6821", 2023-11-05T08:34:04.109 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 3720624375 2023-11-05T08:34:04.109 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.109 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.109 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.109 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.69:6823", 2023-11-05T08:34:04.110 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 3720624375 2023-11-05T08:34:04.110 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.110 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.110 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.110 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_back_addrs": { 2023-11-05T08:34:04.111 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.111 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.111 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.111 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.69:6828", 2023-11-05T08:34:04.111 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 3720624375 2023-11-05T08:34:04.111 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.112 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.112 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.112 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.69:6829", 2023-11-05T08:34:04.112 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 3720624375 2023-11-05T08:34:04.112 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.112 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.113 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.113 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_front_addrs": { 2023-11-05T08:34:04.113 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.113 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.113 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.113 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.69:6824", 2023-11-05T08:34:04.114 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 3720624375 2023-11-05T08:34:04.114 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.114 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.114 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.114 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.69:6826", 2023-11-05T08:34:04.114 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 3720624375 2023-11-05T08:34:04.115 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.115 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.115 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.115 INFO:teuthology.orchestra.run.smithi002.stdout: "public_addr": "172.21.15.69:6818/3720624375", 2023-11-05T08:34:04.115 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_addr": "172.21.15.69:6823/3720624375", 2023-11-05T08:34:04.115 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_back_addr": "172.21.15.69:6829/3720624375", 2023-11-05T08:34:04.116 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_front_addr": "172.21.15.69:6826/3720624375", 2023-11-05T08:34:04.116 INFO:teuthology.orchestra.run.smithi002.stdout: "state": [ 2023-11-05T08:34:04.116 INFO:teuthology.orchestra.run.smithi002.stdout: "exists", 2023-11-05T08:34:04.116 INFO:teuthology.orchestra.run.smithi002.stdout: "up" 2023-11-05T08:34:04.116 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.116 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.117 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.117 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": 7, 2023-11-05T08:34:04.117 INFO:teuthology.orchestra.run.smithi002.stdout: "uuid": "c98d93d9-c1e5-4ac0-8f71-9fc1d2767963", 2023-11-05T08:34:04.117 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2023-11-05T08:34:04.117 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2023-11-05T08:34:04.117 INFO:teuthology.orchestra.run.smithi002.stdout: "weight": 1, 2023-11-05T08:34:04.118 INFO:teuthology.orchestra.run.smithi002.stdout: "primary_affinity": 1, 2023-11-05T08:34:04.118 INFO:teuthology.orchestra.run.smithi002.stdout: "last_clean_begin": 0, 2023-11-05T08:34:04.118 INFO:teuthology.orchestra.run.smithi002.stdout: "last_clean_end": 0, 2023-11-05T08:34:04.118 INFO:teuthology.orchestra.run.smithi002.stdout: "up_from": 22, 2023-11-05T08:34:04.118 INFO:teuthology.orchestra.run.smithi002.stdout: "up_thru": 96, 2023-11-05T08:34:04.118 INFO:teuthology.orchestra.run.smithi002.stdout: "down_at": 0, 2023-11-05T08:34:04.119 INFO:teuthology.orchestra.run.smithi002.stdout: "lost_at": 0, 2023-11-05T08:34:04.119 INFO:teuthology.orchestra.run.smithi002.stdout: "public_addrs": { 2023-11-05T08:34:04.119 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.119 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.119 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.119 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.90:6808", 2023-11-05T08:34:04.120 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2028724524 2023-11-05T08:34:04.120 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.120 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.120 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.120 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.90:6809", 2023-11-05T08:34:04.120 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2028724524 2023-11-05T08:34:04.121 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.121 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.121 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.121 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_addrs": { 2023-11-05T08:34:04.121 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.121 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.122 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.122 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.90:6810", 2023-11-05T08:34:04.122 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2028724524 2023-11-05T08:34:04.122 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.122 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.122 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.123 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.90:6811", 2023-11-05T08:34:04.123 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2028724524 2023-11-05T08:34:04.123 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.123 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.123 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.123 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_back_addrs": { 2023-11-05T08:34:04.124 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.124 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.124 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.124 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.90:6814", 2023-11-05T08:34:04.124 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2028724524 2023-11-05T08:34:04.124 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.125 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.125 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.125 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.90:6815", 2023-11-05T08:34:04.125 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2028724524 2023-11-05T08:34:04.125 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.125 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.126 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.126 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_front_addrs": { 2023-11-05T08:34:04.126 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.126 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.126 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.126 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.90:6812", 2023-11-05T08:34:04.127 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2028724524 2023-11-05T08:34:04.127 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.127 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.127 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.127 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.90:6813", 2023-11-05T08:34:04.127 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2028724524 2023-11-05T08:34:04.128 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.128 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.128 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.128 INFO:teuthology.orchestra.run.smithi002.stdout: "public_addr": "172.21.15.90:6809/2028724524", 2023-11-05T08:34:04.128 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_addr": "172.21.15.90:6811/2028724524", 2023-11-05T08:34:04.128 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_back_addr": "172.21.15.90:6815/2028724524", 2023-11-05T08:34:04.129 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_front_addr": "172.21.15.90:6813/2028724524", 2023-11-05T08:34:04.129 INFO:teuthology.orchestra.run.smithi002.stdout: "state": [ 2023-11-05T08:34:04.129 INFO:teuthology.orchestra.run.smithi002.stdout: "exists", 2023-11-05T08:34:04.129 INFO:teuthology.orchestra.run.smithi002.stdout: "up" 2023-11-05T08:34:04.129 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.129 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.130 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.130 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": 8, 2023-11-05T08:34:04.130 INFO:teuthology.orchestra.run.smithi002.stdout: "uuid": "fcb20ed4-d041-4ef8-bf17-724a43758e50", 2023-11-05T08:34:04.130 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2023-11-05T08:34:04.130 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2023-11-05T08:34:04.130 INFO:teuthology.orchestra.run.smithi002.stdout: "weight": 1, 2023-11-05T08:34:04.131 INFO:teuthology.orchestra.run.smithi002.stdout: "primary_affinity": 1, 2023-11-05T08:34:04.131 INFO:teuthology.orchestra.run.smithi002.stdout: "last_clean_begin": 0, 2023-11-05T08:34:04.131 INFO:teuthology.orchestra.run.smithi002.stdout: "last_clean_end": 0, 2023-11-05T08:34:04.131 INFO:teuthology.orchestra.run.smithi002.stdout: "up_from": 22, 2023-11-05T08:34:04.131 INFO:teuthology.orchestra.run.smithi002.stdout: "up_thru": 93, 2023-11-05T08:34:04.131 INFO:teuthology.orchestra.run.smithi002.stdout: "down_at": 0, 2023-11-05T08:34:04.132 INFO:teuthology.orchestra.run.smithi002.stdout: "lost_at": 0, 2023-11-05T08:34:04.132 INFO:teuthology.orchestra.run.smithi002.stdout: "public_addrs": { 2023-11-05T08:34:04.132 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.132 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.132 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.132 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:6818", 2023-11-05T08:34:04.133 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1795116288 2023-11-05T08:34:04.133 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.133 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.133 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.133 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:6820", 2023-11-05T08:34:04.133 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1795116288 2023-11-05T08:34:04.134 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.134 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.134 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.134 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_addrs": { 2023-11-05T08:34:04.134 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.134 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.135 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.135 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:6822", 2023-11-05T08:34:04.135 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1795116288 2023-11-05T08:34:04.135 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.135 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.135 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.136 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:6823", 2023-11-05T08:34:04.136 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1795116288 2023-11-05T08:34:04.136 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.136 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.136 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.136 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_back_addrs": { 2023-11-05T08:34:04.137 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.137 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.137 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.137 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:6828", 2023-11-05T08:34:04.137 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1795116288 2023-11-05T08:34:04.137 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.138 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.138 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.138 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:6830", 2023-11-05T08:34:04.138 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1795116288 2023-11-05T08:34:04.138 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.138 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.139 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.139 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_front_addrs": { 2023-11-05T08:34:04.139 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.139 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.139 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.139 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:6825", 2023-11-05T08:34:04.140 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1795116288 2023-11-05T08:34:04.140 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.140 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.140 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.140 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:6827", 2023-11-05T08:34:04.140 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1795116288 2023-11-05T08:34:04.141 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.141 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.141 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.141 INFO:teuthology.orchestra.run.smithi002.stdout: "public_addr": "172.21.15.2:6820/1795116288", 2023-11-05T08:34:04.141 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_addr": "172.21.15.2:6823/1795116288", 2023-11-05T08:34:04.141 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_back_addr": "172.21.15.2:6830/1795116288", 2023-11-05T08:34:04.142 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_front_addr": "172.21.15.2:6827/1795116288", 2023-11-05T08:34:04.142 INFO:teuthology.orchestra.run.smithi002.stdout: "state": [ 2023-11-05T08:34:04.142 INFO:teuthology.orchestra.run.smithi002.stdout: "exists", 2023-11-05T08:34:04.142 INFO:teuthology.orchestra.run.smithi002.stdout: "up" 2023-11-05T08:34:04.142 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.142 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.143 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.143 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": 9, 2023-11-05T08:34:04.143 INFO:teuthology.orchestra.run.smithi002.stdout: "uuid": "f85ae771-21d9-4974-a203-7ff48225c8a4", 2023-11-05T08:34:04.143 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2023-11-05T08:34:04.143 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2023-11-05T08:34:04.144 INFO:teuthology.orchestra.run.smithi002.stdout: "weight": 1, 2023-11-05T08:34:04.144 INFO:teuthology.orchestra.run.smithi002.stdout: "primary_affinity": 1, 2023-11-05T08:34:04.144 INFO:teuthology.orchestra.run.smithi002.stdout: "last_clean_begin": 0, 2023-11-05T08:34:04.144 INFO:teuthology.orchestra.run.smithi002.stdout: "last_clean_end": 0, 2023-11-05T08:34:04.144 INFO:teuthology.orchestra.run.smithi002.stdout: "up_from": 21, 2023-11-05T08:34:04.144 INFO:teuthology.orchestra.run.smithi002.stdout: "up_thru": 96, 2023-11-05T08:34:04.144 INFO:teuthology.orchestra.run.smithi002.stdout: "down_at": 0, 2023-11-05T08:34:04.145 INFO:teuthology.orchestra.run.smithi002.stdout: "lost_at": 0, 2023-11-05T08:34:04.145 INFO:teuthology.orchestra.run.smithi002.stdout: "public_addrs": { 2023-11-05T08:34:04.145 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.145 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.145 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.145 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.53:6800", 2023-11-05T08:34:04.146 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1366183694 2023-11-05T08:34:04.146 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.146 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.146 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.146 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.53:6801", 2023-11-05T08:34:04.147 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1366183694 2023-11-05T08:34:04.147 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.147 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.147 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.147 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_addrs": { 2023-11-05T08:34:04.148 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.148 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.148 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.148 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.53:6802", 2023-11-05T08:34:04.148 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1366183694 2023-11-05T08:34:04.148 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.149 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.149 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.149 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.53:6803", 2023-11-05T08:34:04.149 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1366183694 2023-11-05T08:34:04.149 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.149 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.150 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.150 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_back_addrs": { 2023-11-05T08:34:04.150 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.150 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.150 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.151 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.53:6806", 2023-11-05T08:34:04.151 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1366183694 2023-11-05T08:34:04.151 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.151 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.151 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.151 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.53:6807", 2023-11-05T08:34:04.152 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1366183694 2023-11-05T08:34:04.152 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.152 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.152 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.152 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_front_addrs": { 2023-11-05T08:34:04.152 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.153 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.153 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.153 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.53:6804", 2023-11-05T08:34:04.153 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1366183694 2023-11-05T08:34:04.153 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.153 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.154 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.154 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.53:6805", 2023-11-05T08:34:04.154 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1366183694 2023-11-05T08:34:04.154 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.154 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.155 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.155 INFO:teuthology.orchestra.run.smithi002.stdout: "public_addr": "172.21.15.53:6801/1366183694", 2023-11-05T08:34:04.155 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_addr": "172.21.15.53:6803/1366183694", 2023-11-05T08:34:04.155 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_back_addr": "172.21.15.53:6807/1366183694", 2023-11-05T08:34:04.155 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_front_addr": "172.21.15.53:6805/1366183694", 2023-11-05T08:34:04.155 INFO:teuthology.orchestra.run.smithi002.stdout: "state": [ 2023-11-05T08:34:04.156 INFO:teuthology.orchestra.run.smithi002.stdout: "exists", 2023-11-05T08:34:04.156 INFO:teuthology.orchestra.run.smithi002.stdout: "up" 2023-11-05T08:34:04.156 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.156 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.156 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.157 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": 10, 2023-11-05T08:34:04.157 INFO:teuthology.orchestra.run.smithi002.stdout: "uuid": "496442a8-9bc1-4b66-a8bc-294488f6f089", 2023-11-05T08:34:04.157 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2023-11-05T08:34:04.157 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2023-11-05T08:34:04.157 INFO:teuthology.orchestra.run.smithi002.stdout: "weight": 1, 2023-11-05T08:34:04.157 INFO:teuthology.orchestra.run.smithi002.stdout: "primary_affinity": 1, 2023-11-05T08:34:04.158 INFO:teuthology.orchestra.run.smithi002.stdout: "last_clean_begin": 0, 2023-11-05T08:34:04.158 INFO:teuthology.orchestra.run.smithi002.stdout: "last_clean_end": 0, 2023-11-05T08:34:04.158 INFO:teuthology.orchestra.run.smithi002.stdout: "up_from": 22, 2023-11-05T08:34:04.158 INFO:teuthology.orchestra.run.smithi002.stdout: "up_thru": 45, 2023-11-05T08:34:04.158 INFO:teuthology.orchestra.run.smithi002.stdout: "down_at": 0, 2023-11-05T08:34:04.159 INFO:teuthology.orchestra.run.smithi002.stdout: "lost_at": 0, 2023-11-05T08:34:04.159 INFO:teuthology.orchestra.run.smithi002.stdout: "public_addrs": { 2023-11-05T08:34:04.159 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.159 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.159 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.159 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.69:6810", 2023-11-05T08:34:04.160 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2763568647 2023-11-05T08:34:04.160 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.160 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.160 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.160 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.69:6813", 2023-11-05T08:34:04.161 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2763568647 2023-11-05T08:34:04.161 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.161 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.161 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.161 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_addrs": { 2023-11-05T08:34:04.161 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.162 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.162 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.162 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.69:6819", 2023-11-05T08:34:04.162 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2763568647 2023-11-05T08:34:04.162 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.162 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.163 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.163 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.69:6822", 2023-11-05T08:34:04.163 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2763568647 2023-11-05T08:34:04.163 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.163 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.164 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.164 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_back_addrs": { 2023-11-05T08:34:04.164 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.164 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.164 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.164 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.69:6830", 2023-11-05T08:34:04.165 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2763568647 2023-11-05T08:34:04.165 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.165 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.165 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.165 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.69:6831", 2023-11-05T08:34:04.165 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2763568647 2023-11-05T08:34:04.166 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.166 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.166 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.166 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_front_addrs": { 2023-11-05T08:34:04.166 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.167 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.167 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.167 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.69:6825", 2023-11-05T08:34:04.167 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2763568647 2023-11-05T08:34:04.167 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.167 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.167 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.168 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.69:6827", 2023-11-05T08:34:04.168 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2763568647 2023-11-05T08:34:04.168 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.168 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.168 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.168 INFO:teuthology.orchestra.run.smithi002.stdout: "public_addr": "172.21.15.69:6813/2763568647", 2023-11-05T08:34:04.169 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_addr": "172.21.15.69:6822/2763568647", 2023-11-05T08:34:04.169 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_back_addr": "172.21.15.69:6831/2763568647", 2023-11-05T08:34:04.169 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_front_addr": "172.21.15.69:6827/2763568647", 2023-11-05T08:34:04.169 INFO:teuthology.orchestra.run.smithi002.stdout: "state": [ 2023-11-05T08:34:04.169 INFO:teuthology.orchestra.run.smithi002.stdout: "exists", 2023-11-05T08:34:04.170 INFO:teuthology.orchestra.run.smithi002.stdout: "up" 2023-11-05T08:34:04.170 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.170 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.170 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.170 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": 11, 2023-11-05T08:34:04.170 INFO:teuthology.orchestra.run.smithi002.stdout: "uuid": "5680ea76-e74b-43e8-8933-39204c641f00", 2023-11-05T08:34:04.171 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2023-11-05T08:34:04.171 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2023-11-05T08:34:04.171 INFO:teuthology.orchestra.run.smithi002.stdout: "weight": 1, 2023-11-05T08:34:04.171 INFO:teuthology.orchestra.run.smithi002.stdout: "primary_affinity": 1, 2023-11-05T08:34:04.171 INFO:teuthology.orchestra.run.smithi002.stdout: "last_clean_begin": 0, 2023-11-05T08:34:04.171 INFO:teuthology.orchestra.run.smithi002.stdout: "last_clean_end": 0, 2023-11-05T08:34:04.172 INFO:teuthology.orchestra.run.smithi002.stdout: "up_from": 21, 2023-11-05T08:34:04.172 INFO:teuthology.orchestra.run.smithi002.stdout: "up_thru": 45, 2023-11-05T08:34:04.172 INFO:teuthology.orchestra.run.smithi002.stdout: "down_at": 0, 2023-11-05T08:34:04.172 INFO:teuthology.orchestra.run.smithi002.stdout: "lost_at": 0, 2023-11-05T08:34:04.172 INFO:teuthology.orchestra.run.smithi002.stdout: "public_addrs": { 2023-11-05T08:34:04.172 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.173 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.173 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.173 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.90:6816", 2023-11-05T08:34:04.173 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2584956675 2023-11-05T08:34:04.173 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.173 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.174 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.174 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.90:6817", 2023-11-05T08:34:04.174 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2584956675 2023-11-05T08:34:04.174 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.174 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.174 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.175 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_addrs": { 2023-11-05T08:34:04.175 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.175 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.175 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.175 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.90:6818", 2023-11-05T08:34:04.175 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2584956675 2023-11-05T08:34:04.176 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.176 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.176 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.176 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.90:6819", 2023-11-05T08:34:04.176 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2584956675 2023-11-05T08:34:04.176 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.177 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.177 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.177 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_back_addrs": { 2023-11-05T08:34:04.177 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.177 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.178 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.178 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.90:6822", 2023-11-05T08:34:04.178 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2584956675 2023-11-05T08:34:04.178 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.178 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.178 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.179 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.90:6823", 2023-11-05T08:34:04.179 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2584956675 2023-11-05T08:34:04.179 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.179 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.179 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.179 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_front_addrs": { 2023-11-05T08:34:04.180 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.180 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.180 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.180 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.90:6820", 2023-11-05T08:34:04.180 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2584956675 2023-11-05T08:34:04.180 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.181 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.181 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.181 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.90:6821", 2023-11-05T08:34:04.181 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2584956675 2023-11-05T08:34:04.181 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.181 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.182 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.182 INFO:teuthology.orchestra.run.smithi002.stdout: "public_addr": "172.21.15.90:6817/2584956675", 2023-11-05T08:34:04.182 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_addr": "172.21.15.90:6819/2584956675", 2023-11-05T08:34:04.182 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_back_addr": "172.21.15.90:6823/2584956675", 2023-11-05T08:34:04.182 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_front_addr": "172.21.15.90:6821/2584956675", 2023-11-05T08:34:04.183 INFO:teuthology.orchestra.run.smithi002.stdout: "state": [ 2023-11-05T08:34:04.183 INFO:teuthology.orchestra.run.smithi002.stdout: "exists", 2023-11-05T08:34:04.183 INFO:teuthology.orchestra.run.smithi002.stdout: "up" 2023-11-05T08:34:04.183 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.183 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.183 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.184 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": 12, 2023-11-05T08:34:04.184 INFO:teuthology.orchestra.run.smithi002.stdout: "uuid": "2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa", 2023-11-05T08:34:04.184 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2023-11-05T08:34:04.184 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2023-11-05T08:34:04.184 INFO:teuthology.orchestra.run.smithi002.stdout: "weight": 1, 2023-11-05T08:34:04.184 INFO:teuthology.orchestra.run.smithi002.stdout: "primary_affinity": 1, 2023-11-05T08:34:04.184 INFO:teuthology.orchestra.run.smithi002.stdout: "last_clean_begin": 0, 2023-11-05T08:34:04.185 INFO:teuthology.orchestra.run.smithi002.stdout: "last_clean_end": 0, 2023-11-05T08:34:04.185 INFO:teuthology.orchestra.run.smithi002.stdout: "up_from": 22, 2023-11-05T08:34:04.185 INFO:teuthology.orchestra.run.smithi002.stdout: "up_thru": 93, 2023-11-05T08:34:04.185 INFO:teuthology.orchestra.run.smithi002.stdout: "down_at": 0, 2023-11-05T08:34:04.185 INFO:teuthology.orchestra.run.smithi002.stdout: "lost_at": 0, 2023-11-05T08:34:04.185 INFO:teuthology.orchestra.run.smithi002.stdout: "public_addrs": { 2023-11-05T08:34:04.186 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.186 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.186 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.186 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:6819", 2023-11-05T08:34:04.186 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 205377094 2023-11-05T08:34:04.187 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.187 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.187 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.187 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:6821", 2023-11-05T08:34:04.187 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 205377094 2023-11-05T08:34:04.187 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.188 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.188 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.188 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_addrs": { 2023-11-05T08:34:04.188 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.188 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.188 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.189 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:6824", 2023-11-05T08:34:04.189 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 205377094 2023-11-05T08:34:04.189 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.189 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.189 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.189 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:6826", 2023-11-05T08:34:04.190 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 205377094 2023-11-05T08:34:04.190 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.190 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.190 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.190 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_back_addrs": { 2023-11-05T08:34:04.190 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.191 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.191 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.191 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:6832", 2023-11-05T08:34:04.191 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 205377094 2023-11-05T08:34:04.191 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.191 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.191 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.192 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:6833", 2023-11-05T08:34:04.192 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 205377094 2023-11-05T08:34:04.192 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.192 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.192 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.192 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_front_addrs": { 2023-11-05T08:34:04.193 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.193 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.193 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.193 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:6829", 2023-11-05T08:34:04.193 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 205377094 2023-11-05T08:34:04.193 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.194 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.194 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.194 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:6831", 2023-11-05T08:34:04.194 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 205377094 2023-11-05T08:34:04.194 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.195 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.195 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.195 INFO:teuthology.orchestra.run.smithi002.stdout: "public_addr": "172.21.15.2:6821/205377094", 2023-11-05T08:34:04.195 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_addr": "172.21.15.2:6826/205377094", 2023-11-05T08:34:04.195 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_back_addr": "172.21.15.2:6833/205377094", 2023-11-05T08:34:04.195 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_front_addr": "172.21.15.2:6831/205377094", 2023-11-05T08:34:04.195 INFO:teuthology.orchestra.run.smithi002.stdout: "state": [ 2023-11-05T08:34:04.196 INFO:teuthology.orchestra.run.smithi002.stdout: "exists", 2023-11-05T08:34:04.196 INFO:teuthology.orchestra.run.smithi002.stdout: "up" 2023-11-05T08:34:04.196 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.196 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.196 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.197 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": 13, 2023-11-05T08:34:04.197 INFO:teuthology.orchestra.run.smithi002.stdout: "uuid": "356e7e3c-675d-4e78-ba4a-1d993093dc20", 2023-11-05T08:34:04.197 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2023-11-05T08:34:04.197 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2023-11-05T08:34:04.197 INFO:teuthology.orchestra.run.smithi002.stdout: "weight": 1, 2023-11-05T08:34:04.197 INFO:teuthology.orchestra.run.smithi002.stdout: "primary_affinity": 1, 2023-11-05T08:34:04.198 INFO:teuthology.orchestra.run.smithi002.stdout: "last_clean_begin": 0, 2023-11-05T08:34:04.198 INFO:teuthology.orchestra.run.smithi002.stdout: "last_clean_end": 0, 2023-11-05T08:34:04.198 INFO:teuthology.orchestra.run.smithi002.stdout: "up_from": 22, 2023-11-05T08:34:04.198 INFO:teuthology.orchestra.run.smithi002.stdout: "up_thru": 96, 2023-11-05T08:34:04.198 INFO:teuthology.orchestra.run.smithi002.stdout: "down_at": 0, 2023-11-05T08:34:04.198 INFO:teuthology.orchestra.run.smithi002.stdout: "lost_at": 0, 2023-11-05T08:34:04.199 INFO:teuthology.orchestra.run.smithi002.stdout: "public_addrs": { 2023-11-05T08:34:04.199 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.199 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.199 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.199 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.53:6817", 2023-11-05T08:34:04.199 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 714840017 2023-11-05T08:34:04.200 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.200 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.200 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.200 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.53:6819", 2023-11-05T08:34:04.200 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 714840017 2023-11-05T08:34:04.200 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.201 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.201 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.201 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_addrs": { 2023-11-05T08:34:04.201 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.201 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.201 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.202 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.53:6821", 2023-11-05T08:34:04.202 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 714840017 2023-11-05T08:34:04.202 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.202 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.202 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.202 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.53:6823", 2023-11-05T08:34:04.203 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 714840017 2023-11-05T08:34:04.203 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.203 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.203 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.203 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_back_addrs": { 2023-11-05T08:34:04.203 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.204 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.204 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.204 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.53:6830", 2023-11-05T08:34:04.204 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 714840017 2023-11-05T08:34:04.204 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.204 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.205 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.205 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.53:6831", 2023-11-05T08:34:04.205 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 714840017 2023-11-05T08:34:04.205 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.205 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.205 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.206 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_front_addrs": { 2023-11-05T08:34:04.206 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.206 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.206 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.206 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.53:6825", 2023-11-05T08:34:04.206 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 714840017 2023-11-05T08:34:04.207 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.207 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.207 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.207 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.53:6828", 2023-11-05T08:34:04.207 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 714840017 2023-11-05T08:34:04.207 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.208 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.208 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.208 INFO:teuthology.orchestra.run.smithi002.stdout: "public_addr": "172.21.15.53:6819/714840017", 2023-11-05T08:34:04.208 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_addr": "172.21.15.53:6823/714840017", 2023-11-05T08:34:04.208 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_back_addr": "172.21.15.53:6831/714840017", 2023-11-05T08:34:04.208 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_front_addr": "172.21.15.53:6828/714840017", 2023-11-05T08:34:04.209 INFO:teuthology.orchestra.run.smithi002.stdout: "state": [ 2023-11-05T08:34:04.209 INFO:teuthology.orchestra.run.smithi002.stdout: "exists", 2023-11-05T08:34:04.209 INFO:teuthology.orchestra.run.smithi002.stdout: "up" 2023-11-05T08:34:04.209 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.209 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.209 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.210 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": 14, 2023-11-05T08:34:04.210 INFO:teuthology.orchestra.run.smithi002.stdout: "uuid": "1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f", 2023-11-05T08:34:04.210 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2023-11-05T08:34:04.210 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2023-11-05T08:34:04.210 INFO:teuthology.orchestra.run.smithi002.stdout: "weight": 1, 2023-11-05T08:34:04.211 INFO:teuthology.orchestra.run.smithi002.stdout: "primary_affinity": 1, 2023-11-05T08:34:04.211 INFO:teuthology.orchestra.run.smithi002.stdout: "last_clean_begin": 0, 2023-11-05T08:34:04.211 INFO:teuthology.orchestra.run.smithi002.stdout: "last_clean_end": 0, 2023-11-05T08:34:04.211 INFO:teuthology.orchestra.run.smithi002.stdout: "up_from": 22, 2023-11-05T08:34:04.211 INFO:teuthology.orchestra.run.smithi002.stdout: "up_thru": 96, 2023-11-05T08:34:04.211 INFO:teuthology.orchestra.run.smithi002.stdout: "down_at": 0, 2023-11-05T08:34:04.211 INFO:teuthology.orchestra.run.smithi002.stdout: "lost_at": 0, 2023-11-05T08:34:04.212 INFO:teuthology.orchestra.run.smithi002.stdout: "public_addrs": { 2023-11-05T08:34:04.212 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.212 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.212 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.212 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.69:6808", 2023-11-05T08:34:04.213 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 567567051 2023-11-05T08:34:04.213 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.213 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.213 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.213 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.69:6809", 2023-11-05T08:34:04.213 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 567567051 2023-11-05T08:34:04.214 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.214 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.214 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.214 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_addrs": { 2023-11-05T08:34:04.214 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.215 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.215 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.215 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.69:6811", 2023-11-05T08:34:04.217 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 567567051 2023-11-05T08:34:04.218 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.218 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.218 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.218 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.69:6812", 2023-11-05T08:34:04.218 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 567567051 2023-11-05T08:34:04.219 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.219 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.219 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.219 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_back_addrs": { 2023-11-05T08:34:04.219 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.219 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.220 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.220 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.69:6817", 2023-11-05T08:34:04.220 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 567567051 2023-11-05T08:34:04.220 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.220 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.220 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.221 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.69:6820", 2023-11-05T08:34:04.221 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 567567051 2023-11-05T08:34:04.221 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.221 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.221 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.221 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_front_addrs": { 2023-11-05T08:34:04.222 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.222 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.222 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.222 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.69:6814", 2023-11-05T08:34:04.222 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 567567051 2023-11-05T08:34:04.223 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.223 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.223 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.223 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.69:6816", 2023-11-05T08:34:04.223 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 567567051 2023-11-05T08:34:04.223 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.224 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.224 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.224 INFO:teuthology.orchestra.run.smithi002.stdout: "public_addr": "172.21.15.69:6809/567567051", 2023-11-05T08:34:04.224 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_addr": "172.21.15.69:6812/567567051", 2023-11-05T08:34:04.224 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_back_addr": "172.21.15.69:6820/567567051", 2023-11-05T08:34:04.224 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_front_addr": "172.21.15.69:6816/567567051", 2023-11-05T08:34:04.225 INFO:teuthology.orchestra.run.smithi002.stdout: "state": [ 2023-11-05T08:34:04.225 INFO:teuthology.orchestra.run.smithi002.stdout: "exists", 2023-11-05T08:34:04.225 INFO:teuthology.orchestra.run.smithi002.stdout: "up" 2023-11-05T08:34:04.225 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.225 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.226 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.226 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": 15, 2023-11-05T08:34:04.226 INFO:teuthology.orchestra.run.smithi002.stdout: "uuid": "4a37e23e-e1ec-4444-b87b-a9897a448f2a", 2023-11-05T08:34:04.226 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2023-11-05T08:34:04.226 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2023-11-05T08:34:04.226 INFO:teuthology.orchestra.run.smithi002.stdout: "weight": 1, 2023-11-05T08:34:04.227 INFO:teuthology.orchestra.run.smithi002.stdout: "primary_affinity": 1, 2023-11-05T08:34:04.227 INFO:teuthology.orchestra.run.smithi002.stdout: "last_clean_begin": 0, 2023-11-05T08:34:04.227 INFO:teuthology.orchestra.run.smithi002.stdout: "last_clean_end": 0, 2023-11-05T08:34:04.227 INFO:teuthology.orchestra.run.smithi002.stdout: "up_from": 21, 2023-11-05T08:34:04.227 INFO:teuthology.orchestra.run.smithi002.stdout: "up_thru": 40, 2023-11-05T08:34:04.228 INFO:teuthology.orchestra.run.smithi002.stdout: "down_at": 0, 2023-11-05T08:34:04.228 INFO:teuthology.orchestra.run.smithi002.stdout: "lost_at": 0, 2023-11-05T08:34:04.228 INFO:teuthology.orchestra.run.smithi002.stdout: "public_addrs": { 2023-11-05T08:34:04.228 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.228 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.228 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.229 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.90:6824", 2023-11-05T08:34:04.229 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 739151808 2023-11-05T08:34:04.229 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.229 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.229 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.229 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.90:6825", 2023-11-05T08:34:04.230 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 739151808 2023-11-05T08:34:04.230 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.230 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.230 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.230 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_addrs": { 2023-11-05T08:34:04.230 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.231 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.231 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.231 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.90:6826", 2023-11-05T08:34:04.231 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 739151808 2023-11-05T08:34:04.231 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.232 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.232 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.232 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.90:6827", 2023-11-05T08:34:04.232 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 739151808 2023-11-05T08:34:04.232 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.232 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.233 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.233 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_back_addrs": { 2023-11-05T08:34:04.233 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.233 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.233 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.233 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.90:6830", 2023-11-05T08:34:04.234 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 739151808 2023-11-05T08:34:04.234 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.234 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.234 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.234 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.90:6831", 2023-11-05T08:34:04.234 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 739151808 2023-11-05T08:34:04.235 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.235 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.235 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.235 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_front_addrs": { 2023-11-05T08:34:04.235 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-05T08:34:04.235 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.236 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-05T08:34:04.236 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.90:6828", 2023-11-05T08:34:04.236 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 739151808 2023-11-05T08:34:04.236 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.236 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.237 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-05T08:34:04.237 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.90:6829", 2023-11-05T08:34:04.237 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 739151808 2023-11-05T08:34:04.237 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.237 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.237 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.238 INFO:teuthology.orchestra.run.smithi002.stdout: "public_addr": "172.21.15.90:6825/739151808", 2023-11-05T08:34:04.238 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_addr": "172.21.15.90:6827/739151808", 2023-11-05T08:34:04.238 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_back_addr": "172.21.15.90:6831/739151808", 2023-11-05T08:34:04.238 INFO:teuthology.orchestra.run.smithi002.stdout: "heartbeat_front_addr": "172.21.15.90:6829/739151808", 2023-11-05T08:34:04.238 INFO:teuthology.orchestra.run.smithi002.stdout: "state": [ 2023-11-05T08:34:04.238 INFO:teuthology.orchestra.run.smithi002.stdout: "exists", 2023-11-05T08:34:04.239 INFO:teuthology.orchestra.run.smithi002.stdout: "up" 2023-11-05T08:34:04.239 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.239 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.239 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-05T08:34:04.239 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_xinfo": [ 2023-11-05T08:34:04.239 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.240 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": 0, 2023-11-05T08:34:04.240 INFO:teuthology.orchestra.run.smithi002.stdout: "down_stamp": "0.000000", 2023-11-05T08:34:04.240 INFO:teuthology.orchestra.run.smithi002.stdout: "laggy_probability": 0, 2023-11-05T08:34:04.240 INFO:teuthology.orchestra.run.smithi002.stdout: "laggy_interval": 0, 2023-11-05T08:34:04.240 INFO:teuthology.orchestra.run.smithi002.stdout: "features": 4540138322906710015, 2023-11-05T08:34:04.240 INFO:teuthology.orchestra.run.smithi002.stdout: "old_weight": 0, 2023-11-05T08:34:04.241 INFO:teuthology.orchestra.run.smithi002.stdout: "last_purged_snaps_scrub": "2023-11-05T08:32:25.637689+0000", 2023-11-05T08:34:04.241 INFO:teuthology.orchestra.run.smithi002.stdout: "dead_epoch": 37 2023-11-05T08:34:04.241 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.241 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.241 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": 1, 2023-11-05T08:34:04.241 INFO:teuthology.orchestra.run.smithi002.stdout: "down_stamp": "0.000000", 2023-11-05T08:34:04.242 INFO:teuthology.orchestra.run.smithi002.stdout: "laggy_probability": 0, 2023-11-05T08:34:04.242 INFO:teuthology.orchestra.run.smithi002.stdout: "laggy_interval": 0, 2023-11-05T08:34:04.242 INFO:teuthology.orchestra.run.smithi002.stdout: "features": 4540138322906710015, 2023-11-05T08:34:04.242 INFO:teuthology.orchestra.run.smithi002.stdout: "old_weight": 0, 2023-11-05T08:34:04.242 INFO:teuthology.orchestra.run.smithi002.stdout: "last_purged_snaps_scrub": "2023-11-05T08:33:56.093990+0000", 2023-11-05T08:34:04.242 INFO:teuthology.orchestra.run.smithi002.stdout: "dead_epoch": 0 2023-11-05T08:34:04.243 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.243 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.243 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": 2, 2023-11-05T08:34:04.243 INFO:teuthology.orchestra.run.smithi002.stdout: "down_stamp": "0.000000", 2023-11-05T08:34:04.243 INFO:teuthology.orchestra.run.smithi002.stdout: "laggy_probability": 0, 2023-11-05T08:34:04.243 INFO:teuthology.orchestra.run.smithi002.stdout: "laggy_interval": 0, 2023-11-05T08:34:04.244 INFO:teuthology.orchestra.run.smithi002.stdout: "features": 4540138322906710015, 2023-11-05T08:34:04.244 INFO:teuthology.orchestra.run.smithi002.stdout: "old_weight": 0, 2023-11-05T08:34:04.244 INFO:teuthology.orchestra.run.smithi002.stdout: "last_purged_snaps_scrub": "2023-11-05T08:33:48.502629+0000", 2023-11-05T08:34:04.244 INFO:teuthology.orchestra.run.smithi002.stdout: "dead_epoch": 0 2023-11-05T08:34:04.244 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.245 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.245 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": 3, 2023-11-05T08:34:04.245 INFO:teuthology.orchestra.run.smithi002.stdout: "down_stamp": "0.000000", 2023-11-05T08:34:04.245 INFO:teuthology.orchestra.run.smithi002.stdout: "laggy_probability": 0, 2023-11-05T08:34:04.245 INFO:teuthology.orchestra.run.smithi002.stdout: "laggy_interval": 0, 2023-11-05T08:34:04.245 INFO:teuthology.orchestra.run.smithi002.stdout: "features": 4540138322906710015, 2023-11-05T08:34:04.246 INFO:teuthology.orchestra.run.smithi002.stdout: "old_weight": 0, 2023-11-05T08:34:04.246 INFO:teuthology.orchestra.run.smithi002.stdout: "last_purged_snaps_scrub": "2023-11-05T08:33:46.337041+0000", 2023-11-05T08:34:04.246 INFO:teuthology.orchestra.run.smithi002.stdout: "dead_epoch": 45 2023-11-05T08:34:04.246 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.246 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.247 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": 4, 2023-11-05T08:34:04.247 INFO:teuthology.orchestra.run.smithi002.stdout: "down_stamp": "0.000000", 2023-11-05T08:34:04.247 INFO:teuthology.orchestra.run.smithi002.stdout: "laggy_probability": 0, 2023-11-05T08:34:04.247 INFO:teuthology.orchestra.run.smithi002.stdout: "laggy_interval": 0, 2023-11-05T08:34:04.247 INFO:teuthology.orchestra.run.smithi002.stdout: "features": 4540138322906710015, 2023-11-05T08:34:04.247 INFO:teuthology.orchestra.run.smithi002.stdout: "old_weight": 0, 2023-11-05T08:34:04.248 INFO:teuthology.orchestra.run.smithi002.stdout: "last_purged_snaps_scrub": "2023-11-05T08:33:37.463501+0000", 2023-11-05T08:34:04.248 INFO:teuthology.orchestra.run.smithi002.stdout: "dead_epoch": 0 2023-11-05T08:34:04.248 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.248 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.248 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": 5, 2023-11-05T08:34:04.248 INFO:teuthology.orchestra.run.smithi002.stdout: "down_stamp": "0.000000", 2023-11-05T08:34:04.249 INFO:teuthology.orchestra.run.smithi002.stdout: "laggy_probability": 0, 2023-11-05T08:34:04.249 INFO:teuthology.orchestra.run.smithi002.stdout: "laggy_interval": 0, 2023-11-05T08:34:04.249 INFO:teuthology.orchestra.run.smithi002.stdout: "features": 4540138322906710015, 2023-11-05T08:34:04.249 INFO:teuthology.orchestra.run.smithi002.stdout: "old_weight": 0, 2023-11-05T08:34:04.249 INFO:teuthology.orchestra.run.smithi002.stdout: "last_purged_snaps_scrub": "2023-11-05T08:33:54.776271+0000", 2023-11-05T08:34:04.249 INFO:teuthology.orchestra.run.smithi002.stdout: "dead_epoch": 0 2023-11-05T08:34:04.250 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.250 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.250 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": 6, 2023-11-05T08:34:04.250 INFO:teuthology.orchestra.run.smithi002.stdout: "down_stamp": "0.000000", 2023-11-05T08:34:04.250 INFO:teuthology.orchestra.run.smithi002.stdout: "laggy_probability": 0, 2023-11-05T08:34:04.251 INFO:teuthology.orchestra.run.smithi002.stdout: "laggy_interval": 0, 2023-11-05T08:34:04.251 INFO:teuthology.orchestra.run.smithi002.stdout: "features": 4540138322906710015, 2023-11-05T08:34:04.251 INFO:teuthology.orchestra.run.smithi002.stdout: "old_weight": 0, 2023-11-05T08:34:04.251 INFO:teuthology.orchestra.run.smithi002.stdout: "last_purged_snaps_scrub": "2023-11-05T08:33:46.876488+0000", 2023-11-05T08:34:04.251 INFO:teuthology.orchestra.run.smithi002.stdout: "dead_epoch": 0 2023-11-05T08:34:04.251 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.252 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.252 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": 7, 2023-11-05T08:34:04.252 INFO:teuthology.orchestra.run.smithi002.stdout: "down_stamp": "0.000000", 2023-11-05T08:34:04.252 INFO:teuthology.orchestra.run.smithi002.stdout: "laggy_probability": 0, 2023-11-05T08:34:04.252 INFO:teuthology.orchestra.run.smithi002.stdout: "laggy_interval": 0, 2023-11-05T08:34:04.252 INFO:teuthology.orchestra.run.smithi002.stdout: "features": 4540138322906710015, 2023-11-05T08:34:04.253 INFO:teuthology.orchestra.run.smithi002.stdout: "old_weight": 0, 2023-11-05T08:34:04.253 INFO:teuthology.orchestra.run.smithi002.stdout: "last_purged_snaps_scrub": "2023-11-05T08:33:52.787222+0000", 2023-11-05T08:34:04.253 INFO:teuthology.orchestra.run.smithi002.stdout: "dead_epoch": 0 2023-11-05T08:34:04.253 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.253 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.253 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": 8, 2023-11-05T08:34:04.254 INFO:teuthology.orchestra.run.smithi002.stdout: "down_stamp": "0.000000", 2023-11-05T08:34:04.254 INFO:teuthology.orchestra.run.smithi002.stdout: "laggy_probability": 0, 2023-11-05T08:34:04.254 INFO:teuthology.orchestra.run.smithi002.stdout: "laggy_interval": 0, 2023-11-05T08:34:04.254 INFO:teuthology.orchestra.run.smithi002.stdout: "features": 4540138322906710015, 2023-11-05T08:34:04.254 INFO:teuthology.orchestra.run.smithi002.stdout: "old_weight": 0, 2023-11-05T08:34:04.254 INFO:teuthology.orchestra.run.smithi002.stdout: "last_purged_snaps_scrub": "2023-11-05T08:33:37.669537+0000", 2023-11-05T08:34:04.255 INFO:teuthology.orchestra.run.smithi002.stdout: "dead_epoch": 0 2023-11-05T08:34:04.255 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.255 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.255 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": 9, 2023-11-05T08:34:04.255 INFO:teuthology.orchestra.run.smithi002.stdout: "down_stamp": "0.000000", 2023-11-05T08:34:04.255 INFO:teuthology.orchestra.run.smithi002.stdout: "laggy_probability": 0, 2023-11-05T08:34:04.256 INFO:teuthology.orchestra.run.smithi002.stdout: "laggy_interval": 0, 2023-11-05T08:34:04.256 INFO:teuthology.orchestra.run.smithi002.stdout: "features": 4540138322906710015, 2023-11-05T08:34:04.256 INFO:teuthology.orchestra.run.smithi002.stdout: "old_weight": 0, 2023-11-05T08:34:04.256 INFO:teuthology.orchestra.run.smithi002.stdout: "last_purged_snaps_scrub": "2023-11-05T08:33:45.201083+0000", 2023-11-05T08:34:04.256 INFO:teuthology.orchestra.run.smithi002.stdout: "dead_epoch": 0 2023-11-05T08:34:04.256 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.257 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.257 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": 10, 2023-11-05T08:34:04.257 INFO:teuthology.orchestra.run.smithi002.stdout: "down_stamp": "0.000000", 2023-11-05T08:34:04.257 INFO:teuthology.orchestra.run.smithi002.stdout: "laggy_probability": 0, 2023-11-05T08:34:04.257 INFO:teuthology.orchestra.run.smithi002.stdout: "laggy_interval": 0, 2023-11-05T08:34:04.257 INFO:teuthology.orchestra.run.smithi002.stdout: "features": 4540138322906710015, 2023-11-05T08:34:04.258 INFO:teuthology.orchestra.run.smithi002.stdout: "old_weight": 0, 2023-11-05T08:34:04.258 INFO:teuthology.orchestra.run.smithi002.stdout: "last_purged_snaps_scrub": "2023-11-05T08:33:51.913597+0000", 2023-11-05T08:34:04.258 INFO:teuthology.orchestra.run.smithi002.stdout: "dead_epoch": 0 2023-11-05T08:34:04.258 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.258 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.258 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": 11, 2023-11-05T08:34:04.259 INFO:teuthology.orchestra.run.smithi002.stdout: "down_stamp": "0.000000", 2023-11-05T08:34:04.259 INFO:teuthology.orchestra.run.smithi002.stdout: "laggy_probability": 0, 2023-11-05T08:34:04.259 INFO:teuthology.orchestra.run.smithi002.stdout: "laggy_interval": 0, 2023-11-05T08:34:04.259 INFO:teuthology.orchestra.run.smithi002.stdout: "features": 4540138322906710015, 2023-11-05T08:34:04.259 INFO:teuthology.orchestra.run.smithi002.stdout: "old_weight": 0, 2023-11-05T08:34:04.259 INFO:teuthology.orchestra.run.smithi002.stdout: "last_purged_snaps_scrub": "2023-11-05T08:33:26.992305+0000", 2023-11-05T08:34:04.260 INFO:teuthology.orchestra.run.smithi002.stdout: "dead_epoch": 0 2023-11-05T08:34:04.260 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.260 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.260 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": 12, 2023-11-05T08:34:04.260 INFO:teuthology.orchestra.run.smithi002.stdout: "down_stamp": "0.000000", 2023-11-05T08:34:04.260 INFO:teuthology.orchestra.run.smithi002.stdout: "laggy_probability": 0, 2023-11-05T08:34:04.261 INFO:teuthology.orchestra.run.smithi002.stdout: "laggy_interval": 0, 2023-11-05T08:34:04.261 INFO:teuthology.orchestra.run.smithi002.stdout: "features": 4540138322906710015, 2023-11-05T08:34:04.261 INFO:teuthology.orchestra.run.smithi002.stdout: "old_weight": 0, 2023-11-05T08:34:04.261 INFO:teuthology.orchestra.run.smithi002.stdout: "last_purged_snaps_scrub": "2023-11-05T08:33:29.690235+0000", 2023-11-05T08:34:04.261 INFO:teuthology.orchestra.run.smithi002.stdout: "dead_epoch": 0 2023-11-05T08:34:04.261 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.262 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.262 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": 13, 2023-11-05T08:34:04.262 INFO:teuthology.orchestra.run.smithi002.stdout: "down_stamp": "0.000000", 2023-11-05T08:34:04.262 INFO:teuthology.orchestra.run.smithi002.stdout: "laggy_probability": 0, 2023-11-05T08:34:04.262 INFO:teuthology.orchestra.run.smithi002.stdout: "laggy_interval": 0, 2023-11-05T08:34:04.263 INFO:teuthology.orchestra.run.smithi002.stdout: "features": 4540138322906710015, 2023-11-05T08:34:04.263 INFO:teuthology.orchestra.run.smithi002.stdout: "old_weight": 0, 2023-11-05T08:34:04.263 INFO:teuthology.orchestra.run.smithi002.stdout: "last_purged_snaps_scrub": "2023-11-05T08:33:33.658634+0000", 2023-11-05T08:34:04.263 INFO:teuthology.orchestra.run.smithi002.stdout: "dead_epoch": 0 2023-11-05T08:34:04.263 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.263 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.264 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": 14, 2023-11-05T08:34:04.264 INFO:teuthology.orchestra.run.smithi002.stdout: "down_stamp": "0.000000", 2023-11-05T08:34:04.264 INFO:teuthology.orchestra.run.smithi002.stdout: "laggy_probability": 0, 2023-11-05T08:34:04.264 INFO:teuthology.orchestra.run.smithi002.stdout: "laggy_interval": 0, 2023-11-05T08:34:04.264 INFO:teuthology.orchestra.run.smithi002.stdout: "features": 4540138322906710015, 2023-11-05T08:34:04.264 INFO:teuthology.orchestra.run.smithi002.stdout: "old_weight": 0, 2023-11-05T08:34:04.265 INFO:teuthology.orchestra.run.smithi002.stdout: "last_purged_snaps_scrub": "2023-11-05T08:33:40.245020+0000", 2023-11-05T08:34:04.265 INFO:teuthology.orchestra.run.smithi002.stdout: "dead_epoch": 0 2023-11-05T08:34:04.265 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.265 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.265 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": 15, 2023-11-05T08:34:04.265 INFO:teuthology.orchestra.run.smithi002.stdout: "down_stamp": "0.000000", 2023-11-05T08:34:04.266 INFO:teuthology.orchestra.run.smithi002.stdout: "laggy_probability": 0, 2023-11-05T08:34:04.266 INFO:teuthology.orchestra.run.smithi002.stdout: "laggy_interval": 0, 2023-11-05T08:34:04.266 INFO:teuthology.orchestra.run.smithi002.stdout: "features": 4540138322906710015, 2023-11-05T08:34:04.266 INFO:teuthology.orchestra.run.smithi002.stdout: "old_weight": 0, 2023-11-05T08:34:04.266 INFO:teuthology.orchestra.run.smithi002.stdout: "last_purged_snaps_scrub": "2023-11-05T08:33:43.650883+0000", 2023-11-05T08:34:04.266 INFO:teuthology.orchestra.run.smithi002.stdout: "dead_epoch": 0 2023-11-05T08:34:04.267 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.267 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-05T08:34:04.267 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_upmap": [], 2023-11-05T08:34:04.267 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_upmap_items": [], 2023-11-05T08:34:04.267 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_upmap_primaries": [], 2023-11-05T08:34:04.267 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_temp": [ 2023-11-05T08:34:04.268 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.268 INFO:teuthology.orchestra.run.smithi002.stdout: "pgid": "3.10", 2023-11-05T08:34:04.268 INFO:teuthology.orchestra.run.smithi002.stdout: "osds": [ 2023-11-05T08:34:04.268 INFO:teuthology.orchestra.run.smithi002.stdout: 13, 2023-11-05T08:34:04.268 INFO:teuthology.orchestra.run.smithi002.stdout: 6, 2023-11-05T08:34:04.268 INFO:teuthology.orchestra.run.smithi002.stdout: 7, 2023-11-05T08:34:04.269 INFO:teuthology.orchestra.run.smithi002.stdout: 4, 2023-11-05T08:34:04.269 INFO:teuthology.orchestra.run.smithi002.stdout: 8, 2023-11-05T08:34:04.269 INFO:teuthology.orchestra.run.smithi002.stdout: 6, 2023-11-05T08:34:04.269 INFO:teuthology.orchestra.run.smithi002.stdout: 11 2023-11-05T08:34:04.269 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.270 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.270 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.270 INFO:teuthology.orchestra.run.smithi002.stdout: "pgid": "3.13", 2023-11-05T08:34:04.270 INFO:teuthology.orchestra.run.smithi002.stdout: "osds": [ 2023-11-05T08:34:04.270 INFO:teuthology.orchestra.run.smithi002.stdout: 11, 2023-11-05T08:34:04.271 INFO:teuthology.orchestra.run.smithi002.stdout: 7, 2023-11-05T08:34:04.271 INFO:teuthology.orchestra.run.smithi002.stdout: 9, 2023-11-05T08:34:04.271 INFO:teuthology.orchestra.run.smithi002.stdout: 13, 2023-11-05T08:34:04.271 INFO:teuthology.orchestra.run.smithi002.stdout: 2, 2023-11-05T08:34:04.271 INFO:teuthology.orchestra.run.smithi002.stdout: 4, 2023-11-05T08:34:04.271 INFO:teuthology.orchestra.run.smithi002.stdout: 10 2023-11-05T08:34:04.272 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.272 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.272 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.272 INFO:teuthology.orchestra.run.smithi002.stdout: "pgid": "3.18", 2023-11-05T08:34:04.272 INFO:teuthology.orchestra.run.smithi002.stdout: "osds": [ 2023-11-05T08:34:04.273 INFO:teuthology.orchestra.run.smithi002.stdout: 13, 2023-11-05T08:34:04.273 INFO:teuthology.orchestra.run.smithi002.stdout: 6, 2023-11-05T08:34:04.273 INFO:teuthology.orchestra.run.smithi002.stdout: 2, 2023-11-05T08:34:04.273 INFO:teuthology.orchestra.run.smithi002.stdout: 4, 2023-11-05T08:34:04.273 INFO:teuthology.orchestra.run.smithi002.stdout: 5, 2023-11-05T08:34:04.273 INFO:teuthology.orchestra.run.smithi002.stdout: 7, 2023-11-05T08:34:04.274 INFO:teuthology.orchestra.run.smithi002.stdout: 12 2023-11-05T08:34:04.274 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.274 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.274 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-05T08:34:04.274 INFO:teuthology.orchestra.run.smithi002.stdout: "primary_temp": [], 2023-11-05T08:34:04.274 INFO:teuthology.orchestra.run.smithi002.stdout: "blocklist": {}, 2023-11-05T08:34:04.275 INFO:teuthology.orchestra.run.smithi002.stdout: "range_blocklist": {}, 2023-11-05T08:34:04.275 INFO:teuthology.orchestra.run.smithi002.stdout: "erasure_code_profiles": { 2023-11-05T08:34:04.275 INFO:teuthology.orchestra.run.smithi002.stdout: "default": { 2023-11-05T08:34:04.275 INFO:teuthology.orchestra.run.smithi002.stdout: "crush-failure-domain": "osd", 2023-11-05T08:34:04.275 INFO:teuthology.orchestra.run.smithi002.stdout: "k": "2", 2023-11-05T08:34:04.276 INFO:teuthology.orchestra.run.smithi002.stdout: "m": "1", 2023-11-05T08:34:04.276 INFO:teuthology.orchestra.run.smithi002.stdout: "plugin": "jerasure", 2023-11-05T08:34:04.276 INFO:teuthology.orchestra.run.smithi002.stdout: "technique": "reed_sol_van" 2023-11-05T08:34:04.276 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.276 INFO:teuthology.orchestra.run.smithi002.stdout: "shecprofile": { 2023-11-05T08:34:04.276 INFO:teuthology.orchestra.run.smithi002.stdout: "c": "2", 2023-11-05T08:34:04.277 INFO:teuthology.orchestra.run.smithi002.stdout: "crush-device-class": "", 2023-11-05T08:34:04.277 INFO:teuthology.orchestra.run.smithi002.stdout: "crush-failure-domain": "osd", 2023-11-05T08:34:04.277 INFO:teuthology.orchestra.run.smithi002.stdout: "crush-root": "default", 2023-11-05T08:34:04.277 INFO:teuthology.orchestra.run.smithi002.stdout: "k": "4", 2023-11-05T08:34:04.277 INFO:teuthology.orchestra.run.smithi002.stdout: "m": "3", 2023-11-05T08:34:04.277 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "shecprofile", 2023-11-05T08:34:04.278 INFO:teuthology.orchestra.run.smithi002.stdout: "plugin": "shec", 2023-11-05T08:34:04.278 INFO:teuthology.orchestra.run.smithi002.stdout: "technique": "multiple" 2023-11-05T08:34:04.278 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.278 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.278 INFO:teuthology.orchestra.run.smithi002.stdout: "removed_snaps_queue": [ 2023-11-05T08:34:04.279 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.279 INFO:teuthology.orchestra.run.smithi002.stdout: "pool": 3, 2023-11-05T08:34:04.279 INFO:teuthology.orchestra.run.smithi002.stdout: "snaps": [ 2023-11-05T08:34:04.279 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.279 INFO:teuthology.orchestra.run.smithi002.stdout: "begin": 2, 2023-11-05T08:34:04.279 INFO:teuthology.orchestra.run.smithi002.stdout: "length": 2 2023-11-05T08:34:04.280 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.280 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.280 INFO:teuthology.orchestra.run.smithi002.stdout: "begin": 5, 2023-11-05T08:34:04.280 INFO:teuthology.orchestra.run.smithi002.stdout: "length": 4 2023-11-05T08:34:04.280 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.280 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.281 INFO:teuthology.orchestra.run.smithi002.stdout: "begin": 11, 2023-11-05T08:34:04.281 INFO:teuthology.orchestra.run.smithi002.stdout: "length": 1 2023-11-05T08:34:04.281 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.281 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.281 INFO:teuthology.orchestra.run.smithi002.stdout: "begin": 14, 2023-11-05T08:34:04.281 INFO:teuthology.orchestra.run.smithi002.stdout: "length": 1 2023-11-05T08:34:04.282 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.282 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.282 INFO:teuthology.orchestra.run.smithi002.stdout: "begin": 16, 2023-11-05T08:34:04.282 INFO:teuthology.orchestra.run.smithi002.stdout: "length": 1 2023-11-05T08:34:04.282 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.283 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.283 INFO:teuthology.orchestra.run.smithi002.stdout: "begin": 19, 2023-11-05T08:34:04.283 INFO:teuthology.orchestra.run.smithi002.stdout: "length": 1 2023-11-05T08:34:04.283 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.283 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.283 INFO:teuthology.orchestra.run.smithi002.stdout: "begin": 21, 2023-11-05T08:34:04.284 INFO:teuthology.orchestra.run.smithi002.stdout: "length": 1 2023-11-05T08:34:04.284 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.284 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.284 INFO:teuthology.orchestra.run.smithi002.stdout: "begin": 23, 2023-11-05T08:34:04.284 INFO:teuthology.orchestra.run.smithi002.stdout: "length": 2 2023-11-05T08:34:04.284 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.285 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.285 INFO:teuthology.orchestra.run.smithi002.stdout: "begin": 26, 2023-11-05T08:34:04.285 INFO:teuthology.orchestra.run.smithi002.stdout: "length": 3 2023-11-05T08:34:04.285 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.285 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.285 INFO:teuthology.orchestra.run.smithi002.stdout: "begin": 33, 2023-11-05T08:34:04.286 INFO:teuthology.orchestra.run.smithi002.stdout: "length": 1 2023-11-05T08:34:04.286 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.286 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.286 INFO:teuthology.orchestra.run.smithi002.stdout: "begin": 35, 2023-11-05T08:34:04.286 INFO:teuthology.orchestra.run.smithi002.stdout: "length": 1 2023-11-05T08:34:04.287 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.287 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.287 INFO:teuthology.orchestra.run.smithi002.stdout: "begin": 37, 2023-11-05T08:34:04.287 INFO:teuthology.orchestra.run.smithi002.stdout: "length": 3 2023-11-05T08:34:04.287 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.287 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.288 INFO:teuthology.orchestra.run.smithi002.stdout: "begin": 43, 2023-11-05T08:34:04.288 INFO:teuthology.orchestra.run.smithi002.stdout: "length": 2 2023-11-05T08:34:04.288 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.288 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.288 INFO:teuthology.orchestra.run.smithi002.stdout: "begin": 46, 2023-11-05T08:34:04.288 INFO:teuthology.orchestra.run.smithi002.stdout: "length": 3 2023-11-05T08:34:04.289 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.289 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.289 INFO:teuthology.orchestra.run.smithi002.stdout: "begin": 52, 2023-11-05T08:34:04.289 INFO:teuthology.orchestra.run.smithi002.stdout: "length": 1 2023-11-05T08:34:04.289 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.289 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.290 INFO:teuthology.orchestra.run.smithi002.stdout: "begin": 54, 2023-11-05T08:34:04.290 INFO:teuthology.orchestra.run.smithi002.stdout: "length": 1 2023-11-05T08:34:04.290 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.290 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.290 INFO:teuthology.orchestra.run.smithi002.stdout: "begin": 56, 2023-11-05T08:34:04.291 INFO:teuthology.orchestra.run.smithi002.stdout: "length": 1 2023-11-05T08:34:04.291 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.291 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.291 INFO:teuthology.orchestra.run.smithi002.stdout: "begin": 59, 2023-11-05T08:34:04.291 INFO:teuthology.orchestra.run.smithi002.stdout: "length": 1 2023-11-05T08:34:04.291 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.291 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.292 INFO:teuthology.orchestra.run.smithi002.stdout: "begin": 61, 2023-11-05T08:34:04.292 INFO:teuthology.orchestra.run.smithi002.stdout: "length": 2 2023-11-05T08:34:04.292 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.292 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.292 INFO:teuthology.orchestra.run.smithi002.stdout: "begin": 65, 2023-11-05T08:34:04.293 INFO:teuthology.orchestra.run.smithi002.stdout: "length": 1 2023-11-05T08:34:04.293 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.293 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.293 INFO:teuthology.orchestra.run.smithi002.stdout: "begin": 68, 2023-11-05T08:34:04.293 INFO:teuthology.orchestra.run.smithi002.stdout: "length": 3 2023-11-05T08:34:04.293 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:04.293 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.294 INFO:teuthology.orchestra.run.smithi002.stdout: "begin": 73, 2023-11-05T08:34:04.294 INFO:teuthology.orchestra.run.smithi002.stdout: "length": 1 2023-11-05T08:34:04.294 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.294 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.294 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.294 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-05T08:34:04.295 INFO:teuthology.orchestra.run.smithi002.stdout: "new_removed_snaps": [ 2023-11-05T08:34:04.295 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.295 INFO:teuthology.orchestra.run.smithi002.stdout: "pool": 3, 2023-11-05T08:34:04.295 INFO:teuthology.orchestra.run.smithi002.stdout: "snaps": [ 2023-11-05T08:34:04.295 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:04.295 INFO:teuthology.orchestra.run.smithi002.stdout: "begin": 70, 2023-11-05T08:34:04.296 INFO:teuthology.orchestra.run.smithi002.stdout: "length": 1 2023-11-05T08:34:04.296 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.296 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:04.296 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.296 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-05T08:34:04.296 INFO:teuthology.orchestra.run.smithi002.stdout: "new_purged_snaps": [], 2023-11-05T08:34:04.297 INFO:teuthology.orchestra.run.smithi002.stdout: "crush_node_flags": {}, 2023-11-05T08:34:04.297 INFO:teuthology.orchestra.run.smithi002.stdout: "device_class_flags": {}, 2023-11-05T08:34:04.297 INFO:teuthology.orchestra.run.smithi002.stdout: "stretch_mode": { 2023-11-05T08:34:04.297 INFO:teuthology.orchestra.run.smithi002.stdout: "stretch_mode_enabled": false, 2023-11-05T08:34:04.297 INFO:teuthology.orchestra.run.smithi002.stdout: "stretch_bucket_count": 0, 2023-11-05T08:34:04.297 INFO:teuthology.orchestra.run.smithi002.stdout: "degraded_stretch_mode": 0, 2023-11-05T08:34:04.298 INFO:teuthology.orchestra.run.smithi002.stdout: "recovering_stretch_mode": 0, 2023-11-05T08:34:04.298 INFO:teuthology.orchestra.run.smithi002.stdout: "stretch_mode_bucket": 0 2023-11-05T08:34:04.298 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:04.298 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:34:04.298 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-05T08:34:04.300 INFO:tasks.rados.rados.0.smithi090.stdout:383: finishing write tid 1 to smithi09039252-16 2023-11-05T08:34:04.300 INFO:tasks.rados.rados.0.smithi090.stdout:383: finishing write tid 2 to smithi09039252-16 2023-11-05T08:34:04.300 INFO:tasks.rados.rados.0.smithi090.stdout:383: finishing write tid 3 to smithi09039252-16 2023-11-05T08:34:04.300 INFO:tasks.rados.rados.0.smithi090.stdout:383: finishing write tid 4 to smithi09039252-16 2023-11-05T08:34:04.301 INFO:tasks.rados.rados.0.smithi090.stdout:383: finishing write tid 5 to smithi09039252-16 2023-11-05T08:34:04.301 INFO:tasks.rados.rados.0.smithi090.stdout:383: finishing write tid 6 to smithi09039252-16 2023-11-05T08:34:04.301 INFO:tasks.rados.rados.0.smithi090.stdout:383: oid 16 updating version 0 to 34 2023-11-05T08:34:04.301 INFO:tasks.rados.rados.0.smithi090.stdout:383: oid 16 version 34 is already newer than 33 2023-11-05T08:34:04.301 INFO:tasks.rados.rados.0.smithi090.stdout:383: oid 16 updating version 34 to 37 2023-11-05T08:34:04.301 INFO:tasks.rados.rados.0.smithi090.stdout:383: oid 16 version 37 is already newer than 35 2023-11-05T08:34:04.302 INFO:tasks.rados.rados.0.smithi090.stdout:383: oid 16 version 37 is already newer than 36 2023-11-05T08:34:04.302 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 16 v 37 (ObjNum 142 snap 38 seq_num 142) dirty exists 2023-11-05T08:34:04.302 INFO:tasks.rados.rados.0.smithi090.stdout:383: left oid 16 (ObjNum 142 snap 38 seq_num 142) 2023-11-05T08:34:04.303 INFO:tasks.thrashosds.thrasher:j is {'epoch': 110, 'fsid': '8f1fa9b5-06de-4da7-985f-30d4ad103e2d', 'created': '2023-11-05T08:32:15.953286+0000', 'modified': '2023-11-05T08:34:03.819986+0000', 'last_up_change': '2023-11-05T08:33:46.664317+0000', 'last_in_change': '2023-11-05T08:32:20.284340+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 16, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-05T08:32:26.748874+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 15.380000114440918, 'score_stable': 15.380000114440918, 'optimal_score': 0.12999999523162842, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.940394401550293, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.9399999976158142}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-11-05T08:32:30.772862+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 4, 'score_stable': 4, 'optimal_score': 0.5, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.940394401550293, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.9399999976158142}}, {'pool': 3, 'pool_name': 'unique_pool_0', 'create_time': '2023-11-05T08:32:42.050417+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 3, 'size': 7, 'min_size': 5, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '40', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 76, 'snap_epoch': 110, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'shecprofile', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 16384, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '3626fa17-5bde-458d-96fb-f5e70ac6edd6', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 30, 'down_at': 37, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.2:6802', 'nonce': 1948090427}, {'type': 'v1', 'addr': '172.21.15.2:6803', 'nonce': 1948090427}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.2:6804', 'nonce': 1948090427}, {'type': 'v1', 'addr': '172.21.15.2:6805', 'nonce': 1948090427}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.2:6808', 'nonce': 1948090427}, {'type': 'v1', 'addr': '172.21.15.2:6809', 'nonce': 1948090427}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.2:6806', 'nonce': 1948090427}, {'type': 'v1', 'addr': '172.21.15.2:6807', 'nonce': 1948090427}]}, 'public_addr': '172.21.15.2:6803/1948090427', 'cluster_addr': '172.21.15.2:6805/1948090427', 'heartbeat_back_addr': '172.21.15.2:6809/1948090427', 'heartbeat_front_addr': '172.21.15.2:6807/1948090427', 'state': ['exists']}, {'osd': 1, 'uuid': '8f524504-91f5-44f3-afbd-69dfc19f03d3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6816', 'nonce': 121537835}, {'type': 'v1', 'addr': '172.21.15.53:6818', 'nonce': 121537835}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6820', 'nonce': 121537835}, {'type': 'v1', 'addr': '172.21.15.53:6822', 'nonce': 121537835}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6827', 'nonce': 121537835}, {'type': 'v1', 'addr': '172.21.15.53:6829', 'nonce': 121537835}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6824', 'nonce': 121537835}, {'type': 'v1', 'addr': '172.21.15.53:6826', 'nonce': 121537835}]}, 'public_addr': '172.21.15.53:6818/121537835', 'cluster_addr': '172.21.15.53:6822/121537835', 'heartbeat_back_addr': '172.21.15.53:6829/121537835', 'heartbeat_front_addr': '172.21.15.53:6826/121537835', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '696f71d0-a98b-46b5-bdc0-0da3d6065469', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 30, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6800', 'nonce': 1463424062}, {'type': 'v1', 'addr': '172.21.15.69:6801', 'nonce': 1463424062}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6802', 'nonce': 1463424062}, {'type': 'v1', 'addr': '172.21.15.69:6803', 'nonce': 1463424062}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6806', 'nonce': 1463424062}, {'type': 'v1', 'addr': '172.21.15.69:6807', 'nonce': 1463424062}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6804', 'nonce': 1463424062}, {'type': 'v1', 'addr': '172.21.15.69:6805', 'nonce': 1463424062}]}, 'public_addr': '172.21.15.69:6801/1463424062', 'cluster_addr': '172.21.15.69:6803/1463424062', 'heartbeat_back_addr': '172.21.15.69:6807/1463424062', 'heartbeat_front_addr': '172.21.15.69:6805/1463424062', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'fc979d30-659b-47f4-b06b-97a4d41c2e90', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.0463104248046875, 'last_clean_begin': 21, 'last_clean_end': 43, 'up_from': 93, 'up_thru': 43, 'down_at': 45, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6800', 'nonce': 2085535590}, {'type': 'v1', 'addr': '172.21.15.90:6801', 'nonce': 2085535590}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6802', 'nonce': 2085535590}, {'type': 'v1', 'addr': '172.21.15.90:6803', 'nonce': 2085535590}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6806', 'nonce': 2085535590}, {'type': 'v1', 'addr': '172.21.15.90:6807', 'nonce': 2085535590}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6804', 'nonce': 2085535590}, {'type': 'v1', 'addr': '172.21.15.90:6805', 'nonce': 2085535590}]}, 'public_addr': '172.21.15.90:6801/2085535590', 'cluster_addr': '172.21.15.90:6803/2085535590', 'heartbeat_back_addr': '172.21.15.90:6807/2085535590', 'heartbeat_front_addr': '172.21.15.90:6805/2085535590', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '2ed87cb9-9ced-46fd-8346-3f47709bb69e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 44, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.2:6810', 'nonce': 2916800887}, {'type': 'v1', 'addr': '172.21.15.2:6811', 'nonce': 2916800887}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.2:6812', 'nonce': 2916800887}, {'type': 'v1', 'addr': '172.21.15.2:6813', 'nonce': 2916800887}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.2:6816', 'nonce': 2916800887}, {'type': 'v1', 'addr': '172.21.15.2:6817', 'nonce': 2916800887}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.2:6814', 'nonce': 2916800887}, {'type': 'v1', 'addr': '172.21.15.2:6815', 'nonce': 2916800887}]}, 'public_addr': '172.21.15.2:6811/2916800887', 'cluster_addr': '172.21.15.2:6813/2916800887', 'heartbeat_back_addr': '172.21.15.2:6817/2916800887', 'heartbeat_front_addr': '172.21.15.2:6815/2916800887', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '5bb59d8d-9b93-48b2-a5ee-13646b19ce50', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6808', 'nonce': 588137330}, {'type': 'v1', 'addr': '172.21.15.53:6809', 'nonce': 588137330}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6810', 'nonce': 588137330}, {'type': 'v1', 'addr': '172.21.15.53:6811', 'nonce': 588137330}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6814', 'nonce': 588137330}, {'type': 'v1', 'addr': '172.21.15.53:6815', 'nonce': 588137330}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6812', 'nonce': 588137330}, {'type': 'v1', 'addr': '172.21.15.53:6813', 'nonce': 588137330}]}, 'public_addr': '172.21.15.53:6809/588137330', 'cluster_addr': '172.21.15.53:6811/588137330', 'heartbeat_back_addr': '172.21.15.53:6815/588137330', 'heartbeat_front_addr': '172.21.15.53:6813/588137330', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '46cd8f39-841d-4b7b-a4ef-6ec6493eea5c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 93, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6815', 'nonce': 3720624375}, {'type': 'v1', 'addr': '172.21.15.69:6818', 'nonce': 3720624375}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6821', 'nonce': 3720624375}, {'type': 'v1', 'addr': '172.21.15.69:6823', 'nonce': 3720624375}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6828', 'nonce': 3720624375}, {'type': 'v1', 'addr': '172.21.15.69:6829', 'nonce': 3720624375}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6824', 'nonce': 3720624375}, {'type': 'v1', 'addr': '172.21.15.69:6826', 'nonce': 3720624375}]}, 'public_addr': '172.21.15.69:6818/3720624375', 'cluster_addr': '172.21.15.69:6823/3720624375', 'heartbeat_back_addr': '172.21.15.69:6829/3720624375', 'heartbeat_front_addr': '172.21.15.69:6826/3720624375', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'c98d93d9-c1e5-4ac0-8f71-9fc1d2767963', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6808', 'nonce': 2028724524}, {'type': 'v1', 'addr': '172.21.15.90:6809', 'nonce': 2028724524}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6810', 'nonce': 2028724524}, {'type': 'v1', 'addr': '172.21.15.90:6811', 'nonce': 2028724524}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6814', 'nonce': 2028724524}, {'type': 'v1', 'addr': '172.21.15.90:6815', 'nonce': 2028724524}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6812', 'nonce': 2028724524}, {'type': 'v1', 'addr': '172.21.15.90:6813', 'nonce': 2028724524}]}, 'public_addr': '172.21.15.90:6809/2028724524', 'cluster_addr': '172.21.15.90:6811/2028724524', 'heartbeat_back_addr': '172.21.15.90:6815/2028724524', 'heartbeat_front_addr': '172.21.15.90:6813/2028724524', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'fcb20ed4-d041-4ef8-bf17-724a43758e50', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 93, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.2:6818', 'nonce': 1795116288}, {'type': 'v1', 'addr': '172.21.15.2:6820', 'nonce': 1795116288}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.2:6822', 'nonce': 1795116288}, {'type': 'v1', 'addr': '172.21.15.2:6823', 'nonce': 1795116288}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.2:6828', 'nonce': 1795116288}, {'type': 'v1', 'addr': '172.21.15.2:6830', 'nonce': 1795116288}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.2:6825', 'nonce': 1795116288}, {'type': 'v1', 'addr': '172.21.15.2:6827', 'nonce': 1795116288}]}, 'public_addr': '172.21.15.2:6820/1795116288', 'cluster_addr': '172.21.15.2:6823/1795116288', 'heartbeat_back_addr': '172.21.15.2:6830/1795116288', 'heartbeat_front_addr': '172.21.15.2:6827/1795116288', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'f85ae771-21d9-4974-a203-7ff48225c8a4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6800', 'nonce': 1366183694}, {'type': 'v1', 'addr': '172.21.15.53:6801', 'nonce': 1366183694}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6802', 'nonce': 1366183694}, {'type': 'v1', 'addr': '172.21.15.53:6803', 'nonce': 1366183694}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6806', 'nonce': 1366183694}, {'type': 'v1', 'addr': '172.21.15.53:6807', 'nonce': 1366183694}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6804', 'nonce': 1366183694}, {'type': 'v1', 'addr': '172.21.15.53:6805', 'nonce': 1366183694}]}, 'public_addr': '172.21.15.53:6801/1366183694', 'cluster_addr': '172.21.15.53:6803/1366183694', 'heartbeat_back_addr': '172.21.15.53:6807/1366183694', 'heartbeat_front_addr': '172.21.15.53:6805/1366183694', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '496442a8-9bc1-4b66-a8bc-294488f6f089', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 45, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6810', 'nonce': 2763568647}, {'type': 'v1', 'addr': '172.21.15.69:6813', 'nonce': 2763568647}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6819', 'nonce': 2763568647}, {'type': 'v1', 'addr': '172.21.15.69:6822', 'nonce': 2763568647}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6830', 'nonce': 2763568647}, {'type': 'v1', 'addr': '172.21.15.69:6831', 'nonce': 2763568647}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6825', 'nonce': 2763568647}, {'type': 'v1', 'addr': '172.21.15.69:6827', 'nonce': 2763568647}]}, 'public_addr': '172.21.15.69:6813/2763568647', 'cluster_addr': '172.21.15.69:6822/2763568647', 'heartbeat_back_addr': '172.21.15.69:6831/2763568647', 'heartbeat_front_addr': '172.21.15.69:6827/2763568647', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '5680ea76-e74b-43e8-8933-39204c641f00', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 45, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6816', 'nonce': 2584956675}, {'type': 'v1', 'addr': '172.21.15.90:6817', 'nonce': 2584956675}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6818', 'nonce': 2584956675}, {'type': 'v1', 'addr': '172.21.15.90:6819', 'nonce': 2584956675}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6822', 'nonce': 2584956675}, {'type': 'v1', 'addr': '172.21.15.90:6823', 'nonce': 2584956675}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6820', 'nonce': 2584956675}, {'type': 'v1', 'addr': '172.21.15.90:6821', 'nonce': 2584956675}]}, 'public_addr': '172.21.15.90:6817/2584956675', 'cluster_addr': '172.21.15.90:6819/2584956675', 'heartbeat_back_addr': '172.21.15.90:6823/2584956675', 'heartbeat_front_addr': '172.21.15.90:6821/2584956675', 'state': ['exists', 'up']}, {'osd': 12, 'uuid': '2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 93, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.2:6819', 'nonce': 205377094}, {'type': 'v1', 'addr': '172.21.15.2:6821', 'nonce': 205377094}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.2:6824', 'nonce': 205377094}, {'type': 'v1', 'addr': '172.21.15.2:6826', 'nonce': 205377094}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.2:6832', 'nonce': 205377094}, {'type': 'v1', 'addr': '172.21.15.2:6833', 'nonce': 205377094}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.2:6829', 'nonce': 205377094}, {'type': 'v1', 'addr': '172.21.15.2:6831', 'nonce': 205377094}]}, 'public_addr': '172.21.15.2:6821/205377094', 'cluster_addr': '172.21.15.2:6826/205377094', 'heartbeat_back_addr': '172.21.15.2:6833/205377094', 'heartbeat_front_addr': '172.21.15.2:6831/205377094', 'state': ['exists', 'up']}, {'osd': 13, 'uuid': '356e7e3c-675d-4e78-ba4a-1d993093dc20', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6817', 'nonce': 714840017}, {'type': 'v1', 'addr': '172.21.15.53:6819', 'nonce': 714840017}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6821', 'nonce': 714840017}, {'type': 'v1', 'addr': '172.21.15.53:6823', 'nonce': 714840017}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6830', 'nonce': 714840017}, {'type': 'v1', 'addr': '172.21.15.53:6831', 'nonce': 714840017}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6825', 'nonce': 714840017}, {'type': 'v1', 'addr': '172.21.15.53:6828', 'nonce': 714840017}]}, 'public_addr': '172.21.15.53:6819/714840017', 'cluster_addr': '172.21.15.53:6823/714840017', 'heartbeat_back_addr': '172.21.15.53:6831/714840017', 'heartbeat_front_addr': '172.21.15.53:6828/714840017', 'state': ['exists', 'up']}, {'osd': 14, 'uuid': '1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6808', 'nonce': 567567051}, {'type': 'v1', 'addr': '172.21.15.69:6809', 'nonce': 567567051}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6811', 'nonce': 567567051}, {'type': 'v1', 'addr': '172.21.15.69:6812', 'nonce': 567567051}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6817', 'nonce': 567567051}, {'type': 'v1', 'addr': '172.21.15.69:6820', 'nonce': 567567051}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6814', 'nonce': 567567051}, {'type': 'v1', 'addr': '172.21.15.69:6816', 'nonce': 567567051}]}, 'public_addr': '172.21.15.69:6809/567567051', 'cluster_addr': '172.21.15.69:6812/567567051', 'heartbeat_back_addr': '172.21.15.69:6820/567567051', 'heartbeat_front_addr': '172.21.15.69:6816/567567051', 'state': ['exists', 'up']}, {'osd': 15, 'uuid': '4a37e23e-e1ec-4444-b87b-a9897a448f2a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 40, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6824', 'nonce': 739151808}, {'type': 'v1', 'addr': '172.21.15.90:6825', 'nonce': 739151808}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6826', 'nonce': 739151808}, {'type': 'v1', 'addr': '172.21.15.90:6827', 'nonce': 739151808}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6830', 'nonce': 739151808}, {'type': 'v1', 'addr': '172.21.15.90:6831', 'nonce': 739151808}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6828', 'nonce': 739151808}, {'type': 'v1', 'addr': '172.21.15.90:6829', 'nonce': 739151808}]}, 'public_addr': '172.21.15.90:6825/739151808', 'cluster_addr': '172.21.15.90:6827/739151808', 'heartbeat_back_addr': '172.21.15.90:6831/739151808', 'heartbeat_front_addr': '172.21.15.90:6829/739151808', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-11-05T08:32:25.637689+0000', 'dead_epoch': 37}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-11-05T08:33:56.093990+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-11-05T08:33:48.502629+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-11-05T08:33:46.337041+0000', 'dead_epoch': 45}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-11-05T08:33:37.463501+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-11-05T08:33:54.776271+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-11-05T08:33:46.876488+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-11-05T08:33:52.787222+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-11-05T08:33:37.669537+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-11-05T08:33:45.201083+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-11-05T08:33:51.913597+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-11-05T08:33:26.992305+0000', 'dead_epoch': 0}, {'osd': 12, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-11-05T08:33:29.690235+0000', 'dead_epoch': 0}, {'osd': 13, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-11-05T08:33:33.658634+0000', 'dead_epoch': 0}, {'osd': 14, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-11-05T08:33:40.245020+0000', 'dead_epoch': 0}, {'osd': 15, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-11-05T08:33:43.650883+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '3.10', 'osds': [13, 6, 7, 4, 8, 6, 11]}, {'pgid': '3.13', 'osds': [11, 7, 9, 13, 2, 4, 10]}, {'pgid': '3.18', 'osds': [13, 6, 2, 4, 5, 7, 12]}], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}, 'shecprofile': {'c': '2', 'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'k': '4', 'm': '3', 'name': 'shecprofile', 'plugin': 'shec', 'technique': 'multiple'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 5, 'length': 4}, {'begin': 11, 'length': 1}, {'begin': 14, 'length': 1}, {'begin': 16, 'length': 1}, {'begin': 19, 'length': 1}, {'begin': 21, 'length': 1}, {'begin': 23, 'length': 2}, {'begin': 26, 'length': 3}, {'begin': 33, 'length': 1}, {'begin': 35, 'length': 1}, {'begin': 37, 'length': 3}, {'begin': 43, 'length': 2}, {'begin': 46, 'length': 3}, {'begin': 52, 'length': 1}, {'begin': 54, 'length': 1}, {'begin': 56, 'length': 1}, {'begin': 59, 'length': 1}, {'begin': 61, 'length': 2}, {'begin': 65, 'length': 1}, {'begin': 68, 'length': 3}, {'begin': 73, 'length': 1}]}], 'new_removed_snaps': [{'pool': 3, 'snaps': [{'begin': 70, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2023-11-05T08:34:04.304 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-05T08:34:04.304 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-05T08:34:04.305 INFO:teuthology.orchestra.run.smithi002.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-05T08:34:04.306 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-11-05T08:34:04.306 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:34:04.297+0000 7fdc5cd58640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:34:04.401 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:34:04.396+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:34:04.455 INFO:teuthology.orchestra.run.smithi002.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-05T08:34:04.458 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-05T08:34:04.458 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-11-05T08:34:04.489 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-11-05T08:34:04.502 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:34:04.495+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:34:04.511 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1: {} 2023-11-05T08:34:04.511 INFO:teuthology.orchestra.run.smithi002.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-11-05T08:34:04.531 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2: {} 2023-11-05T08:34:04.532 INFO:teuthology.orchestra.run.smithi002.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-11-05T08:34:04.552 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3: {} 2023-11-05T08:34:04.552 INFO:teuthology.orchestra.run.smithi002.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-11-05T08:34:04.563 INFO:teuthology.orchestra.run.smithi002.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-05T08:34:04.566 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-05T08:34:04.566 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-11-05T08:34:04.569 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4: {} 2023-11-05T08:34:04.570 INFO:teuthology.orchestra.run.smithi002.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-11-05T08:34:04.590 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5: {} 2023-11-05T08:34:04.590 INFO:teuthology.orchestra.run.smithi002.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-11-05T08:34:04.603 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:34:04.599+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:34:04.611 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6: {} 2023-11-05T08:34:04.611 INFO:teuthology.orchestra.run.smithi002.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-11-05T08:34:04.633 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7: {} 2023-11-05T08:34:04.634 INFO:teuthology.orchestra.run.smithi002.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-11-05T08:34:04.651 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:34:04.653 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-11-05T08:34:04.654 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8: {} 2023-11-05T08:34:04.654 INFO:teuthology.orchestra.run.smithi002.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-11-05T08:34:04.671 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":142,"stamp":"2023-11-05T08:34:03.822995+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":116531744,"num_objects":111,"num_object_clones":61,"num_object_copies":767,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":71,"num_objects_misplaced":8,"num_objects_unfound":0,"num_objects_dirty":111,"num_whiteouts":4,"num_read":480,"num_read_kb":152086,"num_write":856,"num_write_kb":119260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":23068672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1159,"ondisk_log_size":1159,"up":199,"acting":199,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":190,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":665528,"kb_used_data":198740,"kb_used_omap":29,"kb_used_meta":466530,"kb_avail":1509283912,"statfs":{"total":1546188226560,"available":1545506725888,"internally_reserved":0,"allocated":203509760,"data_stored":305774549,"data_compressed":2512463,"data_compressed_allocated":5447680,"data_compressed_original":11788288,"omap_allocated":30361,"internal_metadata":477727079},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7061504,"num_objects":6,"num_object_clones":6,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":20,"num_read_kb":30871,"num_write":43,"num_write_kb":6896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.038272"},"pg_stats":[{"pgid":"3.19","version":"108'58","reported_seq":359,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-05T08:34:02.297869+0000","last_change":"2023-11-05T08:34:01.803774+0000","last_active":"2023-11-05T08:34:02.297869+0000","last_peered":"2023-11-05T08:34:02.297869+0000","last_clean":"2023-11-05T08:34:02.297869+0000","last_became_active":"2023-11-05T08:32:59.211522+0000","last_became_peered":"2023-11-05T08:32:59.211522+0000","last_unstale":"2023-11-05T08:34:02.297869+0000","last_undegraded":"2023-11-05T08:34:02.297869+0000","last_fullsized":"2023-11-05T08:34:02.297869+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":5,"last_scrub":"39'22","last_scrub_stamp":"2023-11-05T08:33:01.870959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:01.870959+0000","objects_scrubbed":3,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:14.506291+0000","scrub_duration":0.0087276880000000008,"objects_trimmed":0,"snaptrim_duration":0.00096281600000000002,"stat_sum":{"num_bytes":5734400,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":3027,"num_write":11,"num_write_kb":4144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,11,9,13,7,1],"acting":[8,4,11,9,13,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.18","version":"109'63","reported_seq":358,"reported_epoch":109,"state":"active+clean+remapped","last_fresh":"2023-11-05T08:34:02.859311+0000","last_change":"2023-11-05T08:34:02.862802+0000","last_active":"2023-11-05T08:34:02.859311+0000","last_peered":"2023-11-05T08:34:02.859311+0000","last_clean":"2023-11-05T08:34:02.859311+0000","last_became_active":"2023-11-05T08:32:59.217241+0000","last_became_peered":"2023-11-05T08:32:59.217241+0000","last_unstale":"2023-11-05T08:34:02.859311+0000","last_undegraded":"2023-11-05T08:34:02.859311+0000","last_fullsized":"2023-11-05T08:34:02.859311+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":5,"last_scrub":"52'28","last_scrub_stamp":"2023-11-05T08:33:13.971775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:13.971775+0000","objects_scrubbed":5,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:35.661759+0000","scrub_duration":0.019915236999999999,"objects_trimmed":0,"snaptrim_duration":0.037471048,"stat_sum":{"num_bytes":6160384,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":3739,"num_write":31,"num_write_kb":4336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,2147483647,4,5,7,12],"acting":[13,6,2,4,5,7,12],"avail_no_missing":["13(0)","2(2)","4(3)","5(4)","6(1)","7(5)","12(6)"],"object_location_counts":[{"shards":"2(2),4(3),5(4),6(1),7(5),12(6),13(0)","objects":4}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"44","length":"2"},{"start":"49","length":"1"}]},{"pgid":"3.17","version":"105'36","reported_seq":383,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-05T08:34:02.203149+0000","last_change":"2023-11-05T08:34:02.204287+0000","last_active":"2023-11-05T08:34:02.203149+0000","last_peered":"2023-11-05T08:34:02.203149+0000","last_clean":"2023-11-05T08:34:02.203149+0000","last_became_active":"2023-11-05T08:33:50.677365+0000","last_became_peered":"2023-11-05T08:33:50.677365+0000","last_unstale":"2023-11-05T08:34:02.203149+0000","last_undegraded":"2023-11-05T08:34:02.203149+0000","last_fullsized":"2023-11-05T08:34:02.203149+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"49'17","last_scrub_stamp":"2023-11-05T08:33:13.975356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:13.975356+0000","objects_scrubbed":3,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:33.996216+0000","scrub_duration":0.014066680999999999,"objects_trimmed":0,"snaptrim_duration":0.001635697,"stat_sum":{"num_bytes":2588672,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":45,"num_write_kb":2176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":999424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,10,9,7,1,13],"acting":[5,3,10,9,7,1,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.16","version":"97'48","reported_seq":326,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-05T08:34:01.820510+0000","last_change":"2023-11-05T08:34:01.821079+0000","last_active":"2023-11-05T08:34:01.820510+0000","last_peered":"2023-11-05T08:34:01.820510+0000","last_clean":"2023-11-05T08:34:01.820510+0000","last_became_active":"2023-11-05T08:32:59.210858+0000","last_became_peered":"2023-11-05T08:32:59.210858+0000","last_unstale":"2023-11-05T08:34:01.820510+0000","last_undegraded":"2023-11-05T08:34:01.820510+0000","last_fullsized":"2023-11-05T08:34:01.820510+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":5,"last_scrub":"46'13","last_scrub_stamp":"2023-11-05T08:33:00.659926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:00.659926+0000","objects_scrubbed":1,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:06.720172+0000","scrub_duration":0.0077151709999999998,"objects_trimmed":0,"snaptrim_duration":0.00073938499999999996,"stat_sum":{"num_bytes":4669440,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":15,"num_write_kb":4560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15,4,8,9,5,10],"acting":[11,15,4,8,9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.15","version":"36'15","reported_seq":268,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-05T08:34:01.805163+0000","last_change":"2023-11-05T08:34:01.806289+0000","last_active":"2023-11-05T08:34:01.805163+0000","last_peered":"2023-11-05T08:34:01.805163+0000","last_clean":"2023-11-05T08:34:01.805163+0000","last_became_active":"2023-11-05T08:33:50.675923+0000","last_became_peered":"2023-11-05T08:33:50.675923+0000","last_unstale":"2023-11-05T08:34:01.805163+0000","last_undegraded":"2023-11-05T08:34:01.805163+0000","last_fullsized":"2023-11-05T08:34:01.805163+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"36'15","last_scrub_stamp":"2023-11-05T08:33:52.537969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:52.537969+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:35:09.298059+0000","scrub_duration":0.0054037720000000003,"objects_trimmed":0,"snaptrim_duration":0.0020680849999999999,"stat_sum":{"num_bytes":32768,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,3,13,14,2,15],"acting":[9,6,3,13,14,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.14","version":"101'27","reported_seq":281,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-05T08:34:01.806326+0000","last_change":"2023-11-05T08:34:01.807260+0000","last_active":"2023-11-05T08:34:01.806326+0000","last_peered":"2023-11-05T08:34:01.806326+0000","last_clean":"2023-11-05T08:34:01.806326+0000","last_became_active":"2023-11-05T08:33:50.674132+0000","last_became_peered":"2023-11-05T08:33:50.674132+0000","last_unstale":"2023-11-05T08:34:01.806326+0000","last_undegraded":"2023-11-05T08:34:01.806326+0000","last_fullsized":"2023-11-05T08:34:01.806326+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"49'18","last_scrub_stamp":"2023-11-05T08:33:04.812402+0000","last_deep_scrub":"49'18","last_deep_scrub_stamp":"2023-11-05T08:33:04.812402+0000","last_clean_scrub_stamp":"2023-11-05T08:33:04.812402+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:09.862950+0000","scrub_duration":0.0086643829999999995,"objects_trimmed":0,"snaptrim_duration":0.0023598249999999999,"stat_sum":{"num_bytes":294912,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1966080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,15,1,13,8,4],"acting":[3,9,15,1,13,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":98,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-05T08:34:01.152657+0000","last_change":"2023-11-05T08:33:50.142139+0000","last_active":"2023-11-05T08:34:01.152657+0000","last_peered":"2023-11-05T08:34:01.152657+0000","last_clean":"2023-11-05T08:34:01.152657+0000","last_became_active":"2023-11-05T08:32:31.941858+0000","last_became_peered":"2023-11-05T08:32:31.941858+0000","last_unstale":"2023-11-05T08:34:01.152657+0000","last_undegraded":"2023-11-05T08:34:01.152657+0000","last_fullsized":"2023-11-05T08:34:01.152657+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:33:50.141700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:33:50.141700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:51.467865+0000","scrub_duration":0.0025514660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"35'9","reported_seq":186,"reported_epoch":109,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:34:02.826600+0000","last_change":"2023-11-05T08:32:54.153555+0000","last_active":"2023-11-05T08:34:02.826600+0000","last_peered":"2023-11-05T08:34:02.826600+0000","last_clean":"2023-11-05T08:32:48.102597+0000","last_became_active":"2023-11-05T08:32:54.153555+0000","last_became_peered":"2023-11-05T08:32:54.153555+0000","last_unstale":"2023-11-05T08:34:02.826600+0000","last_undegraded":"2023-11-05T08:32:54.121924+0000","last_fullsized":"2023-11-05T08:32:54.120000+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":36,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1089536,"num_objects":1,"num_object_clones":1,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":1064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,13,1,15,12,7,9],"acting":[2147483647,13,1,15,12,7,9],"avail_no_missing":["13(1)","1(2)","7(5)","9(6)","12(4)","15(3)"],"object_location_counts":[{"shards":"1(2),7(5),9(6),12(4),13(1),15(3)","objects":1}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":99,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-05T08:34:01.451381+0000","last_change":"2023-11-05T08:33:52.971633+0000","last_active":"2023-11-05T08:34:01.451381+0000","last_peered":"2023-11-05T08:34:01.451381+0000","last_clean":"2023-11-05T08:34:01.451381+0000","last_became_active":"2023-11-05T08:32:31.941207+0000","last_became_peered":"2023-11-05T08:32:31.941207+0000","last_unstale":"2023-11-05T08:34:01.451381+0000","last_undegraded":"2023-11-05T08:34:01.451381+0000","last_fullsized":"2023-11-05T08:34:01.451381+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:33:52.971456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:33:52.971456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:35:07.735098+0000","scrub_duration":0.002374414,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"39'12","reported_seq":197,"reported_epoch":108,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:34:01.929244+0000","last_change":"2023-11-05T08:32:54.151639+0000","last_active":"2023-11-05T08:34:01.929244+0000","last_peered":"2023-11-05T08:34:01.929244+0000","last_clean":"2023-11-05T08:32:49.074431+0000","last_became_active":"2023-11-05T08:32:54.151639+0000","last_became_peered":"2023-11-05T08:32:54.151639+0000","last_unstale":"2023-11-05T08:34:01.929244+0000","last_undegraded":"2023-11-05T08:32:54.120694+0000","last_fullsized":"2023-11-05T08:32:54.119652+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":34,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030081180000000002,"stat_sum":{"num_bytes":319488,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,2147483647,8,5,11],"acting":[4,6,9,2147483647,8,5,11],"avail_no_missing":["4(0)","5(5)","6(1)","8(4)","9(2)","11(6)"],"object_location_counts":[{"shards":"4(0),5(5),6(1),8(4),9(2),11(6)","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":99,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-11-05T08:33:59.863392+0000","last_change":"2023-11-05T08:33:50.446617+0000","last_active":"2023-11-05T08:33:59.863392+0000","last_peered":"2023-11-05T08:33:59.863392+0000","last_clean":"2023-11-05T08:33:59.863392+0000","last_became_active":"2023-11-05T08:32:31.940312+0000","last_became_peered":"2023-11-05T08:32:31.940312+0000","last_unstale":"2023-11-05T08:33:59.863392+0000","last_undegraded":"2023-11-05T08:33:59.863392+0000","last_fullsized":"2023-11-05T08:33:59.863392+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:33:50.446312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:33:50.446312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:57.987976+0000","scrub_duration":0.0037389620000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"39'12","reported_seq":196,"reported_epoch":108,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:34:01.803840+0000","last_change":"2023-11-05T08:32:54.141915+0000","last_active":"2023-11-05T08:34:01.803840+0000","last_peered":"2023-11-05T08:34:01.803840+0000","last_clean":"2023-11-05T08:32:49.088117+0000","last_became_active":"2023-11-05T08:32:54.141915+0000","last_became_peered":"2023-11-05T08:32:54.141915+0000","last_unstale":"2023-11-05T08:34:01.803840+0000","last_undegraded":"2023-11-05T08:32:54.115420+0000","last_fullsized":"2023-11-05T08:32:54.114438+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":34,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.011061272,"stat_sum":{"num_bytes":1376256,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,4,1,2147483647,14,13],"acting":[8,2,4,1,2147483647,14,13],"avail_no_missing":["8(0)","1(3)","2(1)","4(2)","13(6)","14(5)"],"object_location_counts":[{"shards":"1(3),2(1),4(2),8(0),13(6),14(5)","objects":1}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":96,"reported_epoch":107,"state":"active+undersized","last_fresh":"2023-11-05T08:34:01.632325+0000","last_change":"2023-11-05T08:32:51.087583+0000","last_active":"2023-11-05T08:34:01.632325+0000","last_peered":"2023-11-05T08:34:01.632325+0000","last_clean":"2023-11-05T08:32:49.071396+0000","last_became_active":"2023-11-05T08:32:51.087583+0000","last_became_peered":"2023-11-05T08:32:51.087583+0000","last_unstale":"2023-11-05T08:34:01.632325+0000","last_undegraded":"2023-11-05T08:34:01.632325+0000","last_fullsized":"2023-11-05T08:32:51.085658+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"103'31","reported_seq":222,"reported_epoch":108,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:34:02.250234+0000","last_change":"2023-11-05T08:32:54.139002+0000","last_active":"2023-11-05T08:34:02.250234+0000","last_peered":"2023-11-05T08:34:02.250234+0000","last_clean":"2023-11-05T08:32:49.088591+0000","last_became_active":"2023-11-05T08:32:54.139002+0000","last_became_peered":"2023-11-05T08:32:54.139002+0000","last_unstale":"2023-11-05T08:34:02.250234+0000","last_undegraded":"2023-11-05T08:32:54.118261+0000","last_fullsized":"2023-11-05T08:32:54.117408+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":34,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.020413489999999999,"stat_sum":{"num_bytes":5988352,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":12,"num_read_kb":2382,"num_write":18,"num_write_kb":5848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,4,13,7,2147483647,6],"acting":[15,8,4,13,7,2147483647,6],"avail_no_missing":["15(0)","4(2)","6(6)","7(4)","8(1)","13(3)"],"object_location_counts":[{"shards":"4(2),6(6),7(4),8(1),13(3),15(0)","objects":5}],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.3","version":"97'17","reported_seq":383,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-05T08:34:02.861946+0000","last_change":"2023-11-05T08:34:02.865894+0000","last_active":"2023-11-05T08:34:02.861946+0000","last_peered":"2023-11-05T08:34:02.861946+0000","last_clean":"2023-11-05T08:34:02.861946+0000","last_became_active":"2023-11-05T08:33:50.683698+0000","last_became_peered":"2023-11-05T08:33:50.683698+0000","last_unstale":"2023-11-05T08:34:02.861946+0000","last_undegraded":"2023-11-05T08:34:02.861946+0000","last_fullsized":"2023-11-05T08:34:02.861946+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'17","last_scrub_stamp":"2023-11-05T08:33:53.073917+0000","last_deep_scrub":"97'17","last_deep_scrub_stamp":"2023-11-05T08:33:53.073917+0000","last_clean_scrub_stamp":"2023-11-05T08:33:53.073917+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:35:22.729630+0000","scrub_duration":0.0068216960000000004,"objects_trimmed":0,"snaptrim_duration":0.040008424000000001,"stat_sum":{"num_bytes":524288,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":1537,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,4,15,2,5,3],"acting":[13,7,4,15,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"44","length":"2"},{"start":"49","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":104,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-05T08:34:00.798911+0000","last_change":"2023-11-05T08:33:53.537740+0000","last_active":"2023-11-05T08:34:00.798911+0000","last_peered":"2023-11-05T08:34:00.798911+0000","last_clean":"2023-11-05T08:34:00.798911+0000","last_became_active":"2023-11-05T08:33:47.679907+0000","last_became_peered":"2023-11-05T08:33:47.679907+0000","last_unstale":"2023-11-05T08:34:00.798911+0000","last_undegraded":"2023-11-05T08:34:00.798911+0000","last_fullsized":"2023-11-05T08:34:00.798911+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:33:53.537442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:33:53.537442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:35:15.667752+0000","scrub_duration":0.003485906,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3],"acting":[12,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.0","version":"33'9","reported_seq":342,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-05T08:34:01.808241+0000","last_change":"2023-11-05T08:34:01.808699+0000","last_active":"2023-11-05T08:34:01.808241+0000","last_peered":"2023-11-05T08:34:01.808241+0000","last_clean":"2023-11-05T08:34:01.808241+0000","last_became_active":"2023-11-05T08:32:54.144790+0000","last_became_peered":"2023-11-05T08:32:54.144790+0000","last_unstale":"2023-11-05T08:34:01.808241+0000","last_undegraded":"2023-11-05T08:34:01.808241+0000","last_fullsized":"2023-11-05T08:34:01.808241+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'9","last_scrub_stamp":"2023-11-05T08:33:03.810524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:03.810524+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:07.644773+0000","scrub_duration":0.0066041449999999996,"objects_trimmed":0,"snaptrim_duration":0.00083072500000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,15,1,5,6,4],"acting":[14,2,15,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":99,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-05T08:34:02.933086+0000","last_change":"2023-11-05T08:34:02.933179+0000","last_active":"2023-11-05T08:34:02.933086+0000","last_peered":"2023-11-05T08:34:02.933086+0000","last_clean":"2023-11-05T08:34:02.933086+0000","last_became_active":"2023-11-05T08:32:31.940564+0000","last_became_peered":"2023-11-05T08:32:31.940564+0000","last_unstale":"2023-11-05T08:34:02.933086+0000","last_undegraded":"2023-11-05T08:34:02.933086+0000","last_fullsized":"2023-11-05T08:34:02.933086+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:34:02.932904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:34:02.932904+0000","last_clean_scrub_stamp":"2023-11-05T08:34:02.932904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:35:11.818210+0000","scrub_duration":0.0033566400000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.1","version":"106'40","reported_seq":385,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-05T08:34:01.927993+0000","last_change":"2023-11-05T08:34:01.928527+0000","last_active":"2023-11-05T08:34:01.927993+0000","last_peered":"2023-11-05T08:34:01.927993+0000","last_clean":"2023-11-05T08:34:01.927993+0000","last_became_active":"2023-11-05T08:32:54.144895+0000","last_became_peered":"2023-11-05T08:32:54.144895+0000","last_unstale":"2023-11-05T08:34:01.927993+0000","last_undegraded":"2023-11-05T08:34:01.927993+0000","last_fullsized":"2023-11-05T08:34:01.927993+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'11","last_scrub_stamp":"2023-11-05T08:33:04.003557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:04.003557+0000","objects_scrubbed":3,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:15.185676+0000","scrub_duration":0.010761797,"objects_trimmed":0,"snaptrim_duration":0.00073924699999999995,"stat_sum":{"num_bytes":1949696,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":772,"num_write":7,"num_write_kb":2480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,5,10,2,13],"acting":[4,11,9,5,10,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":96,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-11-05T08:33:59.863171+0000","last_change":"2023-11-05T08:33:47.700202+0000","last_active":"2023-11-05T08:33:59.863171+0000","last_peered":"2023-11-05T08:33:59.863171+0000","last_clean":"2023-11-05T08:33:59.863171+0000","last_became_active":"2023-11-05T08:33:47.699438+0000","last_became_peered":"2023-11-05T08:33:47.699438+0000","last_unstale":"2023-11-05T08:33:59.863171+0000","last_undegraded":"2023-11-05T08:33:59.863171+0000","last_fullsized":"2023-11-05T08:33:59.863171+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"102'30","reported_seq":398,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-11-05T08:34:02.857713+0000","last_change":"2023-11-05T08:34:02.861748+0000","last_active":"2023-11-05T08:34:02.857713+0000","last_peered":"2023-11-05T08:34:02.857713+0000","last_clean":"2023-11-05T08:34:02.857713+0000","last_became_active":"2023-11-05T08:33:50.678084+0000","last_became_peered":"2023-11-05T08:33:50.678084+0000","last_unstale":"2023-11-05T08:34:02.857713+0000","last_undegraded":"2023-11-05T08:34:02.857713+0000","last_fullsized":"2023-11-05T08:34:02.857713+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'25","last_scrub_stamp":"2023-11-05T08:33:52.100890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:52.100890+0000","objects_scrubbed":1,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:35:12.125601+0000","scrub_duration":0.007958952,"objects_trimmed":0,"snaptrim_duration":0.036335376000000003,"stat_sum":{"num_bytes":3670016,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,3,6,15,1,11],"acting":[13,2,3,6,15,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"44","length":"2"},{"start":"49","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":99,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-11-05T08:34:01.452342+0000","last_change":"2023-11-05T08:34:00.133458+0000","last_active":"2023-11-05T08:34:01.452342+0000","last_peered":"2023-11-05T08:34:01.452342+0000","last_clean":"2023-11-05T08:34:01.452342+0000","last_became_active":"2023-11-05T08:32:31.939705+0000","last_became_peered":"2023-11-05T08:32:31.939705+0000","last_unstale":"2023-11-05T08:34:01.452342+0000","last_undegraded":"2023-11-05T08:34:01.452342+0000","last_fullsized":"2023-11-05T08:34:01.452342+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:34:00.133184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:34:00.133184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:35:05.010050+0000","scrub_duration":0.0035944599999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"25'157","reported_seq":298,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-05T08:34:01.920856+0000","last_change":"2023-11-05T08:33:41.087862+0000","last_active":"2023-11-05T08:34:01.920856+0000","last_peered":"2023-11-05T08:34:01.920856+0000","last_clean":"2023-11-05T08:34:01.920856+0000","last_became_active":"2023-11-05T08:32:28.758834+0000","last_became_peered":"2023-11-05T08:32:28.758834+0000","last_unstale":"2023-11-05T08:34:01.920856+0000","last_undegraded":"2023-11-05T08:34:01.920856+0000","last_fullsized":"2023-11-05T08:34:01.920856+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'157","last_scrub_stamp":"2023-11-05T08:33:41.087696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:27.693099+0000","last_clean_scrub_stamp":"2023-11-05T08:33:41.087696+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:35:09.900066+0000","scrub_duration":0.0045831459999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.8","version":"108'34","reported_seq":222,"reported_epoch":108,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:34:02.300393+0000","last_change":"2023-11-05T08:32:54.151946+0000","last_active":"2023-11-05T08:34:02.300393+0000","last_peered":"2023-11-05T08:34:02.300393+0000","last_clean":"2023-11-05T08:32:49.090022+0000","last_became_active":"2023-11-05T08:32:54.151946+0000","last_became_peered":"2023-11-05T08:32:54.151946+0000","last_unstale":"2023-11-05T08:34:02.300393+0000","last_undegraded":"2023-11-05T08:32:54.119646+0000","last_fullsized":"2023-11-05T08:32:54.118811+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":34,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.013731910999999999,"stat_sum":{"num_bytes":4816896,"num_objects":6,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":10,"num_read_kb":1684,"num_write":27,"num_write_kb":4144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2147483647,2,1,14,15,6],"acting":[5,2147483647,2,1,14,15,6],"avail_no_missing":["5(0)","1(3)","2(2)","6(6)","14(4)","15(5)"],"object_location_counts":[{"shards":"1(3),2(2),5(0),6(6),14(4),15(5)","objects":6}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.9","version":"103'64","reported_seq":315,"reported_epoch":108,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:34:01.843421+0000","last_change":"2023-11-05T08:33:50.691854+0000","last_active":"2023-11-05T08:34:01.843421+0000","last_peered":"2023-11-05T08:34:01.843421+0000","last_clean":"2023-11-05T08:32:49.072716+0000","last_became_active":"2023-11-05T08:33:50.691854+0000","last_became_peered":"2023-11-05T08:33:50.691854+0000","last_unstale":"2023-11-05T08:34:01.843421+0000","last_undegraded":"2023-11-05T08:33:50.658262+0000","last_fullsized":"2023-11-05T08:33:50.657391+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":34,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015323719999999999,"stat_sum":{"num_bytes":14761984,"num_objects":13,"num_object_clones":9,"num_object_copies":91,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":44,"num_read_kb":36208,"num_write":100,"num_write_kb":14416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":10584064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,2147483647,3,15,8,10],"acting":[14,1,2147483647,3,15,8,10],"avail_no_missing":["14(0)","1(1)","3(3)","8(5)","10(6)","15(4)"],"object_location_counts":[{"shards":"1(1),3(3),8(5),10(6),14(0),15(4)","objects":13}],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.a","version":"108'47","reported_seq":407,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-05T08:34:01.823848+0000","last_change":"2023-11-05T08:34:01.818604+0000","last_active":"2023-11-05T08:34:01.823848+0000","last_peered":"2023-11-05T08:34:01.823848+0000","last_clean":"2023-11-05T08:34:01.823848+0000","last_became_active":"2023-11-05T08:33:50.684400+0000","last_became_peered":"2023-11-05T08:33:50.684400+0000","last_unstale":"2023-11-05T08:34:01.823848+0000","last_undegraded":"2023-11-05T08:34:01.823848+0000","last_fullsized":"2023-11-05T08:34:01.823848+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'44","last_scrub_stamp":"2023-11-05T08:34:01.079975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:34:01.079975+0000","objects_scrubbed":3,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:35:14.986189+0000","scrub_duration":0.0091389039999999998,"objects_trimmed":0,"snaptrim_duration":0.0080629699999999992,"stat_sum":{"num_bytes":3538944,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":2,"num_read":25,"num_read_kb":17912,"num_write":36,"num_write_kb":3696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":6406144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,9,14,6,11],"acting":[3,7,1,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.b","version":"108'58","reported_seq":253,"reported_epoch":108,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:34:02.347090+0000","last_change":"2023-11-05T08:32:51.095575+0000","last_active":"2023-11-05T08:34:02.347090+0000","last_peered":"2023-11-05T08:34:02.347090+0000","last_clean":"2023-11-05T08:32:49.072970+0000","last_became_active":"2023-11-05T08:32:51.095575+0000","last_became_peered":"2023-11-05T08:32:51.095575+0000","last_unstale":"2023-11-05T08:34:02.347090+0000","last_undegraded":"2023-11-05T08:32:51.085902+0000","last_fullsized":"2023-11-05T08:32:51.084936+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":34,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:12.100399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021279290000000002,"stat_sum":{"num_bytes":12435456,"num_objects":13,"num_object_clones":9,"num_object_copies":91,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":38,"num_read_kb":17439,"num_write":64,"num_write_kb":12768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2147483647,1,12,6,7,9],"acting":[4,2147483647,1,12,6,7,9],"avail_no_missing":["4(0)","1(2)","6(4)","7(5)","9(6)","12(3)"],"object_location_counts":[{"shards":"1(2),4(0),6(4),7(5),9(6),12(3)","objects":13}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.c","version":"79'6","reported_seq":186,"reported_epoch":108,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:34:01.820645+0000","last_change":"2023-11-05T08:32:51.099852+0000","last_active":"2023-11-05T08:34:01.820645+0000","last_peered":"2023-11-05T08:34:01.820645+0000","last_clean":"2023-11-05T08:32:49.087645+0000","last_became_active":"2023-11-05T08:32:51.099852+0000","last_became_peered":"2023-11-05T08:32:51.099852+0000","last_unstale":"2023-11-05T08:34:01.820645+0000","last_undegraded":"2023-11-05T08:32:51.089699+0000","last_fullsized":"2023-11-05T08:32:51.089041+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":34,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015630430000000001,"stat_sum":{"num_bytes":999424,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":7,"num_read_kb":914,"num_write":4,"num_write_kb":976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,2147483647,10,6,12,5],"acting":[11,8,2147483647,10,6,12,5],"avail_no_missing":["11(0)","5(6)","6(4)","8(1)","10(3)","12(5)"],"object_location_counts":[{"shards":"5(6),6(4),8(1),10(3),11(0),12(5)","objects":3}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.d","version":"103'36","reported_seq":257,"reported_epoch":108,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:34:01.810896+0000","last_change":"2023-11-05T08:33:50.690639+0000","last_active":"2023-11-05T08:34:01.810896+0000","last_peered":"2023-11-05T08:34:01.810896+0000","last_clean":"2023-11-05T08:32:49.088145+0000","last_became_active":"2023-11-05T08:33:50.690639+0000","last_became_peered":"2023-11-05T08:33:50.690639+0000","last_unstale":"2023-11-05T08:34:01.810896+0000","last_undegraded":"2023-11-05T08:33:50.664683+0000","last_fullsized":"2023-11-05T08:33:50.663817+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":34,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.024583713,"stat_sum":{"num_bytes":5013504,"num_objects":7,"num_object_clones":5,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":6009,"num_write":23,"num_write_kb":3536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3112960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,2147483647,8,4],"acting":[7,3,14,10,2147483647,8,4],"avail_no_missing":["7(0)","3(1)","4(6)","8(5)","10(3)","14(2)"],"object_location_counts":[{"shards":"3(1),4(6),7(0),8(5),10(3),14(2)","objects":7}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.e","version":"108'128","reported_seq":451,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-05T08:34:01.821498+0000","last_change":"2023-11-05T08:34:01.821498+0000","last_active":"2023-11-05T08:34:01.821498+0000","last_peered":"2023-11-05T08:34:01.821498+0000","last_clean":"2023-11-05T08:34:01.821498+0000","last_became_active":"2023-11-05T08:32:44.949185+0000","last_became_peered":"2023-11-05T08:32:44.949185+0000","last_unstale":"2023-11-05T08:34:01.821498+0000","last_undegraded":"2023-11-05T08:34:01.821498+0000","last_fullsized":"2023-11-05T08:34:01.821498+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'96","last_scrub_stamp":"2023-11-05T08:33:48.102281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:48.102281+0000","objects_scrubbed":12,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:35:04.316205+0000","scrub_duration":0.014167007000000001,"objects_trimmed":0,"snaptrim_duration":0.011717086999999999,"stat_sum":{"num_bytes":7028736,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":12450,"num_write":75,"num_write_kb":13632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1,11,8,6,12],"acting":[2,9,1,11,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.f","version":"108'85","reported_seq":284,"reported_epoch":108,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:34:01.938307+0000","last_change":"2023-11-05T08:32:51.104425+0000","last_active":"2023-11-05T08:34:01.938307+0000","last_peered":"2023-11-05T08:34:01.938307+0000","last_clean":"2023-11-05T08:32:49.098585+0000","last_became_active":"2023-11-05T08:32:51.104425+0000","last_became_peered":"2023-11-05T08:32:51.104425+0000","last_unstale":"2023-11-05T08:34:01.938307+0000","last_undegraded":"2023-11-05T08:32:51.091474+0000","last_fullsized":"2023-11-05T08:32:51.089784+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":34,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012093372,"stat_sum":{"num_bytes":22069248,"num_objects":20,"num_object_clones":16,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":50,"num_read_kb":25938,"num_write":122,"num_write_kb":23184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,14,13,8,10,2147483647],"acting":[11,2,14,13,8,10,2147483647],"avail_no_missing":["11(0)","2(1)","8(4)","10(5)","13(3)","14(2)"],"object_location_counts":[{"shards":"2(1),8(4),10(5),11(0),13(3),14(2)","objects":20}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.10","version":"109'74","reported_seq":362,"reported_epoch":109,"state":"active+clean+remapped","last_fresh":"2023-11-05T08:34:02.845998+0000","last_change":"2023-11-05T08:34:02.857480+0000","last_active":"2023-11-05T08:34:02.845998+0000","last_peered":"2023-11-05T08:34:02.845998+0000","last_clean":"2023-11-05T08:34:02.845998+0000","last_became_active":"2023-11-05T08:32:59.210399+0000","last_became_peered":"2023-11-05T08:32:59.210399+0000","last_unstale":"2023-11-05T08:34:02.845998+0000","last_undegraded":"2023-11-05T08:34:02.845998+0000","last_fullsized":"2023-11-05T08:34:02.845998+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":5,"last_scrub":"33'9","last_scrub_stamp":"2023-11-05T08:33:00.792248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:00.792248+0000","objects_scrubbed":3,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:24.232263+0000","scrub_duration":0.0073349119999999999,"objects_trimmed":0,"snaptrim_duration":0.032180799000000003,"stat_sum":{"num_bytes":6897664,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":5385,"num_write":48,"num_write_kb":8880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,7,4,8,2147483647,11],"acting":[13,6,7,4,8,6,11],"avail_no_missing":["13(0)","4(3)","6(1)","6(5)","7(2)","8(4)","11(6)"],"object_location_counts":[{"shards":"4(3),6(1),6(5),7(2),8(4),11(6),13(0)","objects":4}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"44","length":"2"},{"start":"49","length":"1"}]},{"pgid":"3.11","version":"97'31","reported_seq":382,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-05T08:34:01.802710+0000","last_change":"2023-11-05T08:34:01.803250+0000","last_active":"2023-11-05T08:34:01.802710+0000","last_peered":"2023-11-05T08:34:01.802710+0000","last_clean":"2023-11-05T08:34:01.802710+0000","last_became_active":"2023-11-05T08:32:59.214193+0000","last_became_peered":"2023-11-05T08:32:59.214193+0000","last_unstale":"2023-11-05T08:34:01.802710+0000","last_undegraded":"2023-11-05T08:34:01.802710+0000","last_fullsized":"2023-11-05T08:34:01.802710+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":5,"last_scrub":"46'17","last_scrub_stamp":"2023-11-05T08:33:00.070510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:00.070510+0000","objects_scrubbed":3,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:03.268732+0000","scrub_duration":0.0081678800000000006,"objects_trimmed":0,"snaptrim_duration":0.00078513400000000005,"stat_sum":{"num_bytes":3981312,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":16565,"num_write":5,"num_write_kb":1936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,15,9,5,6,1],"acting":[10,11,15,9,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.12","version":"37'7","reported_seq":297,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-11-05T08:34:01.804941+0000","last_change":"2023-11-05T08:34:01.806121+0000","last_active":"2023-11-05T08:34:01.804941+0000","last_peered":"2023-11-05T08:34:01.804941+0000","last_clean":"2023-11-05T08:34:01.804941+0000","last_became_active":"2023-11-05T08:33:50.685746+0000","last_became_peered":"2023-11-05T08:33:50.685746+0000","last_unstale":"2023-11-05T08:34:01.804941+0000","last_undegraded":"2023-11-05T08:34:01.804941+0000","last_fullsized":"2023-11-05T08:34:01.804941+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"37'7","last_scrub_stamp":"2023-11-05T08:33:02.650387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:02.650387+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:19.568123+0000","scrub_duration":0.0067664470000000001,"objects_trimmed":0,"snaptrim_duration":0.002076825,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8,6,4,3,15],"acting":[9,7,8,6,4,3,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.13","version":"97'25","reported_seq":357,"reported_epoch":108,"state":"active+clean+remapped","last_fresh":"2023-11-05T08:34:01.821656+0000","last_change":"2023-11-05T08:34:01.822172+0000","last_active":"2023-11-05T08:34:01.821656+0000","last_peered":"2023-11-05T08:34:01.821656+0000","last_clean":"2023-11-05T08:34:01.821656+0000","last_became_active":"2023-11-05T08:32:59.212969+0000","last_became_peered":"2023-11-05T08:32:59.212969+0000","last_unstale":"2023-11-05T08:34:01.821656+0000","last_undegraded":"2023-11-05T08:34:01.821656+0000","last_fullsized":"2023-11-05T08:34:01.821656+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'14","last_scrub_stamp":"2023-11-05T08:33:02.644028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:02.644028+0000","objects_scrubbed":1,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:10.863657+0000","scrub_duration":0.007386386,"objects_trimmed":0,"snaptrim_duration":0.001657958,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,9,13,2147483647,4,10],"acting":[11,7,9,13,2,4,10],"avail_no_missing":["11(0)","2(4)","4(5)","7(1)","9(2)","10(6)","13(3)"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"49","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":115941376,"num_objects":109,"num_object_clones":61,"num_object_copies":763,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":71,"num_objects_misplaced":8,"num_objects_unfound":0,"num_objects_dirty":109,"num_whiteouts":4,"num_read":344,"num_read_kb":151968,"num_write":653,"num_write_kb":115800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":23068672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186142720,"data_stored":294432768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1002,"ondisk_log_size":1002,"up":182,"acting":182,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":15,"acting":15,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":13898,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":22,"seq":94489280534,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46516,"kb_used_data":15332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325324,"statfs":{"total":96636764160,"available":96589131776,"internally_reserved":0,"allocated":15699968,"data_stored":26085315,"data_compressed":174438,"data_compressed_allocated":393216,"data_compressed_original":1232896,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":1,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.256,"15min":1.256},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":3.4590000000000001,"5min":3.4590000000000001,"15min":3.4590000000000001},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.341,"5min":1.341,"15min":1.341},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":2.7280000000000002,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":1.3640000000000001}]},{"osd":2,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":2.2080000000000002,"5min":2.2080000000000002,"15min":2.2080000000000002},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.218,"5min":1.218,"15min":1.218},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":3.0139999999999998,"5min":3.0139999999999998,"15min":3.0139999999999998},"last":1.2170000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2410000000000001}]},{"osd":4,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.222,"15min":1.222},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":3.5979999999999999,"5min":3.5979999999999999,"15min":3.5979999999999999},"last":0.997},{"interface":"front","average":{"1min":1.321,"5min":1.321,"15min":1.321},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":3.383,"5min":3.383,"15min":3.383},"last":1.7470000000000001}]},{"osd":5,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.466,"15min":1.466},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":3.5059999999999998,"5min":3.5059999999999998,"15min":3.5059999999999998},"last":1.4219999999999999},{"interface":"front","average":{"1min":1.5029999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":3.1579999999999999,"5min":3.1579999999999999,"15min":3.1579999999999999},"last":1.054}]},{"osd":6,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.625,"5min":1.625,"15min":1.625},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":3.4820000000000002,"5min":3.4820000000000002,"15min":3.4820000000000002},"last":1.621},{"interface":"front","average":{"1min":1.579,"5min":1.579,"15min":1.579},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":3.0059999999999998,"5min":3.0059999999999998,"15min":3.0059999999999998},"last":1.294}]},{"osd":7,"last update":"Sun Nov 5 08:33:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"min":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"max":{"1min":2.165,"5min":2.165,"15min":2.165},"last":2.0070000000000001},{"interface":"front","average":{"1min":1.54,"5min":1.54,"15min":1.54},"min":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"max":{"1min":2.2759999999999998,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":1.889}]},{"osd":8,"last update":"Sun Nov 5 08:33:52 2023","interfaces":[{"interface":"back","average":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.6850000000000001},"min":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":3.5739999999999998,"5min":3.5739999999999998,"15min":3.5739999999999998},"last":1.8340000000000001},{"interface":"front","average":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"min":{"1min":1.044,"5min":1.044,"15min":1.044},"max":{"1min":2.9569999999999999,"5min":2.9569999999999999,"15min":2.9569999999999999},"last":1.7090000000000001}]},{"osd":9,"last update":"Sun Nov 5 08:33:52 2023","interfaces":[{"interface":"back","average":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":1.7849999999999999},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":4.6520000000000001,"5min":4.6520000000000001,"15min":4.6520000000000001},"last":1.8009999999999999},{"interface":"front","average":{"1min":2.0720000000000001,"5min":2.0720000000000001,"15min":2.0720000000000001},"min":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"max":{"1min":4.7450000000000001,"5min":4.7450000000000001,"15min":4.7450000000000001},"last":1.9299999999999999}]},{"osd":11,"last update":"Sun Nov 5 08:33:48 2023","interfaces":[{"interface":"back","average":{"1min":1.6619999999999999,"5min":1.6619999999999999,"15min":1.6619999999999999},"min":{"1min":1.2,"5min":1.2,"15min":1.2},"max":{"1min":2.5129999999999999,"5min":2.5129999999999999,"15min":2.5129999999999999},"last":2.1259999999999999},{"interface":"front","average":{"1min":1.7909999999999999,"5min":1.7909999999999999,"15min":1.7909999999999999},"min":{"1min":1.056,"5min":1.056,"15min":1.056},"max":{"1min":3.6619999999999999,"5min":3.6619999999999999,"15min":3.6619999999999999},"last":2.1640000000000001}]},{"osd":12,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.962,"5min":1.962,"15min":1.962},"min":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"max":{"1min":5.7270000000000003,"5min":5.7270000000000003,"15min":5.7270000000000003},"last":2.0950000000000002},{"interface":"front","average":{"1min":1.9239999999999999,"5min":1.9239999999999999,"15min":1.9239999999999999},"min":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"max":{"1min":6.093,"5min":6.093,"15min":6.093},"last":2.2410000000000001}]},{"osd":14,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.9470000000000001,"5min":1.9470000000000001,"15min":1.9470000000000001},"min":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"max":{"1min":3.7050000000000001,"5min":3.7050000000000001,"15min":3.7050000000000001},"last":2.0529999999999999},{"interface":"front","average":{"1min":1.982,"5min":1.982,"15min":1.982},"min":{"1min":1.099,"5min":1.099,"15min":1.099},"max":{"1min":3.2349999999999999,"5min":3.2349999999999999,"15min":3.2349999999999999},"last":2.2109999999999999}]},{"osd":15,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.9370000000000001,"5min":1.9370000000000001,"15min":1.9370000000000001},"min":{"1min":1.155,"5min":1.155,"15min":1.155},"max":{"1min":3.048,"5min":3.048,"15min":3.048},"last":2.3050000000000002},{"interface":"front","average":{"1min":2.129,"5min":2.129,"15min":2.129},"min":{"1min":1.135,"5min":1.135,"15min":1.135},"max":{"1min":5.3920000000000003,"5min":5.3920000000000003,"15min":5.3920000000000003},"last":1.964}]}]},{"osd":15,"up_from":21,"seq":90194313238,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37980,"kb_used_data":10956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94333860,"statfs":{"total":96636764160,"available":96597872640,"internally_reserved":0,"allocated":11218944,"data_stored":16767293,"data_compressed":165545,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,4,5,6,7,8,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":1,"last update":"Sun Nov 5 08:33:55 2023","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.139,"15min":1.139},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":3.0329999999999999,"5min":3.0329999999999999,"15min":3.0329999999999999},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":8.6170000000000009,"5min":8.6170000000000009,"15min":8.6170000000000009},"last":0.435}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.131}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96399999999999997}]},{"osd":4,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.125,"15min":1.125},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":2.4649999999999999,"5min":2.4649999999999999,"15min":2.4649999999999999},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":2.331,"5min":2.331,"15min":2.331},"last":0.67200000000000004}]},{"osd":5,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.113,"15min":1.113},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":2.9460000000000002,"5min":2.9460000000000002,"15min":2.9460000000000002},"last":0.48399999999999999},{"interface":"front","average":{"1min":1.452,"5min":1.452,"15min":1.452},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":3.8849999999999998,"5min":3.8849999999999998,"15min":3.8849999999999998},"last":1.042}]},{"osd":6,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.119,"15min":1.119},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":2.4769999999999999,"5min":2.4769999999999999,"15min":2.4769999999999999},"last":0.84199999999999997},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":3.0880000000000001,"5min":3.0880000000000001,"15min":3.0880000000000001},"last":0.88800000000000001}]},{"osd":7,"last update":"Sun Nov 5 08:33:44 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.169,"15min":1.169},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.996,"5min":2.996,"15min":2.996},"last":1.0600000000000001},{"interface":"front","average":{"1min":1.389,"5min":1.389,"15min":1.389},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":5.444,"5min":5.444,"15min":5.444},"last":0.54300000000000004}]},{"osd":8,"last update":"Sun Nov 5 08:33:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":3.7389999999999999,"5min":3.7389999999999999,"15min":3.7389999999999999},"last":1.1970000000000001},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":4.2489999999999997,"5min":4.2489999999999997,"15min":4.2489999999999997},"last":0.97999999999999998}]},{"osd":9,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.5900000000000001,"5min":1.5900000000000001,"15min":1.5900000000000001},"min":{"1min":1.044,"5min":1.044,"15min":1.044},"max":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"last":1.2330000000000001},{"interface":"front","average":{"1min":1.883,"5min":1.883,"15min":1.883},"min":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"max":{"1min":13.029999999999999,"5min":13.029999999999999,"15min":13.029999999999999},"last":0.77600000000000002}]},{"osd":11,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"min":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":2.9590000000000001,"5min":2.9590000000000001,"15min":2.9590000000000001},"last":1.1599999999999999},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"min":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"max":{"1min":2.6469999999999998,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":0.72699999999999998}]},{"osd":13,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.591,"5min":1.591,"15min":1.591},"min":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"max":{"1min":3.3580000000000001,"5min":3.3580000000000001,"15min":3.3580000000000001},"last":1.2709999999999999},{"interface":"front","average":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":1.7849999999999999},"min":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"max":{"1min":5.1379999999999999,"5min":5.1379999999999999,"15min":5.1379999999999999},"last":1.018}]},{"osd":14,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.718,"5min":1.718,"15min":1.718},"min":{"1min":0.999,"5min":0.999,"15min":0.999},"max":{"1min":5.7720000000000002,"5min":5.7720000000000002,"15min":5.7720000000000002},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.696,"5min":1.696,"15min":1.696},"min":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"max":{"1min":7.5039999999999996,"5min":7.5039999999999996,"15min":7.5039999999999996},"last":0.94299999999999995}]}]},{"osd":14,"up_from":22,"seq":94489280533,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40056,"kb_used_data":13036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331784,"statfs":{"total":96636764160,"available":96595746816,"internally_reserved":0,"allocated":13348864,"data_stored":25578375,"data_compressed":165545,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":1,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":1.45},{"interface":"front","average":{"1min":1.131,"5min":1.131,"15min":1.131},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.732,"5min":1.732,"15min":1.732},"last":0.78500000000000003}]},{"osd":2,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.86099999999999999},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.839,"5min":1.839,"15min":1.839},"last":1.179}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3149999999999999}]},{"osd":4,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":1.0640000000000001},{"interface":"front","average":{"1min":0.997,"5min":0.997,"15min":0.997},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.391,"5min":1.391,"15min":1.391},"last":1.347}]},{"osd":5,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.101,"5min":1.101,"15min":1.101},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.605,"5min":1.605,"15min":1.605},"last":0.92400000000000004}]},{"osd":6,"last update":"Sun Nov 5 08:33:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.575,"5min":1.575,"15min":1.575},"last":0.59799999999999998},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.9610000000000001,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.97299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.133}]},{"osd":8,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.7050000000000001,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.22,"5min":1.22,"15min":1.22},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":1.242}]},{"osd":10,"last update":"Sun Nov 5 08:33:44 2023","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.29,"15min":1.29},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.802,"5min":1.802,"15min":1.802},"last":1.028},{"interface":"front","average":{"1min":1.325,"5min":1.325,"15min":1.325},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.8049999999999999,"5min":1.8049999999999999,"15min":1.8049999999999999},"last":1.528}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.429}]},{"osd":12,"last update":"Sun Nov 5 08:33:53 2023","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":1.4430000000000001},"min":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"max":{"1min":2.2389999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.153},{"interface":"front","average":{"1min":1.544,"5min":1.544,"15min":1.544},"min":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"max":{"1min":2.1320000000000001,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":1.478}]},{"osd":13,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.5900000000000001,"5min":1.5900000000000001,"15min":1.5900000000000001},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":2.4590000000000001,"5min":2.4590000000000001,"15min":2.4590000000000001},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"min":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":2.028,"5min":2.028,"15min":2.028},"last":1.214}]},{"osd":15,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":2.2429999999999999,"5min":2.2429999999999999,"15min":2.2429999999999999},"last":1.4139999999999999},{"interface":"front","average":{"1min":1.6399999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":2.3559999999999999,"5min":2.3559999999999999,"15min":2.3559999999999999},"last":1.548}]}]},{"osd":0,"up_from":21,"seq":90194313223,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":4392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":4497408,"data_stored":4339168,"data_compressed":21230,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,9,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]}]},{"osd":3,"up_from":93,"seq":399431958532,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38104,"kb_used_data":9924,"kb_used_omap":6,"kb_used_meta":28153,"kb_avail":94333736,"statfs":{"total":96636764160,"available":96597745664,"internally_reserved":0,"allocated":10162176,"data_stored":12466071,"data_compressed":161651,"data_compressed_allocated":348160,"data_compressed_original":696320,"omap_allocated":6448,"internal_metadata":28829392},"hb_peers":[2,4,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2589999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91100000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2250000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4350000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4790000000000001}]}]},{"osd":6,"up_from":21,"seq":90194313237,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47140,"kb_used_data":15960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324700,"statfs":{"total":96636764160,"available":96588492800,"internally_reserved":0,"allocated":16343040,"data_stored":21761825,"data_compressed":163600,"data_compressed_allocated":352256,"data_compressed_original":704512,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,3,4,5,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.02,"15min":1.02},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":0.84999999999999998},{"interface":"front","average":{"1min":1.153,"5min":1.153,"15min":1.153},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.899,"5min":1.899,"15min":1.899},"last":1.0569999999999999}]},{"osd":2,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.76000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4370000000000001}]},{"osd":4,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.038,"15min":1.038},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":0.874},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.617,"5min":1.617,"15min":1.617},"last":0.64200000000000002}]},{"osd":5,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.532,"5min":1.532,"15min":1.532},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.212,"5min":1.212,"15min":1.212},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.577,"5min":1.577,"15min":1.577},"last":0.92100000000000004}]},{"osd":7,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.72,"5min":1.72,"15min":1.72},"last":0.95399999999999996},{"interface":"front","average":{"1min":1.069,"5min":1.069,"15min":1.069},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.6859999999999999,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":1.292}]},{"osd":8,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.958,"5min":1.958,"15min":1.958},"last":0.59999999999999998},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.772,"5min":1.772,"15min":1.772},"last":1.724}]},{"osd":9,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.484},{"interface":"front","average":{"1min":1.306,"5min":1.306,"15min":1.306},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.7749999999999999,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":1.26}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3999999999999999}]},{"osd":11,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.379,"15min":1.379},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":2.0579999999999998,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":1.093},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":2.1930000000000001,"5min":2.1930000000000001,"15min":2.1930000000000001},"last":1.5249999999999999}]},{"osd":12,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.548,"5min":1.548,"15min":1.548},"min":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":2.1139999999999999,"5min":2.1139999999999999,"15min":2.1139999999999999},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.841,"5min":1.841,"15min":1.841},"last":1.1379999999999999}]},{"osd":13,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":1.6930000000000001,"5min":1.6930000000000001,"15min":1.6930000000000001},"min":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"max":{"1min":2.2759999999999998,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":1.589},{"interface":"front","average":{"1min":1.532,"5min":1.532,"15min":1.532},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.9710000000000001,"5min":1.9710000000000001,"15min":1.9710000000000001},"last":1.5660000000000001}]},{"osd":14,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":2.319,"5min":2.319,"15min":2.319},"last":1.327},{"interface":"front","average":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":2.4870000000000001,"5min":2.4870000000000001,"15min":2.4870000000000001},"last":1.6559999999999999}]},{"osd":15,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.679,"5min":1.679,"15min":1.679},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.4009999999999998,"5min":2.4009999999999998,"15min":2.4009999999999998},"last":1.1970000000000001},{"interface":"front","average":{"1min":1.482,"5min":1.482,"15min":1.482},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"last":1.623}]}]},{"osd":1,"up_from":22,"seq":94489280533,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46080,"kb_used_data":14900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325760,"statfs":{"total":96636764160,"available":96589578240,"internally_reserved":0,"allocated":15257600,"data_stored":21203847,"data_compressed":165545,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,5,6,7,8,9,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":2,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.272,"5min":1.272,"15min":1.272},"last":0.84899999999999998},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.1240000000000001,"5min":2.1240000000000001,"15min":2.1240000000000001},"last":0.77100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.873}]},{"osd":4,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.6100000000000001,"5min":1.6100000000000001,"15min":1.6100000000000001},"last":1.117}]},{"osd":5,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.071,"15min":1.071},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":2.6110000000000002,"5min":2.6110000000000002,"15min":2.6110000000000002},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.7310000000000001,"5min":1.7310000000000001,"15min":1.7310000000000001},"last":1.349}]},{"osd":6,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.135,"15min":1.135},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.61599999999999999},{"interface":"front","average":{"1min":1.246,"5min":1.246,"15min":1.246},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.1759999999999999}]},{"osd":7,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.282,"15min":1.282},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.708,"5min":1.708,"15min":1.708},"last":0.82599999999999996},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":2.3090000000000002,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":1.254}]},{"osd":8,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.288,"15min":1.288},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":2.5630000000000002,"5min":2.5630000000000002,"15min":2.5630000000000002},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.238,"5min":1.238,"15min":1.238},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":1.397}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5089999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.46}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3109999999999999}]},{"osd":14,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"min":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"max":{"1min":2.2320000000000002,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":1.4810000000000001},{"interface":"front","average":{"1min":1.498,"5min":1.498,"15min":1.498},"min":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"max":{"1min":2.7610000000000001,"5min":2.7610000000000001,"15min":2.7610000000000001},"last":1.038}]},{"osd":15,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.5640000000000001,"5min":1.5640000000000001,"15min":1.5640000000000001},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":2.7250000000000001,"5min":2.7250000000000001,"15min":2.7250000000000001},"last":1.55},{"interface":"front","average":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"min":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":1.655}]}]},{"osd":2,"up_from":22,"seq":94489280533,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42868,"kb_used_data":11688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328972,"statfs":{"total":96636764160,"available":96592867328,"internally_reserved":0,"allocated":11968512,"data_stored":20028839,"data_compressed":172494,"data_compressed_allocated":389120,"data_compressed_original":1224704,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,3,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.5529999999999999,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.482,"5min":1.482,"15min":1.482},"last":0.94699999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94199999999999995}]},{"osd":5,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.5029999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.79800000000000004}]},{"osd":6,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.5},{"interface":"front","average":{"1min":1.077,"5min":1.077,"15min":1.077},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":0.32700000000000001}]},{"osd":7,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.167,"15min":1.167},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":2.0880000000000001,"5min":2.0880000000000001,"15min":2.0880000000000001},"last":0.629},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.744,"5min":1.744,"15min":1.744},"last":0.81499999999999995}]},{"osd":8,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.831,"5min":1.831,"15min":1.831},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.129,"5min":1.129,"15min":1.129},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":2.1859999999999999,"5min":2.1859999999999999,"15min":2.1859999999999999},"last":0.40500000000000003}]},{"osd":9,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.7969999999999999,"5min":1.7969999999999999,"15min":1.7969999999999999},"last":0.84499999999999997},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":2.27,"5min":2.27,"15min":2.27},"last":0.70499999999999996}]},{"osd":10,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":2.976,"5min":2.976,"15min":2.976},"last":0.44800000000000001},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":2.5739999999999998,"5min":2.5739999999999998,"15min":2.5739999999999998},"last":0.91000000000000003}]},{"osd":11,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.9009999999999998,"5min":2.9009999999999998,"15min":2.9009999999999998},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":2.6859999999999999,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.73499999999999999}]},{"osd":12,"last update":"Sun Nov 5 08:33:45 2023","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.2,"15min":1.2},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.75600000000000001}]}]},{"osd":4,"up_from":22,"seq":94489280533,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44584,"kb_used_data":13400,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327256,"statfs":{"total":96636764160,"available":96591110144,"internally_reserved":0,"allocated":13721600,"data_stored":19343677,"data_compressed":165545,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,5,6,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":1,"last update":"Sun Nov 5 08:33:43 2023","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.6659999999999999,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":0.995}]},{"osd":2,"last update":"Sun Nov 5 08:33:43 2023","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":0.82399999999999995},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.9850000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":1.4790000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2849999999999999}]},{"osd":5,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.0299999999999998,"5min":2.0299999999999998,"15min":2.0299999999999998},"last":1.24},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.7,"5min":1.7,"15min":1.7},"last":1.4390000000000001}]},{"osd":6,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":0.995,"15min":0.995},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.7290000000000001,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":1.238,"5min":1.238,"15min":1.238},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":2.2789999999999999,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":1.101}]},{"osd":7,"last update":"Sun Nov 5 08:33:43 2023","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.889,"5min":1.889,"15min":1.889},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":1.514}]},{"osd":8,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.171,"15min":1.171},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":2.2490000000000001,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.7789999999999999,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":1.1839999999999999}]},{"osd":9,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":2.5249999999999999,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":1.359},{"interface":"front","average":{"1min":1.468,"5min":1.468,"15min":1.468},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.3290000000000002,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":1.647}]},{"osd":10,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":2.0459999999999998,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.401},{"interface":"front","average":{"1min":1.508,"5min":1.508,"15min":1.508},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.3919999999999999,"5min":2.3919999999999999,"15min":2.3919999999999999},"last":1.569}]},{"osd":11,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.431,"15min":1.431},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.1280000000000001,"5min":2.1280000000000001,"15min":2.1280000000000001},"last":1.3260000000000001},{"interface":"front","average":{"1min":1.3,"5min":1.3,"15min":1.3},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.9379999999999999,"5min":1.9379999999999999,"15min":1.9379999999999999},"last":2.0960000000000001}]},{"osd":12,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.3,"5min":1.3,"15min":1.3},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":2.161,"5min":2.161,"15min":2.161},"last":1.9219999999999999}]},{"osd":13,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.5640000000000001,"5min":1.5640000000000001,"15min":1.5640000000000001},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":2.3420000000000001,"5min":2.3420000000000001,"15min":2.3420000000000001},"last":2.036},{"interface":"front","average":{"1min":1.538,"5min":1.538,"15min":1.538},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":2.3140000000000001,"5min":2.3140000000000001,"15min":2.3140000000000001},"last":1.6890000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5109999999999999}]},{"osd":15,"last update":"Sun Nov 5 08:33:57 2023","interfaces":[{"interface":"back","average":{"1min":1.6990000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":2.5750000000000002,"5min":2.5750000000000002,"15min":2.5750000000000002},"last":1.6080000000000001},{"interface":"front","average":{"1min":1.6719999999999999,"5min":1.6719999999999999,"15min":1.6719999999999999},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":2.3940000000000001,"5min":2.3940000000000001,"15min":2.3940000000000001},"last":1.879}]}]},{"osd":5,"up_from":22,"seq":94489280533,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34200,"kb_used_data":7180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337640,"statfs":{"total":96636764160,"available":96601743360,"internally_reserved":0,"allocated":7352320,"data_stored":8196459,"data_compressed":163600,"data_compressed_allocated":352256,"data_compressed_original":704512,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,6,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":1,"last update":"Sun Nov 5 08:33:45 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.97599999999999998}]},{"osd":2,"last update":"Sun Nov 5 08:33:45 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.575,"5min":1.575,"15min":1.575},"last":1.3939999999999999},{"interface":"front","average":{"1min":1.04,"5min":1.04,"15min":1.04},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.546,"5min":1.546,"15min":1.546},"last":1.26}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":4,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.645,"5min":1.645,"15min":1.645},"last":1.831},{"interface":"front","average":{"1min":1.036,"5min":1.036,"15min":1.036},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.77,"5min":1.77,"15min":1.77},"last":1.47}]},{"osd":6,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":1.3620000000000001},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":1.512}]},{"osd":7,"last update":"Sun Nov 5 08:33:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.696,"5min":1.696,"15min":1.696},"last":1.3089999999999999},{"interface":"front","average":{"1min":1.177,"5min":1.177,"15min":1.177},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":1.212}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5760000000000001}]},{"osd":9,"last update":"Sun Nov 5 08:33:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.3,"15min":1.3},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.6990000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":1.4359999999999999},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"min":{"1min":0.88,"5min":0.88,"15min":0.88},"max":{"1min":1.8180000000000001,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":1.931}]},{"osd":10,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.7909999999999999},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":1.8799999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4990000000000001}]},{"osd":12,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":1.635,"15min":1.635},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":3.488,"5min":3.488,"15min":3.488},"last":2.0089999999999999},{"interface":"front","average":{"1min":1.46,"5min":1.46,"15min":1.46},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.1970000000000001,"5min":2.1970000000000001,"15min":2.1970000000000001},"last":1.9690000000000001}]},{"osd":13,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.6799999999999999,"5min":1.6799999999999999,"15min":1.6799999999999999},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":3.714,"5min":3.714,"15min":3.714},"last":2.1819999999999999},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.5649999999999999,"15min":1.5649999999999999},"min":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"max":{"1min":2.121,"5min":2.121,"15min":2.121},"last":2.0369999999999999}]},{"osd":14,"last update":"Sun Nov 5 08:33:45 2023","interfaces":[{"interface":"back","average":{"1min":1.718,"5min":1.718,"15min":1.718},"min":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"max":{"1min":2.1920000000000002,"5min":2.1920000000000002,"15min":2.1920000000000002},"last":2.1309999999999998},{"interface":"front","average":{"1min":1.647,"5min":1.647,"15min":1.647},"min":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"max":{"1min":2.2930000000000001,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":2.1059999999999999}]},{"osd":15,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.974,"5min":1.974,"15min":1.974},"last":1.637},{"interface":"front","average":{"1min":1.8140000000000001,"5min":1.8140000000000001,"15min":1.8140000000000001},"min":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"max":{"1min":3.5870000000000002,"5min":3.5870000000000002,"15min":3.5870000000000002},"last":2.0600000000000001}]}]},{"osd":7,"up_from":22,"seq":94489280534,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39656,"kb_used_data":12632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94332184,"statfs":{"total":96636764160,"available":96596156416,"internally_reserved":0,"allocated":12935168,"data_stored":17976199,"data_compressed":165545,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1930000000000001}]},{"osd":1,"last update":"Sun Nov 5 08:33:47 2023","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":2.1360000000000001,"5min":2.1360000000000001,"15min":2.1360000000000001},"last":1.258},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.85299999999999998}]},{"osd":2,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.401,"15min":1.401},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":1.889,"5min":1.889,"15min":1.889},"last":1.875},{"interface":"front","average":{"1min":1.238,"5min":1.238,"15min":1.238},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.663,"5min":1.663,"15min":1.663},"last":1.349}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4370000000000001}]},{"osd":4,"last update":"Sun Nov 5 08:33:30 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.095,"15min":1.095},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.7949999999999999,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":2.294,"5min":2.294,"15min":2.294},"last":1.0840000000000001}]},{"osd":5,"last update":"Sun Nov 5 08:33:30 2023","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.591,"5min":1.591,"15min":1.591},"last":1.5740000000000001},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.726,"5min":1.726,"15min":1.726},"last":1.302}]},{"osd":6,"last update":"Sun Nov 5 08:33:30 2023","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.232,"15min":1.232},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.6040000000000001,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":1.7689999999999999},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":2.0510000000000002,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.7170000000000001}]},{"osd":8,"last update":"Sun Nov 5 08:33:30 2023","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.381,"5min":1.381,"15min":1.381},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.48,"5min":2.48,"15min":2.48},"last":1.657}]},{"osd":9,"last update":"Sun Nov 5 08:33:47 2023","interfaces":[{"interface":"back","average":{"1min":1.6799999999999999,"5min":1.6799999999999999,"15min":1.6799999999999999},"min":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"max":{"1min":2.823,"5min":2.823,"15min":2.823},"last":2.0099999999999998},{"interface":"front","average":{"1min":1.48,"5min":1.48,"15min":1.48},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":2.3450000000000002,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":1.9350000000000001}]},{"osd":10,"last update":"Sun Nov 5 08:33:30 2023","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":1.478,"15min":1.478},"min":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"max":{"1min":1.931,"5min":1.931,"15min":1.931},"last":1.397},{"interface":"front","average":{"1min":1.671,"5min":1.671,"15min":1.671},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.5550000000000002,"5min":2.5550000000000002,"15min":2.5550000000000002},"last":2.1539999999999999}]},{"osd":11,"last update":"Sun Nov 5 08:33:47 2023","interfaces":[{"interface":"back","average":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":2.093,"5min":2.093,"15min":2.093},"last":1.6220000000000001},{"interface":"front","average":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":2.6600000000000001,"5min":2.6600000000000001,"15min":2.6600000000000001},"last":2.2069999999999999}]},{"osd":12,"last update":"Sun Nov 5 08:33:52 2023","interfaces":[{"interface":"back","average":{"1min":1.841,"5min":1.841,"15min":1.841},"min":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"max":{"1min":2.7250000000000001,"5min":2.7250000000000001,"15min":2.7250000000000001},"last":2.306},{"interface":"front","average":{"1min":1.639,"5min":1.639,"15min":1.639},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.3780000000000001,"5min":2.3780000000000001,"15min":2.3780000000000001},"last":2.1880000000000002}]},{"osd":13,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":1.667,"5min":1.667,"15min":1.667},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.589,"5min":2.589,"15min":2.589},"last":2.0430000000000001},{"interface":"front","average":{"1min":1.625,"5min":1.625,"15min":1.625},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":2.3050000000000002,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":1.9690000000000001}]},{"osd":14,"last update":"Sun Nov 5 08:33:30 2023","interfaces":[{"interface":"back","average":{"1min":1.75,"5min":1.75,"15min":1.75},"min":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":2.4910000000000001,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":2.1200000000000001},{"interface":"front","average":{"1min":1.7529999999999999,"5min":1.7529999999999999,"15min":1.7529999999999999},"min":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"max":{"1min":2.9750000000000001,"5min":2.9750000000000001,"15min":2.9750000000000001},"last":1.823}]},{"osd":15,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":1.7010000000000001,"5min":1.7010000000000001,"15min":1.7010000000000001},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":3.1760000000000002,"5min":3.1760000000000002,"15min":3.1760000000000002},"last":2.0859999999999999},{"interface":"front","average":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":2.73,"5min":2.73,"15min":2.73},"last":2.258}]}]},{"osd":8,"up_from":22,"seq":94489280533,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50488,"kb_used_data":19304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321352,"statfs":{"total":96636764160,"available":96585064448,"internally_reserved":0,"allocated":19767296,"data_stored":34982791,"data_compressed":165545,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":1,"last update":"Sun Nov 5 08:33:46 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":1.3140000000000001},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":1.22}]},{"osd":2,"last update":"Sun Nov 5 08:33:46 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":1.1559999999999999},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":1.4950000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4330000000000001}]},{"osd":4,"last update":"Sun Nov 5 08:33:46 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":1.585},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":1.524}]},{"osd":5,"last update":"Sun Nov 5 08:33:33 2023","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.786,"5min":1.786,"15min":1.786},"last":1.6950000000000001},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":1.8280000000000001}]},{"osd":6,"last update":"Sun Nov 5 08:33:33 2023","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.6799999999999999,"5min":1.6799999999999999,"15min":1.6799999999999999},"last":1.7889999999999999},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":1.875}]},{"osd":7,"last update":"Sun Nov 5 08:33:33 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.556,"5min":1.556,"15min":1.556},"last":1.7430000000000001},{"interface":"front","average":{"1min":1.214,"5min":1.214,"15min":1.214},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":2.157,"5min":2.157,"15min":2.157},"last":1.9510000000000001}]},{"osd":9,"last update":"Sun Nov 5 08:33:33 2023","interfaces":[{"interface":"back","average":{"1min":2.1509999999999998,"5min":2.1509999999999998,"15min":2.1509999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":19.849,"5min":19.849,"15min":19.849},"last":2.234},{"interface":"front","average":{"1min":2.2170000000000001,"5min":2.2170000000000001,"15min":2.2170000000000001},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":19.867999999999999,"5min":19.867999999999999,"15min":19.867999999999999},"last":1.917}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]},{"osd":11,"last update":"Sun Nov 5 08:33:33 2023","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.298,"15min":1.298},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":2.0710000000000002,"5min":2.0710000000000002,"15min":2.0710000000000002},"last":2.21},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":2.0960000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":13,"last update":"Sun Nov 5 08:33:33 2023","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":2.0670000000000002},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.8560000000000001,"5min":1.8560000000000001,"15min":1.8560000000000001},"last":2.0310000000000001}]},{"osd":14,"last update":"Sun Nov 5 08:33:33 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":2.226,"5min":2.226,"15min":2.226},"last":1.9930000000000001},{"interface":"front","average":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":1.4430000000000001},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.891,"5min":1.891,"15min":1.891},"last":2.266}]},{"osd":15,"last update":"Sun Nov 5 08:33:55 2023","interfaces":[{"interface":"back","average":{"1min":1.552,"5min":1.552,"15min":1.552},"min":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"max":{"1min":2.1629999999999998,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":2.1360000000000001},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.4490000000000001,"15min":1.4490000000000001},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.1970000000000001,"5min":2.1970000000000001,"15min":2.1970000000000001},"last":2.169}]}]},{"osd":9,"up_from":21,"seq":90194313237,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42728,"kb_used_data":11544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329112,"statfs":{"total":96636764160,"available":96593010688,"internally_reserved":0,"allocated":11821056,"data_stored":14945159,"data_compressed":165545,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,6,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":1,"last update":"Sun Nov 5 08:34:00 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.506,"5min":1.506,"15min":1.506},"last":0.75},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.734,"5min":1.734,"15min":1.734},"last":1.0640000000000001}]},{"osd":2,"last update":"Sun Nov 5 08:34:00 2023","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.81200000000000006},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.5620000000000001,"5min":1.5620000000000001,"15min":1.5620000000000001},"last":0.94499999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.23}]},{"osd":4,"last update":"Sun Nov 5 08:34:00 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.095,"15min":1.095},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.478,"5min":1.478,"15min":1.478},"last":0.86499999999999999},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.595,"5min":1.595,"15min":1.595},"last":1.52}]},{"osd":6,"last update":"Sun Nov 5 08:33:32 2023","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.101,"15min":1.101},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":1.641,"5min":1.641,"15min":1.641},"last":1.1399999999999999},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.486,"5min":1.486,"15min":1.486},"last":1.3740000000000001}]},{"osd":7,"last update":"Sun Nov 5 08:34:00 2023","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"min":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"max":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.482},{"interface":"front","average":{"1min":1.369,"5min":1.369,"15min":1.369},"min":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":1.925,"5min":1.925,"15min":1.925},"last":1.6599999999999999}]},{"osd":8,"last update":"Sun Nov 5 08:33:32 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.202,"15min":1.202},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":1.4179999999999999},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.944,"5min":1.944,"15min":1.944},"last":1.617}]},{"osd":10,"last update":"Sun Nov 5 08:33:32 2023","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.752,"5min":1.752,"15min":1.752},"last":1.1870000000000001},{"interface":"front","average":{"1min":1.319,"5min":1.319,"15min":1.319},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":2.1240000000000001,"5min":2.1240000000000001,"15min":2.1240000000000001},"last":1.002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2669999999999999}]},{"osd":12,"last update":"Sun Nov 5 08:33:32 2023","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.381,"15min":1.381},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":2.0710000000000002,"5min":2.0710000000000002,"15min":2.0710000000000002},"last":1.2709999999999999},{"interface":"front","average":{"1min":1.462,"5min":1.462,"15min":1.462},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":2.0350000000000001,"5min":2.0350000000000001,"15min":2.0350000000000001},"last":1.4550000000000001}]},{"osd":13,"last update":"Sun Nov 5 08:33:32 2023","interfaces":[{"interface":"back","average":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"min":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"max":{"1min":2.8010000000000002,"5min":2.8010000000000002,"15min":2.8010000000000002},"last":1.748},{"interface":"front","average":{"1min":1.478,"5min":1.478,"15min":1.478},"min":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"max":{"1min":2.177,"5min":2.177,"15min":2.177},"last":1.7150000000000001}]},{"osd":14,"last update":"Sun Nov 5 08:33:32 2023","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.506,"15min":1.506},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.849,"5min":1.849,"15min":1.849},"last":1.772},{"interface":"front","average":{"1min":1.6699999999999999,"5min":1.6699999999999999,"15min":1.6699999999999999},"min":{"1min":1.294,"5min":1.294,"15min":1.294},"max":{"1min":2.9710000000000001,"5min":2.9710000000000001,"15min":2.9710000000000001},"last":1.591}]},{"osd":15,"last update":"Sun Nov 5 08:33:32 2023","interfaces":[{"interface":"back","average":{"1min":1.603,"5min":1.603,"15min":1.603},"min":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"max":{"1min":1.9670000000000001,"5min":1.9670000000000001,"15min":1.9670000000000001},"last":1.821},{"interface":"front","average":{"1min":1.716,"5min":1.716,"15min":1.716},"min":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"max":{"1min":3.0950000000000002,"5min":3.0950000000000002,"15min":3.0950000000000002},"last":1.6919999999999999}]}]},{"osd":10,"up_from":22,"seq":94489280533,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41780,"kb_used_data":14760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330060,"statfs":{"total":96636764160,"available":96593981440,"internally_reserved":0,"allocated":15114240,"data_stored":26340231,"data_compressed":165545,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,4,5,6,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":1,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.86399999999999999}]},{"osd":2,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.768,"5min":1.768,"15min":1.768},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.01,"5min":1.01,"15min":1.01},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":2.9729999999999999,"5min":2.9729999999999999,"15min":2.9729999999999999},"last":0.81699999999999995}]},{"osd":4,"last update":"Sun Nov 5 08:33:57 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":1.0600000000000001},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.482,"5min":1.482,"15min":1.482},"last":1.177}]},{"osd":5,"last update":"Sun Nov 5 08:33:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":3.4790000000000001,"5min":3.4790000000000001,"15min":3.4790000000000001},"last":1.151},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":3.5339999999999998,"5min":3.5339999999999998,"15min":3.5339999999999998},"last":0.90600000000000003}]},{"osd":6,"last update":"Sun Nov 5 08:33:57 2023","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.55,"5min":1.55,"15min":1.55},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.5469999999999999,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":1.0269999999999999}]},{"osd":8,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.105,"15min":1.105},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.661,"5min":1.661,"15min":1.661},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.673,"5min":1.673,"15min":1.673},"last":1.0820000000000001}]},{"osd":9,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.264,"15min":1.264},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":2.8199999999999998,"5min":2.8199999999999998,"15min":2.8199999999999998},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.319,"5min":1.319,"15min":1.319},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":1.827,"5min":1.827,"15min":1.827},"last":1.28}]},{"osd":11,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":1.877,"5min":1.877,"15min":1.877},"last":1.234},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":2.8889999999999998,"5min":2.8889999999999998,"15min":2.8889999999999998},"last":1.256}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":13,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.536,"5min":1.536,"15min":1.536},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":2.778,"5min":2.778,"15min":2.778},"last":1.355},{"interface":"front","average":{"1min":1.6200000000000001,"5min":1.6200000000000001,"15min":1.6200000000000001},"min":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"max":{"1min":3.681,"5min":3.681,"15min":3.681},"last":1.1279999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]},{"osd":15,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":2.0550000000000002,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":1.331},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.5289999999999999,"15min":1.5289999999999999},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.4620000000000002,"5min":2.4620000000000002,"15min":2.4620000000000002},"last":1.3080000000000001}]}]},{"osd":11,"up_from":21,"seq":90194313238,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47192,"kb_used_data":16008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324648,"statfs":{"total":96636764160,"available":96588439552,"internally_reserved":0,"allocated":16392192,"data_stored":25136007,"data_compressed":165545,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,4,5,6,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":1,"last update":"Sun Nov 5 08:33:53 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.044,"15min":1.044},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":1.069,"5min":1.069,"15min":1.069},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.85199999999999998}]},{"osd":2,"last update":"Sun Nov 5 08:33:48 2023","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.216,"15min":1.216},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":2.298,"5min":2.298,"15min":2.298},"last":1.117},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":2.9020000000000001,"5min":2.9020000000000001,"15min":2.9020000000000001},"last":1.1499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":4,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.167,"15min":1.167},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.7390000000000001,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":1.048},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.91900000000000004}]},{"osd":5,"last update":"Sun Nov 5 08:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.244,"15min":1.244},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":3.0459999999999998,"5min":3.0459999999999998,"15min":3.0459999999999998},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":3.3799999999999999,"5min":3.3799999999999999,"15min":3.3799999999999999},"last":1.3200000000000001}]},{"osd":6,"last update":"Sun Nov 5 08:33:30 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.244,"15min":1.244},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":2.0630000000000002,"5min":2.0630000000000002,"15min":2.0630000000000002},"last":1.4319999999999999},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":2.7770000000000001,"5min":2.7770000000000001,"15min":2.7770000000000001},"last":1.532}]},{"osd":7,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.76,"5min":1.76,"15min":1.76},"last":1.508},{"interface":"front","average":{"1min":1.395,"5min":1.395,"15min":1.395},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.909,"5min":1.909,"15min":1.909},"last":1.6850000000000001}]},{"osd":8,"last update":"Sun Nov 5 08:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.9079999999999999,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.948,"5min":1.948,"15min":1.948},"last":1.774}]},{"osd":9,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":2.089,"5min":2.089,"15min":2.089},"last":1.472},{"interface":"front","average":{"1min":1.5660000000000001,"5min":1.5660000000000001,"15min":1.5660000000000001},"min":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"max":{"1min":2.1720000000000002,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":1.6499999999999999}]},{"osd":10,"last update":"Sun Nov 5 08:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.599,"5min":1.599,"15min":1.599},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.6890000000000001,"5min":2.6890000000000001,"15min":2.6890000000000001},"last":1.7969999999999999},{"interface":"front","average":{"1min":1.371,"5min":1.371,"15min":1.371},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":2.3679999999999999,"5min":2.3679999999999999,"15min":2.3679999999999999},"last":1.569}]},{"osd":12,"last update":"Sun Nov 5 08:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"max":{"1min":2.423,"5min":2.423,"15min":2.423},"last":1.8660000000000001},{"interface":"front","average":{"1min":1.702,"5min":1.702,"15min":1.702},"min":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"max":{"1min":2.6160000000000001,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":1.75}]},{"osd":13,"last update":"Sun Nov 5 08:33:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.4450000000000001,"15min":1.4450000000000001},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":2.48,"5min":2.48,"15min":2.48},"last":1.623},{"interface":"front","average":{"1min":1.714,"5min":1.714,"15min":1.714},"min":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"max":{"1min":2.6219999999999999,"5min":2.6219999999999999,"15min":2.6219999999999999},"last":1.595}]},{"osd":14,"last update":"Sun Nov 5 08:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.8580000000000001,"5min":1.8580000000000001,"15min":1.8580000000000001},"min":{"1min":1.137,"5min":1.137,"15min":1.137},"max":{"1min":5.375,"5min":5.375,"15min":5.375},"last":1.724},{"interface":"front","average":{"1min":1.754,"5min":1.754,"15min":1.754},"min":{"1min":1.038,"5min":1.038,"15min":1.038},"max":{"1min":2.5369999999999999,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.9179999999999999}]},{"osd":15,"last update":"Sun Nov 5 08:33:30 2023","interfaces":[{"interface":"back","average":{"1min":2.125,"5min":2.125,"15min":2.125},"min":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":1.843},{"interface":"front","average":{"1min":1.762,"5min":1.762,"15min":1.762},"min":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"max":{"1min":5.2549999999999999,"5min":5.2549999999999999,"15min":5.2549999999999999},"last":1.819}]}]},{"osd":12,"up_from":22,"seq":94489280533,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34744,"kb_used_data":7724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337096,"statfs":{"total":96636764160,"available":96601186304,"internally_reserved":0,"allocated":7909376,"data_stored":10623293,"data_compressed":165545,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[2,3,4,5,6,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":1,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.603,"5min":1.603,"15min":1.603},"last":1.2909999999999999}]},{"osd":2,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":1.0309999999999999},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.64400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0980000000000001}]},{"osd":4,"last update":"Sun Nov 5 08:34:02 2023","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.429,"5min":1.429,"15min":1.429},"last":1.0680000000000001}]},{"osd":5,"last update":"Sun Nov 5 08:34:02 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":1.123},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":1.2809999999999999}]},{"osd":6,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.73299999999999998},{"interface":"front","average":{"1min":1.073,"5min":1.073,"15min":1.073},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":1.343}]},{"osd":9,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.208,"15min":1.208},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":2.2109999999999999,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":0.99099999999999999},{"interface":"front","average":{"1min":1.206,"5min":1.206,"15min":1.206},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.964,"5min":1.964,"15min":1.964},"last":0.96399999999999997}]},{"osd":11,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":2.1659999999999999,"5min":2.1659999999999999,"15min":2.1659999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.8680000000000001,"5min":1.8680000000000001,"15min":1.8680000000000001},"last":1.149}]},{"osd":13,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.25,"5min":2.25,"15min":2.25},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":2.399,"5min":2.399,"15min":2.399},"last":1.391}]},{"osd":14,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":2.1070000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.218},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":2.3420000000000001,"5min":2.3420000000000001,"15min":2.3420000000000001},"last":1.2450000000000001}]},{"osd":15,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.2959999999999998,"5min":2.2959999999999998,"15min":2.2959999999999998},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.4099999999999999,"15min":1.4099999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.873,"5min":1.873,"15min":1.873},"last":1.3169999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6949,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6949,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4165632,"data_stored":4165632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14594048,"data_stored":21921792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11247616,"data_stored":20656128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8929280,"data_stored":11505664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12468224,"data_stored":18362368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":6107136,"data_stored":7225344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15101952,"data_stored":20791296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12271616,"data_stored":18165760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":18612224,"data_stored":33574912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":10563584,"data_stored":13963264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":13856768,"data_stored":25358336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":15134720,"data_stored":24154112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":6656000,"data_stored":9641984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":14376960,"data_stored":24563712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":12091392,"data_stored":24596480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":9965568,"data_stored":15785984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-05T08:34:04.677 INFO:tasks.thrashosds.thrasher:Setting 3.9 to [15, 11, 8, 1, 6, 3, 13] 2023-11-05T08:34:04.677 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.9', '15', '11', '8', '1', '6', '3', '13'] 2023-11-05T08:34:04.677 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 3.9 15 11 8 1 6 3 13 2023-11-05T08:34:04.678 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9: {} 2023-11-05T08:34:04.678 INFO:teuthology.orchestra.run.smithi002.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-11-05T08:34:04.690 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:34:04.690 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [], 2023-11-05T08:34:04.690 INFO:teuthology.orchestra.run.smithi002.stdout: "num_ops": 0 2023-11-05T08:34:04.690 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:34:04.697 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10: {} 2023-11-05T08:34:04.698 INFO:teuthology.orchestra.run.smithi002.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-11-05T08:34:04.704 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:34:04.700+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:34:04.705 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-11-05T08:34:04.719 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11: {} 2023-11-05T08:34:04.719 INFO:teuthology.orchestra.run.smithi002.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-11-05T08:34:04.738 INFO:teuthology.orchestra.run.smithi002.stdout:osd.12: {} 2023-11-05T08:34:04.738 INFO:teuthology.orchestra.run.smithi002.stderr:osd.12: osd_enable_op_tracker = 'true' 2023-11-05T08:34:04.760 INFO:teuthology.orchestra.run.smithi002.stdout:osd.13: {} 2023-11-05T08:34:04.760 INFO:teuthology.orchestra.run.smithi002.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-11-05T08:34:04.781 INFO:teuthology.orchestra.run.smithi002.stdout:osd.14: {} 2023-11-05T08:34:04.781 INFO:teuthology.orchestra.run.smithi002.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-11-05T08:34:04.801 INFO:teuthology.orchestra.run.smithi002.stdout:osd.15: {} 2023-11-05T08:34:04.801 INFO:teuthology.orchestra.run.smithi002.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-11-05T08:34:04.805 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:34:04.801+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:34:04.806 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-05T08:34:04.806 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-05T08:34:04.839 INFO:tasks.rados.rados.0.smithi090.stdout:376: done (4 left) 2023-11-05T08:34:04.840 INFO:tasks.rados.rados.0.smithi090.stdout:377: done (3 left) 2023-11-05T08:34:04.840 INFO:tasks.rados.rados.0.smithi090.stdout:381: done (2 left) 2023-11-05T08:34:04.840 INFO:tasks.rados.rados.0.smithi090.stdout:383: done (1 left) 2023-11-05T08:34:04.840 INFO:tasks.rados.rados.0.smithi090.stdout:385: done (0 left) 2023-11-05T08:34:04.840 INFO:tasks.rados.rados.0.smithi090.stdout:386: delete oid 18 current snap is 39 2023-11-05T08:34:04.841 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:34:04.841 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [], 2023-11-05T08:34:04.841 INFO:teuthology.orchestra.run.smithi002.stdout: "complaint_time": 30, 2023-11-05T08:34:04.841 INFO:teuthology.orchestra.run.smithi002.stdout: "num_blocked_ops": 0 2023-11-05T08:34:04.842 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:34:04.847 INFO:tasks.rados.rados.0.smithi090.stdout:386: done (0 left) 2023-11-05T08:34:04.847 INFO:tasks.rados.rados.0.smithi090.stdout:387: rollback oid 7 current snap is 39 2023-11-05T08:34:04.847 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 7 to 33 2023-11-05T08:34:04.848 INFO:tasks.rados.rados.0.smithi090.stdout:388: snap_create 2023-11-05T08:34:04.858 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-11-05T08:34:04.858 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-11-05T08:34:04.863 INFO:tasks.rados.rados.0.smithi090.stdout:387: finishing rollback tid 0 to smithi09039252-7 2023-11-05T08:34:04.863 INFO:tasks.rados.rados.0.smithi090.stdout:387: finishing rollback tid 1 to smithi09039252-7 2023-11-05T08:34:04.863 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 7 v 33 (ObjNum 129 snap 0 seq_num 2670186576) dirty dne 2023-11-05T08:34:04.951 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:34:04.949+0000 7fdc5cd58640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:34:05.006 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:34:05.003+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:34:05.021 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:34:05.021 INFO:teuthology.orchestra.run.smithi002.stdout: "size": 20, 2023-11-05T08:34:05.021 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 600, 2023-11-05T08:34:05.022 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [ 2023-11-05T08:34:05.022 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.022 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:287 3.11s0 3:88a306be:::smithi09039252-16:head [append 0~770048] snapc 8=[8,7,6,5,3] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:34:05.022 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:53.182407+0000", 2023-11-05T08:34:05.022 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 71.836248976999997, 2023-11-05T08:34:05.023 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.011495974, 2023-11-05T08:34:05.023 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:05.023 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:05.023 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:34:05.023 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:34:05.024 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:05.024 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 287 2023-11-05T08:34:05.024 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.024 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:05.024 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.025 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:05.025 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.182407+0000", 2023-11-05T08:34:05.025 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.025 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.025 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.026 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:05.026 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.182407+0000", 2023-11-05T08:34:05.026 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.026 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.026 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.027 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:05.027 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.182409+0000", 2023-11-05T08:34:05.027 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.3590000000000002e-06 2023-11-05T08:34:05.027 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.027 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.027 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:05.028 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.183780+0000", 2023-11-05T08:34:05.028 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00137108 2023-11-05T08:34:05.028 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.028 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.028 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:05.029 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.183800+0000", 2023-11-05T08:34:05.029 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.9819000000000001e-05 2023-11-05T08:34:05.029 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.029 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.029 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:05.030 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.183810+0000", 2023-11-05T08:34:05.030 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.6940000000000007e-06 2023-11-05T08:34:05.030 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.030 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.030 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.031 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.183873+0000", 2023-11-05T08:34:05.031 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.3536999999999998e-05 2023-11-05T08:34:05.031 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.031 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.031 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for peered", 2023-11-05T08:34:05.031 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.183877+0000", 2023-11-05T08:34:05.032 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.1779999999999996e-06 2023-11-05T08:34:05.033 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.033 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.033 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.034 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.171366+0000", 2023-11-05T08:34:05.034 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.98748831999999997 2023-11-05T08:34:05.034 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.034 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.034 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:34:05.035 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.171750+0000", 2023-11-05T08:34:05.035 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00038434800000000001 2023-11-05T08:34:05.035 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.035 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.035 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:05.036 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.174288+0000", 2023-11-05T08:34:05.036 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.002537957 2023-11-05T08:34:05.036 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.036 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.036 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:05.037 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.181565+0000", 2023-11-05T08:34:05.037 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0072769530000000001 2023-11-05T08:34:05.037 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.037 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.037 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:34:05.038 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.193786+0000", 2023-11-05T08:34:05.038 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.012221037000000001 2023-11-05T08:34:05.038 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.038 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.039 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:05.039 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.193903+0000", 2023-11-05T08:34:05.039 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000116692 2023-11-05T08:34:05.039 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.039 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:05.040 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.040 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.040 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.040 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:288 3.11s0 3:88a306be:::smithi09039252-16:head [append 0~376832] snapc 8=[8,7,6,5,3] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:34:05.040 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:53.186355+0000", 2023-11-05T08:34:05.041 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 71.832301197999996, 2023-11-05T08:34:05.041 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0078397800000001, 2023-11-05T08:34:05.041 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:05.041 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:05.041 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:34:05.041 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:34:05.042 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:05.042 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 288 2023-11-05T08:34:05.042 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.042 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:05.042 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.043 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:05.043 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186355+0000", 2023-11-05T08:34:05.043 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.043 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.043 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.044 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:05.044 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186355+0000", 2023-11-05T08:34:05.044 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.044 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.044 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.045 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:05.045 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186355+0000", 2023-11-05T08:34:05.045 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.9400000000000004e-07 2023-11-05T08:34:05.045 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.045 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.046 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:05.046 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186649+0000", 2023-11-05T08:34:05.046 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00029353199999999999 2023-11-05T08:34:05.046 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.046 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.046 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:05.047 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186662+0000", 2023-11-05T08:34:05.047 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.2728000000000001e-05 2023-11-05T08:34:05.047 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.047 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.047 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:05.048 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186667+0000", 2023-11-05T08:34:05.048 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.8270000000000003e-06 2023-11-05T08:34:05.048 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.048 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.048 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.049 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186742+0000", 2023-11-05T08:34:05.049 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.4328999999999999e-05 2023-11-05T08:34:05.049 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.049 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.050 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for peered", 2023-11-05T08:34:05.050 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186744+0000", 2023-11-05T08:34:05.050 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.266e-06 2023-11-05T08:34:05.050 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.051 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.051 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.051 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.176861+0000", 2023-11-05T08:34:05.051 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.990116577 2023-11-05T08:34:05.052 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.052 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.052 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:34:05.053 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.177014+0000", 2023-11-05T08:34:05.053 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000153661 2023-11-05T08:34:05.053 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.053 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.053 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:05.053 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.178253+0000", 2023-11-05T08:34:05.054 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0012384399999999999 2023-11-05T08:34:05.054 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.054 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.054 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:05.055 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.181931+0000", 2023-11-05T08:34:05.055 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0036777730000000001 2023-11-05T08:34:05.055 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.055 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.055 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:34:05.056 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.194093+0000", 2023-11-05T08:34:05.056 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.012162885999999999 2023-11-05T08:34:05.056 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.056 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.056 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:05.057 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.194194+0000", 2023-11-05T08:34:05.057 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00010086699999999999 2023-11-05T08:34:05.057 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.057 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:05.057 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.058 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.058 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.058 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:289 3.11s0 3:88a306be:::smithi09039252-16:head [setxattr _header (56) in=7b] snapc 8=[8,7,6,5,3] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:34:05.058 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:53.186682+0000", 2023-11-05T08:34:05.059 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 71.831973851000001, 2023-11-05T08:34:05.059 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.008829143, 2023-11-05T08:34:05.059 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:05.059 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:05.059 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:34:05.060 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:34:05.060 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:05.060 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 289 2023-11-05T08:34:05.060 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.060 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:05.061 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.061 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:05.061 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186682+0000", 2023-11-05T08:34:05.061 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.061 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.062 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.062 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:05.062 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186682+0000", 2023-11-05T08:34:05.062 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.062 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.063 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.063 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:05.063 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186683+0000", 2023-11-05T08:34:05.063 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.8999999999999996e-07 2023-11-05T08:34:05.064 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.064 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.064 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:05.064 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186687+0000", 2023-11-05T08:34:05.064 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.933e-06 2023-11-05T08:34:05.064 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.065 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.065 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:05.065 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186697+0000", 2023-11-05T08:34:05.065 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.6120000000000005e-06 2023-11-05T08:34:05.066 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.066 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.066 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:05.066 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186704+0000", 2023-11-05T08:34:05.066 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.8929999999999999e-06 2023-11-05T08:34:05.066 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.067 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.067 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.067 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186817+0000", 2023-11-05T08:34:05.067 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000113229 2023-11-05T08:34:05.067 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.068 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.068 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for peered", 2023-11-05T08:34:05.068 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186819+0000", 2023-11-05T08:34:05.068 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.57e-06 2023-11-05T08:34:05.068 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.069 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.069 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.069 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.179928+0000", 2023-11-05T08:34:05.069 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.993109099 2023-11-05T08:34:05.069 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.070 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.070 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:34:05.070 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.180084+0000", 2023-11-05T08:34:05.070 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00015647 2023-11-05T08:34:05.070 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.071 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.071 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:05.071 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.181012+0000", 2023-11-05T08:34:05.071 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00092730299999999998 2023-11-05T08:34:05.072 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.072 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.072 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:05.072 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.183961+0000", 2023-11-05T08:34:05.073 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0029494510000000001 2023-11-05T08:34:05.073 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.073 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.073 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:34:05.073 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.195373+0000", 2023-11-05T08:34:05.074 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.011411607000000001 2023-11-05T08:34:05.074 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.074 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.075 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:05.075 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.195511+0000", 2023-11-05T08:34:05.075 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000138286 2023-11-05T08:34:05.075 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.075 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:05.076 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.076 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.076 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.076 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:290 3.11s0 3:88a306be:::smithi09039252-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:34:05.077 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:53.186775+0000", 2023-11-05T08:34:05.077 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 71.831880366000007, 2023-11-05T08:34:05.077 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0117209439999999, 2023-11-05T08:34:05.077 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:05.078 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "started", 2023-11-05T08:34:05.078 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:34:05.078 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:34:05.078 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:05.079 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 290 2023-11-05T08:34:05.079 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.079 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:05.079 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.080 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:05.080 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186775+0000", 2023-11-05T08:34:05.080 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.080 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.081 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.081 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:05.081 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186775+0000", 2023-11-05T08:34:05.081 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.082 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.082 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.082 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:05.082 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186776+0000", 2023-11-05T08:34:05.083 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.7999999999999993e-07 2023-11-05T08:34:05.083 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.083 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.083 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:05.083 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186780+0000", 2023-11-05T08:34:05.084 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.9249999999999997e-06 2023-11-05T08:34:05.084 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.084 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.084 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:05.084 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186792+0000", 2023-11-05T08:34:05.085 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.1841999999999999e-05 2023-11-05T08:34:05.085 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.085 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.085 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:05.085 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186800+0000", 2023-11-05T08:34:05.086 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.1559999999999995e-06 2023-11-05T08:34:05.086 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.086 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.086 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.086 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186855+0000", 2023-11-05T08:34:05.086 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.4892999999999998e-05 2023-11-05T08:34:05.087 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.087 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.087 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for peered", 2023-11-05T08:34:05.087 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.186856+0000", 2023-11-05T08:34:05.087 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.1999999999999999e-07 2023-11-05T08:34:05.088 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.088 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.088 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.088 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.181764+0000", 2023-11-05T08:34:05.088 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.99490820300000005 2023-11-05T08:34:05.089 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.089 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.089 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for rw locks", 2023-11-05T08:34:05.089 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.181917+0000", 2023-11-05T08:34:05.089 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00015282300000000001 2023-11-05T08:34:05.090 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.090 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.090 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.090 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.196083+0000", 2023-11-05T08:34:05.090 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.014166247999999999 2023-11-05T08:34:05.091 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.091 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.091 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:34:05.092 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.196295+0000", 2023-11-05T08:34:05.092 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00021158599999999999 2023-11-05T08:34:05.092 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.092 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.092 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:05.093 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.198496+0000", 2023-11-05T08:34:05.093 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0022016679999999999 2023-11-05T08:34:05.093 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.093 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:05.093 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.094 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.094 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.094 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:334 3.1s0 3:863368a9:::smithi09039252-6:head [read 0~98304,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2023-11-05T08:34:05.094 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:04.226390+0000", 2023-11-05T08:34:05.094 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 60.792266083000001, 2023-11-05T08:34:05.094 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.017587591999999999, 2023-11-05T08:34:05.095 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:05.095 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "started", 2023-11-05T08:34:05.095 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:34:05.095 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:34:05.095 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:05.096 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 334 2023-11-05T08:34:05.096 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.096 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:05.096 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.096 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:05.097 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226390+0000", 2023-11-05T08:34:05.097 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.097 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.097 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.097 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:05.097 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226390+0000", 2023-11-05T08:34:05.098 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.098 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.098 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.098 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:05.098 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226390+0000", 2023-11-05T08:34:05.099 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.1999999999999998e-07 2023-11-05T08:34:05.099 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.099 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.099 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:05.099 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226396+0000", 2023-11-05T08:34:05.100 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.3229999999999997e-06 2023-11-05T08:34:05.100 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.100 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.100 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:05.100 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226409+0000", 2023-11-05T08:34:05.100 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.3096e-05 2023-11-05T08:34:05.101 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.101 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.101 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:05.101 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.226417+0000", 2023-11-05T08:34:05.102 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.1300000000000001e-06 2023-11-05T08:34:05.102 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.102 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.102 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.102 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.233238+0000", 2023-11-05T08:34:05.102 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0068212999999999998 2023-11-05T08:34:05.103 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.103 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.103 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:34:05.103 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.233474+0000", 2023-11-05T08:34:05.103 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000235286 2023-11-05T08:34:05.104 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.104 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.104 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:05.104 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:04.243977+0000", 2023-11-05T08:34:05.104 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.010503637 2023-11-05T08:34:05.105 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.105 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:05.105 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.105 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.105 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.105 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:416 3.1s0 3:8539e931:::smithi09039252-41:head [append 0~671744] snapc 1c=[1c,1b,1a,17,7] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2023-11-05T08:34:05.106 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:15.354023+0000", 2023-11-05T08:34:05.106 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 49.664632844000003, 2023-11-05T08:34:05.106 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.016832657000000001, 2023-11-05T08:34:05.106 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:05.106 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:05.107 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:34:05.107 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:34:05.107 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:05.107 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 416 2023-11-05T08:34:05.107 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.108 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:05.108 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.108 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:05.108 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.354023+0000", 2023-11-05T08:34:05.108 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.109 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.109 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.109 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:05.109 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.354023+0000", 2023-11-05T08:34:05.109 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.110 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.110 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.110 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:05.110 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.354026+0000", 2023-11-05T08:34:05.110 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.7669999999999999e-06 2023-11-05T08:34:05.111 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.111 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.111 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:05.111 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.355125+0000", 2023-11-05T08:34:05.111 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001099399 2023-11-05T08:34:05.111 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.112 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.112 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:05.112 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.355163+0000", 2023-11-05T08:34:05.112 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.8232000000000003e-05 2023-11-05T08:34:05.112 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.113 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.113 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:05.113 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.355182+0000", 2023-11-05T08:34:05.113 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.9153000000000001e-05 2023-11-05T08:34:05.113 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.114 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.114 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.114 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.355293+0000", 2023-11-05T08:34:05.114 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00011033500000000001 2023-11-05T08:34:05.114 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.114 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.115 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "op must wait for map", 2023-11-05T08:34:05.115 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.355304+0000", 2023-11-05T08:34:05.115 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.1253000000000001e-05 2023-11-05T08:34:05.115 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.115 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.116 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.116 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.359303+0000", 2023-11-05T08:34:05.116 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.003999173 2023-11-05T08:34:05.116 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.116 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.117 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:34:05.117 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.359491+0000", 2023-11-05T08:34:05.117 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00018750099999999999 2023-11-05T08:34:05.117 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.117 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.117 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:05.118 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.360889+0000", 2023-11-05T08:34:05.118 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001397971 2023-11-05T08:34:05.118 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.118 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.119 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:05.119 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.363175+0000", 2023-11-05T08:34:05.119 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.002286738 2023-11-05T08:34:05.119 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.119 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.120 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:34:05.120 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.370683+0000", 2023-11-05T08:34:05.120 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0075076700000000001 2023-11-05T08:34:05.120 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.121 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.121 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:05.121 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.370856+0000", 2023-11-05T08:34:05.121 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00017246500000000001 2023-11-05T08:34:05.121 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.122 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:05.122 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.122 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.122 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.122 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:418 3.1s0 3:8539e931:::smithi09039252-41:head [setxattr _header (57) in=7b] snapc 1c=[1c,1b,1a,17,7] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2023-11-05T08:34:05.123 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:15.367398+0000", 2023-11-05T08:34:05.123 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 49.651257653999998, 2023-11-05T08:34:05.123 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0079965609999999993, 2023-11-05T08:34:05.123 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:05.123 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:05.124 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:34:05.124 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:34:05.124 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:05.124 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 418 2023-11-05T08:34:05.124 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.125 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:05.125 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.125 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:05.125 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.367398+0000", 2023-11-05T08:34:05.125 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.126 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.126 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.126 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:05.126 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.367398+0000", 2023-11-05T08:34:05.126 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.126 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.127 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.127 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:05.127 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.367399+0000", 2023-11-05T08:34:05.127 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0699999999999999e-06 2023-11-05T08:34:05.127 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.128 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.128 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:05.128 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.367404+0000", 2023-11-05T08:34:05.128 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.1000000000000003e-06 2023-11-05T08:34:05.128 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.129 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.129 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:05.129 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.367416+0000", 2023-11-05T08:34:05.129 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.154e-05 2023-11-05T08:34:05.129 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.129 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.130 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:05.130 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.367424+0000", 2023-11-05T08:34:05.130 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.2889999999999998e-06 2023-11-05T08:34:05.130 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.130 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.131 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.131 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.369523+0000", 2023-11-05T08:34:05.131 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0020993610000000001 2023-11-05T08:34:05.131 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.131 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.132 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:34:05.132 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.369627+0000", 2023-11-05T08:34:05.132 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00010375800000000001 2023-11-05T08:34:05.132 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.132 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.133 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:05.133 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.370111+0000", 2023-11-05T08:34:05.133 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000484303 2023-11-05T08:34:05.133 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.133 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.134 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:05.134 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.371243+0000", 2023-11-05T08:34:05.134 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001131993 2023-11-05T08:34:05.134 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.134 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.134 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:34:05.135 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.375135+0000", 2023-11-05T08:34:05.135 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0038920199999999999 2023-11-05T08:34:05.135 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.135 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.135 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:05.136 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.375395+0000", 2023-11-05T08:34:05.136 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000259127 2023-11-05T08:34:05.136 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.136 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:05.136 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.137 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.137 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.137 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:419 3.1s0 3:8539e931:::smithi09039252-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2023-11-05T08:34:05.137 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:15.367440+0000", 2023-11-05T08:34:05.137 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 49.651215915000002, 2023-11-05T08:34:05.137 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.011845488, 2023-11-05T08:34:05.138 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:05.138 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "started", 2023-11-05T08:34:05.138 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:34:05.138 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:34:05.138 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:05.139 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 419 2023-11-05T08:34:05.139 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.139 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:05.139 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.139 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:05.140 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.367440+0000", 2023-11-05T08:34:05.140 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.140 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.140 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.140 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:05.141 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.367440+0000", 2023-11-05T08:34:05.141 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.141 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.141 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.141 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:05.142 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.367441+0000", 2023-11-05T08:34:05.142 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.8299999999999996e-07 2023-11-05T08:34:05.142 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.142 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.142 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:05.143 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.367444+0000", 2023-11-05T08:34:05.143 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.2499999999999998e-06 2023-11-05T08:34:05.143 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.143 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.143 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:05.144 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.367459+0000", 2023-11-05T08:34:05.144 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4763000000000001e-05 2023-11-05T08:34:05.144 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.144 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.144 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:05.145 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.367467+0000", 2023-11-05T08:34:05.145 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.0700000000000007e-06 2023-11-05T08:34:05.145 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.145 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.145 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.145 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.370959+0000", 2023-11-05T08:34:05.146 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.003491854 2023-11-05T08:34:05.146 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.146 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.146 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for rw locks", 2023-11-05T08:34:05.146 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.371115+0000", 2023-11-05T08:34:05.147 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000156102 2023-11-05T08:34:05.147 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.147 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.147 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.147 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.376485+0000", 2023-11-05T08:34:05.148 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0053701720000000003 2023-11-05T08:34:05.148 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.148 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.148 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:34:05.148 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.376689+0000", 2023-11-05T08:34:05.149 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000203771 2023-11-05T08:34:05.149 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.149 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.149 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:05.149 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.379285+0000", 2023-11-05T08:34:05.149 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.002596723 2023-11-05T08:34:05.150 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.150 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:05.150 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.150 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.150 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.151 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:496 3.bs0 3:dd412ebb:::smithi09039252-37:head [append 0~802816] snapc 1c=[7] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2023-11-05T08:34:05.151 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:19.389413+0000", 2023-11-05T08:34:05.151 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 45.629242245, 2023-11-05T08:34:05.151 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.027801348, 2023-11-05T08:34:05.151 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:05.152 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:05.152 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:34:05.152 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:34:05.152 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:05.152 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 496 2023-11-05T08:34:05.153 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.153 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:05.153 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.153 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:05.153 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.389413+0000", 2023-11-05T08:34:05.153 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.154 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.154 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.154 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:05.154 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.389413+0000", 2023-11-05T08:34:05.154 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.155 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.155 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.155 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:05.155 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.389417+0000", 2023-11-05T08:34:05.155 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.4769999999999999e-06 2023-11-05T08:34:05.156 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.156 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.156 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:05.156 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.391074+0000", 2023-11-05T08:34:05.156 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001657419 2023-11-05T08:34:05.157 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.157 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.157 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:05.157 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.391113+0000", 2023-11-05T08:34:05.157 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.8279999999999999e-05 2023-11-05T08:34:05.158 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.158 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.158 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:05.158 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.391131+0000", 2023-11-05T08:34:05.158 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.8428999999999999e-05 2023-11-05T08:34:05.159 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.159 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.159 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.159 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.391273+0000", 2023-11-05T08:34:05.159 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000141707 2023-11-05T08:34:05.160 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.160 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.160 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "op must wait for map", 2023-11-05T08:34:05.160 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.391282+0000", 2023-11-05T08:34:05.160 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.0410000000000003e-06 2023-11-05T08:34:05.160 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.161 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.161 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.161 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.397377+0000", 2023-11-05T08:34:05.161 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.006094721 2023-11-05T08:34:05.161 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.162 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.162 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:34:05.162 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.397568+0000", 2023-11-05T08:34:05.162 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00019143799999999999 2023-11-05T08:34:05.162 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.163 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.163 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:05.163 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.399855+0000", 2023-11-05T08:34:05.163 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.002287491 2023-11-05T08:34:05.163 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.164 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.164 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:05.164 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.405746+0000", 2023-11-05T08:34:05.164 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0058907439999999998 2023-11-05T08:34:05.164 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.165 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.165 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:34:05.165 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.416872+0000", 2023-11-05T08:34:05.165 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.011125823999999999 2023-11-05T08:34:05.165 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.166 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.166 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:05.166 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.417215+0000", 2023-11-05T08:34:05.166 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000342777 2023-11-05T08:34:05.166 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.167 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:05.167 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.167 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.167 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.167 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:497 3.bs0 3:dd412ebb:::smithi09039252-37:head [append 0~638976] snapc 1c=[7] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2023-11-05T08:34:05.168 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:19.394162+0000", 2023-11-05T08:34:05.168 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 45.624493993000002, 2023-11-05T08:34:05.168 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.029484893000000002, 2023-11-05T08:34:05.168 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:05.168 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:05.169 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:34:05.169 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:34:05.169 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:05.169 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 497 2023-11-05T08:34:05.169 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.170 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:05.170 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.170 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:05.170 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.394162+0000", 2023-11-05T08:34:05.170 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.171 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.171 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.171 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:05.171 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.394162+0000", 2023-11-05T08:34:05.171 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.172 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.172 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.172 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:05.172 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.394164+0000", 2023-11-05T08:34:05.172 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.2000000000000001e-06 2023-11-05T08:34:05.172 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.173 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.173 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:05.173 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.394956+0000", 2023-11-05T08:34:05.173 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00079165300000000002 2023-11-05T08:34:05.173 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.174 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.174 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:05.174 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.394980+0000", 2023-11-05T08:34:05.174 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.4851999999999999e-05 2023-11-05T08:34:05.175 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.175 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.175 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:05.175 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.394996+0000", 2023-11-05T08:34:05.175 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.5498999999999998e-05 2023-11-05T08:34:05.176 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.176 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.176 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.176 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395277+0000", 2023-11-05T08:34:05.176 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00028066300000000001 2023-11-05T08:34:05.177 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.177 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.177 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:34:05.177 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395284+0000", 2023-11-05T08:34:05.177 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.6580000000000002e-06 2023-11-05T08:34:05.178 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.178 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.178 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.178 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.402581+0000", 2023-11-05T08:34:05.178 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0072969560000000003 2023-11-05T08:34:05.179 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.179 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.179 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:34:05.179 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.402751+0000", 2023-11-05T08:34:05.179 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000169922 2023-11-05T08:34:05.180 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.180 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.180 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:05.180 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.404210+0000", 2023-11-05T08:34:05.180 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0014591649999999999 2023-11-05T08:34:05.181 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.181 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.181 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:05.181 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.408453+0000", 2023-11-05T08:34:05.181 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0042427790000000003 2023-11-05T08:34:05.182 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.182 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.182 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:34:05.182 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.423250+0000", 2023-11-05T08:34:05.182 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.014797258000000001 2023-11-05T08:34:05.183 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.183 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.183 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:05.183 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.423647+0000", 2023-11-05T08:34:05.183 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00039628800000000001 2023-11-05T08:34:05.184 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.184 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:05.184 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.184 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.184 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.185 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:498 3.bs0 3:dd412ebb:::smithi09039252-37:head [append 0~131072] snapc 1c=[7] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2023-11-05T08:34:05.185 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:19.395177+0000", 2023-11-05T08:34:05.185 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 45.623479025999998, 2023-11-05T08:34:05.185 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.029134344999999999, 2023-11-05T08:34:05.185 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:05.186 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:05.186 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:34:05.186 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:34:05.186 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:05.186 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 498 2023-11-05T08:34:05.187 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.187 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:05.187 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.187 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:05.187 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395177+0000", 2023-11-05T08:34:05.188 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.188 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.188 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.188 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:05.188 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395177+0000", 2023-11-05T08:34:05.189 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.189 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.189 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.189 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:05.189 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395178+0000", 2023-11-05T08:34:05.190 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.499e-06 2023-11-05T08:34:05.190 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.190 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.190 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:05.190 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395362+0000", 2023-11-05T08:34:05.191 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00018334499999999999 2023-11-05T08:34:05.191 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.191 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.191 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:05.191 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395382+0000", 2023-11-05T08:34:05.192 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.0520999999999999e-05 2023-11-05T08:34:05.192 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.192 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.192 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:05.192 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395395+0000", 2023-11-05T08:34:05.193 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.2641e-05 2023-11-05T08:34:05.193 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.193 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.193 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.193 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395577+0000", 2023-11-05T08:34:05.194 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00018265999999999999 2023-11-05T08:34:05.194 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.194 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.194 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:34:05.194 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395590+0000", 2023-11-05T08:34:05.195 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.2476e-05 2023-11-05T08:34:05.195 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.195 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.195 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.195 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.405933+0000", 2023-11-05T08:34:05.195 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.010343629999999999 2023-11-05T08:34:05.196 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.196 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.196 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:34:05.196 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.406106+0000", 2023-11-05T08:34:05.196 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000172172 2023-11-05T08:34:05.197 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.197 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.197 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:05.197 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.406978+0000", 2023-11-05T08:34:05.197 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00087286199999999997 2023-11-05T08:34:05.198 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.198 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.198 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:05.198 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.408818+0000", 2023-11-05T08:34:05.198 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0018392 2023-11-05T08:34:05.198 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.199 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.199 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:34:05.199 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.423925+0000", 2023-11-05T08:34:05.199 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.015107517000000001 2023-11-05T08:34:05.199 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.200 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.200 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:05.200 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.424311+0000", 2023-11-05T08:34:05.200 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00038582200000000002 2023-11-05T08:34:05.200 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.201 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:05.201 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.201 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.201 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.201 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:499 3.bs0 3:dd412ebb:::smithi09039252-37:head [setxattr _header (57) in=7b] snapc 1c=[7] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2023-11-05T08:34:05.202 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:19.395440+0000", 2023-11-05T08:34:05.202 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 45.62321523, 2023-11-05T08:34:05.202 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.029659106000000001, 2023-11-05T08:34:05.202 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:05.202 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:05.202 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:34:05.203 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:34:05.203 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:05.203 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 499 2023-11-05T08:34:05.203 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.203 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:05.204 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.204 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:05.204 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395440+0000", 2023-11-05T08:34:05.204 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.204 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.205 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.205 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:05.205 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395440+0000", 2023-11-05T08:34:05.205 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.205 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.206 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.206 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:05.206 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395442+0000", 2023-11-05T08:34:05.206 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.02e-06 2023-11-05T08:34:05.206 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.207 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.207 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:05.207 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395447+0000", 2023-11-05T08:34:05.207 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.9880000000000001e-06 2023-11-05T08:34:05.207 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.207 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.208 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:05.208 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395462+0000", 2023-11-05T08:34:05.208 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4055e-05 2023-11-05T08:34:05.208 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.208 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.209 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:05.209 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395473+0000", 2023-11-05T08:34:05.209 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.1578e-05 2023-11-05T08:34:05.209 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.209 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.210 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.210 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395707+0000", 2023-11-05T08:34:05.210 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000233415 2023-11-05T08:34:05.210 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.210 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.211 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:34:05.211 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395721+0000", 2023-11-05T08:34:05.211 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4136e-05 2023-11-05T08:34:05.211 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.211 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.212 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.212 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.407602+0000", 2023-11-05T08:34:05.212 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.011880885000000001 2023-11-05T08:34:05.212 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.212 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.213 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:34:05.213 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.407685+0000", 2023-11-05T08:34:05.213 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.3615999999999994e-05 2023-11-05T08:34:05.213 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.213 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.213 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:05.214 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.408132+0000", 2023-11-05T08:34:05.214 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00044693100000000002 2023-11-05T08:34:05.214 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.214 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.215 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:05.215 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.409160+0000", 2023-11-05T08:34:05.215 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0010283740000000001 2023-11-05T08:34:05.215 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.215 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.216 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:34:05.216 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.424623+0000", 2023-11-05T08:34:05.216 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.015462075 2023-11-05T08:34:05.216 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.216 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.217 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:05.217 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.425100+0000", 2023-11-05T08:34:05.217 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00047703300000000003 2023-11-05T08:34:05.217 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.218 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:05.218 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.218 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.218 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.219 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:500 3.bs0 3:dd412ebb:::smithi09039252-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e66)", 2023-11-05T08:34:05.219 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:19.395494+0000", 2023-11-05T08:34:05.219 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 45.623162069000003, 2023-11-05T08:34:05.219 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.034730260999999998, 2023-11-05T08:34:05.219 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:05.220 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "started", 2023-11-05T08:34:05.220 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:34:05.220 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:34:05.220 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:05.221 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 500 2023-11-05T08:34:05.221 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.221 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:05.221 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.221 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:05.222 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395494+0000", 2023-11-05T08:34:05.222 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.222 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.223 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.223 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:05.223 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395494+0000", 2023-11-05T08:34:05.224 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.224 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.224 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.224 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:05.225 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395494+0000", 2023-11-05T08:34:05.225 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.5099999999999998e-07 2023-11-05T08:34:05.225 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.225 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.226 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:05.226 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395499+0000", 2023-11-05T08:34:05.226 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.2880000000000003e-06 2023-11-05T08:34:05.226 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.227 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.227 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:05.227 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395512+0000", 2023-11-05T08:34:05.227 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.3227999999999999e-05 2023-11-05T08:34:05.228 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.228 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.228 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:05.228 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395521+0000", 2023-11-05T08:34:05.228 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.4469999999999995e-06 2023-11-05T08:34:05.229 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.229 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.229 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.229 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395903+0000", 2023-11-05T08:34:05.230 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00038185600000000002 2023-11-05T08:34:05.230 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.230 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.230 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:34:05.230 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.395916+0000", 2023-11-05T08:34:05.231 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.2233e-05 2023-11-05T08:34:05.231 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.231 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.231 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.232 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.408539+0000", 2023-11-05T08:34:05.232 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.012623013000000001 2023-11-05T08:34:05.232 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.232 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.233 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for rw locks", 2023-11-05T08:34:05.233 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.408699+0000", 2023-11-05T08:34:05.233 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00016055099999999999 2023-11-05T08:34:05.233 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.233 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.234 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.234 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.425989+0000", 2023-11-05T08:34:05.234 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.017289562000000001 2023-11-05T08:34:05.234 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.234 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.235 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:34:05.235 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.426378+0000", 2023-11-05T08:34:05.235 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00038890600000000001 2023-11-05T08:34:05.235 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.235 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.237 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:05.237 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.430224+0000", 2023-11-05T08:34:05.238 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0038463260000000002 2023-11-05T08:34:05.238 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.238 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:05.238 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.239 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.239 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.239 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:683 3.bs0 3:dc0954a5:::smithi09039252-29:head [append 0~753664] snapc 38=[7] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2023-11-05T08:34:05.239 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:45.863783+0000", 2023-11-05T08:34:05.239 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 19.154873059, 2023-11-05T08:34:05.240 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.012483342999999999, 2023-11-05T08:34:05.240 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:05.240 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:05.240 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:34:05.240 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:34:05.241 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:05.241 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 683 2023-11-05T08:34:05.241 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.241 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:05.242 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.242 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:05.242 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.863783+0000", 2023-11-05T08:34:05.242 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.242 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.243 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.243 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:05.243 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.863783+0000", 2023-11-05T08:34:05.243 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.243 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.244 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.244 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:05.244 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.863785+0000", 2023-11-05T08:34:05.244 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.3700000000000002e-06 2023-11-05T08:34:05.245 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.245 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.245 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:05.245 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.864802+0000", 2023-11-05T08:34:05.246 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0010165700000000001 2023-11-05T08:34:05.246 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.246 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.246 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:05.246 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.864836+0000", 2023-11-05T08:34:05.247 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.4629000000000003e-05 2023-11-05T08:34:05.247 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.247 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.247 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:05.247 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.864852+0000", 2023-11-05T08:34:05.248 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.6197000000000001e-05 2023-11-05T08:34:05.248 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.248 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.248 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.249 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.864998+0000", 2023-11-05T08:34:05.249 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00014591699999999999 2023-11-05T08:34:05.249 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.249 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.249 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:34:05.250 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.865174+0000", 2023-11-05T08:34:05.250 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00017605600000000001 2023-11-05T08:34:05.250 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.250 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.250 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:05.251 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.867180+0000", 2023-11-05T08:34:05.251 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.002005714 2023-11-05T08:34:05.251 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.251 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.251 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:05.252 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.875021+0000", 2023-11-05T08:34:05.252 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0078406120000000003 2023-11-05T08:34:05.252 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.252 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.252 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:34:05.253 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.875690+0000", 2023-11-05T08:34:05.253 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00066926500000000005 2023-11-05T08:34:05.253 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.253 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.254 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:05.254 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.876266+0000", 2023-11-05T08:34:05.254 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000576013 2023-11-05T08:34:05.254 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.254 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:05.255 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.255 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.255 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.255 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:684 3.bs0 3:dc0954a5:::smithi09039252-29:head [append 0~688128] snapc 38=[7] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2023-11-05T08:34:05.255 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:45.869613+0000", 2023-11-05T08:34:05.256 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 19.149042751, 2023-11-05T08:34:05.256 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.014313928, 2023-11-05T08:34:05.256 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:05.256 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:05.257 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:34:05.257 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:34:05.257 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:05.257 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 684 2023-11-05T08:34:05.257 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.258 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:05.258 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.258 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:05.258 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.869613+0000", 2023-11-05T08:34:05.259 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.259 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.259 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.259 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:05.259 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.869613+0000", 2023-11-05T08:34:05.260 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.260 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.260 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.260 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:05.260 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.869616+0000", 2023-11-05T08:34:05.261 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.2059999999999999e-06 2023-11-05T08:34:05.261 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.261 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.261 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:05.261 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.870840+0000", 2023-11-05T08:34:05.262 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001223949 2023-11-05T08:34:05.262 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.262 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.262 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:05.263 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.870885+0000", 2023-11-05T08:34:05.263 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.5195999999999999e-05 2023-11-05T08:34:05.263 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.263 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.263 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:05.264 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.870917+0000", 2023-11-05T08:34:05.264 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.1982e-05 2023-11-05T08:34:05.264 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.264 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.265 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.265 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.871067+0000", 2023-11-05T08:34:05.265 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00015016399999999999 2023-11-05T08:34:05.265 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.265 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.266 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:34:05.266 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.871319+0000", 2023-11-05T08:34:05.266 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000252007 2023-11-05T08:34:05.266 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.266 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.267 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:05.267 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.872813+0000", 2023-11-05T08:34:05.267 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001493716 2023-11-05T08:34:05.267 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.268 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.268 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:05.268 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.880804+0000", 2023-11-05T08:34:05.268 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0079913180000000007 2023-11-05T08:34:05.268 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.269 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.269 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:34:05.269 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.883483+0000", 2023-11-05T08:34:05.269 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.002678274 2023-11-05T08:34:05.269 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.270 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.270 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:05.270 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.883927+0000", 2023-11-05T08:34:05.270 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00044411599999999999 2023-11-05T08:34:05.271 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.271 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:05.271 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.271 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.272 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.272 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:685 3.bs0 3:dc0954a5:::smithi09039252-29:head [append 0~475136] snapc 38=[7] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2023-11-05T08:34:05.272 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:45.873661+0000", 2023-11-05T08:34:05.272 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 19.144995196, 2023-11-05T08:34:05.272 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.011676574, 2023-11-05T08:34:05.273 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:05.273 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:05.273 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:34:05.273 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:34:05.274 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:05.274 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 685 2023-11-05T08:34:05.274 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.274 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:05.275 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.275 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:05.275 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.873661+0000", 2023-11-05T08:34:05.275 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.275 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.276 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.276 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:05.276 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.873661+0000", 2023-11-05T08:34:05.276 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.276 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.277 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.277 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:05.277 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.873663+0000", 2023-11-05T08:34:05.277 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.4229999999999999e-06 2023-11-05T08:34:05.278 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.278 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.278 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:05.278 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.874360+0000", 2023-11-05T08:34:05.278 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00069731100000000005 2023-11-05T08:34:05.279 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.279 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.279 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:05.279 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.874403+0000", 2023-11-05T08:34:05.280 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.2409e-05 2023-11-05T08:34:05.280 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.280 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.280 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:05.280 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.874424+0000", 2023-11-05T08:34:05.281 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.1719e-05 2023-11-05T08:34:05.281 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.281 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.281 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.282 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.876739+0000", 2023-11-05T08:34:05.282 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.002314698 2023-11-05T08:34:05.282 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.282 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.282 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:34:05.283 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.877046+0000", 2023-11-05T08:34:05.283 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00030658299999999997 2023-11-05T08:34:05.283 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.283 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.283 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:05.284 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.878985+0000", 2023-11-05T08:34:05.284 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001938995 2023-11-05T08:34:05.284 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.284 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.285 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:05.285 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.883187+0000", 2023-11-05T08:34:05.285 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0042024280000000002 2023-11-05T08:34:05.285 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.285 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.286 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:34:05.286 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.885058+0000", 2023-11-05T08:34:05.286 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001870495 2023-11-05T08:34:05.286 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.286 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.287 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:05.287 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.885337+0000", 2023-11-05T08:34:05.287 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00027951300000000001 2023-11-05T08:34:05.287 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.287 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:05.288 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.288 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.288 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.288 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:686 3.bs0 3:dc0954a5:::smithi09039252-29:head [setxattr _header (57) in=7b] snapc 38=[7] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2023-11-05T08:34:05.289 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:45.874462+0000", 2023-11-05T08:34:05.289 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 19.144193838, 2023-11-05T08:34:05.289 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.012938209000000001, 2023-11-05T08:34:05.289 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:05.289 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:05.290 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:34:05.290 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:34:05.290 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:05.290 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 686 2023-11-05T08:34:05.290 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.291 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:05.291 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.291 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:05.291 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.874462+0000", 2023-11-05T08:34:05.292 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.292 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.292 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.292 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:05.292 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.874462+0000", 2023-11-05T08:34:05.293 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.293 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.293 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.293 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:05.293 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.874464+0000", 2023-11-05T08:34:05.294 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.03e-06 2023-11-05T08:34:05.294 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.294 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.294 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:05.294 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.874473+0000", 2023-11-05T08:34:05.295 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.1630000000000002e-06 2023-11-05T08:34:05.295 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.295 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.295 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:05.296 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.874500+0000", 2023-11-05T08:34:05.296 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.6716e-05 2023-11-05T08:34:05.296 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.296 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.296 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:05.297 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.874516+0000", 2023-11-05T08:34:05.297 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.6079999999999999e-05 2023-11-05T08:34:05.297 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.297 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.297 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.298 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.880891+0000", 2023-11-05T08:34:05.298 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0063756309999999997 2023-11-05T08:34:05.298 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.298 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.298 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:34:05.299 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.881087+0000", 2023-11-05T08:34:05.299 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00019510900000000001 2023-11-05T08:34:05.299 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.299 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.299 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:05.300 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.881866+0000", 2023-11-05T08:34:05.300 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00077925900000000003 2023-11-05T08:34:05.300 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.300 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.301 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:05.301 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.884464+0000", 2023-11-05T08:34:05.301 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0025980180000000001 2023-11-05T08:34:05.301 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.301 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.302 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:34:05.302 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.887087+0000", 2023-11-05T08:34:05.302 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0026230440000000002 2023-11-05T08:34:05.302 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.302 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.303 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:05.303 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.887400+0000", 2023-11-05T08:34:05.303 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00031315899999999997 2023-11-05T08:34:05.303 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.303 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:05.304 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.304 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.304 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.304 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:687 3.bs0 3:dc0954a5:::smithi09039252-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e92)", 2023-11-05T08:34:05.305 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:45.874544+0000", 2023-11-05T08:34:05.305 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 19.144111420000002, 2023-11-05T08:34:05.305 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.016693900000000001, 2023-11-05T08:34:05.305 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:05.305 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "started", 2023-11-05T08:34:05.306 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:34:05.306 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:34:05.306 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:05.306 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 687 2023-11-05T08:34:05.306 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.307 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:05.307 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.307 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:05.307 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.874544+0000", 2023-11-05T08:34:05.307 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.308 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.308 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.308 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:05.308 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.874544+0000", 2023-11-05T08:34:05.308 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.309 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.309 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.309 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:05.309 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.874546+0000", 2023-11-05T08:34:05.309 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.61e-06 2023-11-05T08:34:05.310 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.310 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.310 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:05.310 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.874552+0000", 2023-11-05T08:34:05.311 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.1940000000000003e-06 2023-11-05T08:34:05.311 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.311 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.311 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:05.311 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.874572+0000", 2023-11-05T08:34:05.312 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.0315000000000001e-05 2023-11-05T08:34:05.312 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.312 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.312 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:05.312 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.874588+0000", 2023-11-05T08:34:05.313 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.5656999999999999e-05 2023-11-05T08:34:05.313 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.313 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.313 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.314 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.885395+0000", 2023-11-05T08:34:05.314 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.010806921000000001 2023-11-05T08:34:05.314 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.314 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.314 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for rw locks", 2023-11-05T08:34:05.315 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.885553+0000", 2023-11-05T08:34:05.315 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00015769699999999999 2023-11-05T08:34:05.315 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.315 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.316 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.316 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.888190+0000", 2023-11-05T08:34:05.316 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0026371430000000002 2023-11-05T08:34:05.316 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.316 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.317 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:34:05.317 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.888432+0000", 2023-11-05T08:34:05.317 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00024208399999999999 2023-11-05T08:34:05.317 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.318 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.318 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:05.318 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.891238+0000", 2023-11-05T08:34:05.318 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0028062790000000001 2023-11-05T08:34:05.319 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.319 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:05.319 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.319 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.319 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.320 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:769 3.1s0 3:863368a9:::smithi09039252-6:head [copy-from ver 49] snapc 42=[42,41,3d] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2023-11-05T08:34:05.320 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:53.729919+0000", 2023-11-05T08:34:05.320 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 11.288736322, 2023-11-05T08:34:05.320 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.020653845000000001, 2023-11-05T08:34:05.321 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:05.321 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:05.321 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:34:05.321 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:34:05.321 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:05.322 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 769 2023-11-05T08:34:05.322 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.322 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:05.322 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.323 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:05.323 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:53.729919+0000", 2023-11-05T08:34:05.323 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.323 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.323 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.324 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:05.324 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:53.729919+0000", 2023-11-05T08:34:05.324 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.324 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.324 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.325 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:05.325 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:53.729921+0000", 2023-11-05T08:34:05.325 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.1059999999999998e-06 2023-11-05T08:34:05.325 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.325 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.326 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:05.326 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:53.729933+0000", 2023-11-05T08:34:05.326 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.1987000000000001e-05 2023-11-05T08:34:05.326 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.327 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.327 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:05.327 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:53.729957+0000", 2023-11-05T08:34:05.327 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.3104000000000001e-05 2023-11-05T08:34:05.327 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.328 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.328 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:05.328 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:53.729968+0000", 2023-11-05T08:34:05.328 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.1868000000000001e-05 2023-11-05T08:34:05.328 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.329 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.329 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.329 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:53.730058+0000", 2023-11-05T08:34:05.329 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.9419999999999994e-05 2023-11-05T08:34:05.329 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.330 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.330 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:34:05.330 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:53.730191+0000", 2023-11-05T08:34:05.330 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00013337599999999999 2023-11-05T08:34:05.331 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.331 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.331 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:05.331 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:53.739127+0000", 2023-11-05T08:34:05.331 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0089357489999999998 2023-11-05T08:34:05.332 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.332 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.332 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:05.332 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:53.742419+0000", 2023-11-05T08:34:05.332 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0032924500000000002 2023-11-05T08:34:05.333 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.333 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.333 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:34:05.333 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:53.750457+0000", 2023-11-05T08:34:05.334 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0080377639999999993 2023-11-05T08:34:05.334 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.334 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.334 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:05.334 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:53.750573+0000", 2023-11-05T08:34:05.335 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00011602100000000001 2023-11-05T08:34:05.335 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.335 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:05.335 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.336 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.336 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.336 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:770 3.1s0 3:863368a9:::smithi09039252-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e100)", 2023-11-05T08:34:05.336 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:53.729996+0000", 2023-11-05T08:34:05.336 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 11.288660065, 2023-11-05T08:34:05.337 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.012086387000000001, 2023-11-05T08:34:05.337 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:05.337 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "started", 2023-11-05T08:34:05.337 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:34:05.338 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:34:05.338 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:05.338 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 770 2023-11-05T08:34:05.338 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.338 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:05.339 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.339 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:05.339 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:53.729996+0000", 2023-11-05T08:34:05.339 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.339 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.340 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.340 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:05.340 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:53.729996+0000", 2023-11-05T08:34:05.340 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.341 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.341 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.341 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:05.341 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:53.729996+0000", 2023-11-05T08:34:05.341 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.2699999999999999e-07 2023-11-05T08:34:05.342 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.342 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.342 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:05.342 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:53.730003+0000", 2023-11-05T08:34:05.342 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.5930000000000002e-06 2023-11-05T08:34:05.343 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.343 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.343 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:05.343 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:53.730019+0000", 2023-11-05T08:34:05.343 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.6214000000000001e-05 2023-11-05T08:34:05.344 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.344 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.344 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:05.344 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:53.730025+0000", 2023-11-05T08:34:05.345 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.0360000000000003e-06 2023-11-05T08:34:05.345 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.345 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.345 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.345 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:53.730418+0000", 2023-11-05T08:34:05.346 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000392913 2023-11-05T08:34:05.346 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.346 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.346 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for blocked object", 2023-11-05T08:34:05.346 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:53.730492+0000", 2023-11-05T08:34:05.347 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.3862000000000005e-05 2023-11-05T08:34:05.347 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.347 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.347 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.347 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:53.741841+0000", 2023-11-05T08:34:05.348 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.011348955000000001 2023-11-05T08:34:05.348 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.348 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.348 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:34:05.349 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:53.741956+0000", 2023-11-05T08:34:05.349 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00011527399999999999 2023-11-05T08:34:05.349 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.349 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.349 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:05.350 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:53.742082+0000", 2023-11-05T08:34:05.350 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00012601299999999999 2023-11-05T08:34:05.350 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.350 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:05.350 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.351 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.351 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:05.351 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:34:05.353 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2023-11-05T08:34:05.354 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:34:05.347+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:34:05.453 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:34:05.449+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:34:05.522 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:34:05.522 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [], 2023-11-05T08:34:05.522 INFO:teuthology.orchestra.run.smithi002.stdout: "num_ops": 0 2023-11-05T08:34:05.522 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:34:05.536 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2023-11-05T08:34:05.555 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:34:05.552+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:34:05.656 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:34:05.651+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:34:05.705 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:34:05.705 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [], 2023-11-05T08:34:05.706 INFO:teuthology.orchestra.run.smithi002.stdout: "complaint_time": 30, 2023-11-05T08:34:05.706 INFO:teuthology.orchestra.run.smithi002.stdout: "num_blocked_ops": 0 2023-11-05T08:34:05.706 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:34:05.721 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2023-11-05T08:34:05.768 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:34:05.765+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:34:05.846 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:34:05.846 INFO:teuthology.orchestra.run.smithi002.stdout: "size": 20, 2023-11-05T08:34:05.846 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 600, 2023-11-05T08:34:05.846 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [ 2023-11-05T08:34:05.847 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.847 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:278 3.14s0 3:292b6c2e:::smithi09039252-40:head [copy-from ver 10] snapc 8=[8,7,6,5,3] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:34:05.847 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:53.143346+0000", 2023-11-05T08:34:05.847 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 72.699440555999999, 2023-11-05T08:34:05.847 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0524983919999999, 2023-11-05T08:34:05.848 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:05.848 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:05.849 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:34:05.849 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:34:05.849 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:05.849 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 278 2023-11-05T08:34:05.850 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.850 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:05.850 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.850 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:05.851 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143346+0000", 2023-11-05T08:34:05.851 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.851 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.851 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.852 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:05.852 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143346+0000", 2023-11-05T08:34:05.852 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.852 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.853 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.853 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:05.853 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143347+0000", 2023-11-05T08:34:05.853 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4470000000000001e-06 2023-11-05T08:34:05.853 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.854 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.854 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:05.854 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143353+0000", 2023-11-05T08:34:05.854 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.7790000000000001e-06 2023-11-05T08:34:05.855 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.855 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.855 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:05.855 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143367+0000", 2023-11-05T08:34:05.855 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4321000000000001e-05 2023-11-05T08:34:05.856 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.856 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.856 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:05.856 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143374+0000", 2023-11-05T08:34:05.857 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.8029999999999999e-06 2023-11-05T08:34:05.857 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.857 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.857 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.858 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143446+0000", 2023-11-05T08:34:05.858 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.2080000000000001e-05 2023-11-05T08:34:05.858 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.858 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.859 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for peered", 2023-11-05T08:34:05.859 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143450+0000", 2023-11-05T08:34:05.859 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.5449999999999999e-06 2023-11-05T08:34:05.859 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.860 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.860 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.860 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.169316+0000", 2023-11-05T08:34:05.860 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.025866494 2023-11-05T08:34:05.861 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.861 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.861 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:34:05.861 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.169711+0000", 2023-11-05T08:34:05.862 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00039465700000000003 2023-11-05T08:34:05.862 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.862 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.862 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:05.863 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.185825+0000", 2023-11-05T08:34:05.863 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.016113624 2023-11-05T08:34:05.863 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.863 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.864 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:05.864 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.190579+0000", 2023-11-05T08:34:05.864 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0047547370000000002 2023-11-05T08:34:05.864 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.865 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.865 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:34:05.865 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.195484+0000", 2023-11-05T08:34:05.865 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0049050200000000004 2023-11-05T08:34:05.865 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.866 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.866 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:05.866 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.195844+0000", 2023-11-05T08:34:05.866 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000359885 2023-11-05T08:34:05.866 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.867 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:05.867 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.867 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.867 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.868 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:279 3.14s0 3:292b6c2e:::smithi09039252-40:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2023-11-05T08:34:05.868 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:53.143390+0000", 2023-11-05T08:34:05.868 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 72.699396480000004, 2023-11-05T08:34:05.868 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0469768799999999, 2023-11-05T08:34:05.869 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:05.869 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "started", 2023-11-05T08:34:05.869 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:34:05.869 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:34:05.870 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:05.870 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 279 2023-11-05T08:34:05.870 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.870 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:05.870 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.871 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:05.871 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143390+0000", 2023-11-05T08:34:05.871 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.871 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.872 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.872 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:05.872 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143390+0000", 2023-11-05T08:34:05.872 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.872 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.873 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.873 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:05.873 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143390+0000", 2023-11-05T08:34:05.873 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.5999999999999999e-07 2023-11-05T08:34:05.873 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.874 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.874 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:05.874 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143392+0000", 2023-11-05T08:34:05.874 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.0209999999999998e-06 2023-11-05T08:34:05.875 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.875 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.875 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:05.875 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143399+0000", 2023-11-05T08:34:05.875 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.6869999999999997e-06 2023-11-05T08:34:05.876 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.876 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.876 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:05.876 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143405+0000", 2023-11-05T08:34:05.876 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.0619999999999997e-06 2023-11-05T08:34:05.877 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.877 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.877 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.877 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143483+0000", 2023-11-05T08:34:05.878 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.8268000000000003e-05 2023-11-05T08:34:05.878 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.878 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.878 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for peered", 2023-11-05T08:34:05.879 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:53.143486+0000", 2023-11-05T08:34:05.879 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.322e-06 2023-11-05T08:34:05.879 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.879 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.879 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.880 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.172078+0000", 2023-11-05T08:34:05.880 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0285927589999999 2023-11-05T08:34:05.880 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.880 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.881 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for blocked object", 2023-11-05T08:34:05.881 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.172240+0000", 2023-11-05T08:34:05.881 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00016178899999999999 2023-11-05T08:34:05.881 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.881 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.882 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.882 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.189404+0000", 2023-11-05T08:34:05.882 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.017164220000000001 2023-11-05T08:34:05.882 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.883 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.883 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:34:05.883 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.189710+0000", 2023-11-05T08:34:05.883 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000305617 2023-11-05T08:34:05.883 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.884 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.884 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:05.884 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.190367+0000", 2023-11-05T08:34:05.884 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00065677499999999996 2023-11-05T08:34:05.884 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.885 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:05.885 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.885 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.885 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.886 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es4 41/30 ECSubWrite(tid=40, reqid=osd.2.0:40, at_version=41'27, trim_to=0'0, roll_forward_to=40'25))", 2023-11-05T08:34:05.886 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:54.120687+0000", 2023-11-05T08:34:05.886 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 71.722099696000001, 2023-11-05T08:34:05.886 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0097705859999999995, 2023-11-05T08:34:05.886 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:05.887 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:34:05.887 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:05.887 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.887 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:05.887 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120687+0000", 2023-11-05T08:34:05.888 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.888 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.888 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.888 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:05.888 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120687+0000", 2023-11-05T08:34:05.889 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.889 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.889 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.889 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:05.889 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120688+0000", 2023-11-05T08:34:05.890 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.64e-06 2023-11-05T08:34:05.890 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.890 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.890 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:05.891 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120741+0000", 2023-11-05T08:34:05.891 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.2274000000000001e-05 2023-11-05T08:34:05.891 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.891 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.892 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:05.892 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120770+0000", 2023-11-05T08:34:05.892 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.9768999999999999e-05 2023-11-05T08:34:05.892 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.892 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.892 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:05.893 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120795+0000", 2023-11-05T08:34:05.893 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.4805000000000001e-05 2023-11-05T08:34:05.893 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.893 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.894 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.894 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.124189+0000", 2023-11-05T08:34:05.894 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0033935279999999998 2023-11-05T08:34:05.894 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.895 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.895 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:05.895 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.124214+0000", 2023-11-05T08:34:05.895 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.5823e-05 2023-11-05T08:34:05.895 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.896 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.896 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:05.896 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.130330+0000", 2023-11-05T08:34:05.896 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0061156190000000001 2023-11-05T08:34:05.896 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.897 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.897 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:05.897 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.130457+0000", 2023-11-05T08:34:05.897 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00012712800000000001 2023-11-05T08:34:05.897 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.898 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:05.898 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.898 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.898 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.899 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDPGPushReply(3.14s0 47/46 [PushReplyOp(3:292b6c2e:::smithi09039252-40:head)])", 2023-11-05T08:34:05.899 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:00.292862+0000", 2023-11-05T08:34:05.899 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 65.549924789000002, 2023-11-05T08:34:05.899 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0063631419999999996, 2023-11-05T08:34:05.900 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:05.900 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:34:05.900 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:05.900 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.900 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:05.901 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.292862+0000", 2023-11-05T08:34:05.901 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.901 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.901 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.901 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:05.902 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.292862+0000", 2023-11-05T08:34:05.902 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.902 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.902 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.903 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:05.903 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.292862+0000", 2023-11-05T08:34:05.903 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.0200000000000001e-07 2023-11-05T08:34:05.903 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.903 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.904 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:05.904 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.292867+0000", 2023-11-05T08:34:05.904 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.7829999999999998e-06 2023-11-05T08:34:05.904 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.904 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.905 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:05.905 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.292877+0000", 2023-11-05T08:34:05.905 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0312000000000001e-05 2023-11-05T08:34:05.905 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.905 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.906 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:05.906 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.292889+0000", 2023-11-05T08:34:05.906 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.2019e-05 2023-11-05T08:34:05.906 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.907 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.907 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.907 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.293399+0000", 2023-11-05T08:34:05.907 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00050963099999999997 2023-11-05T08:34:05.907 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.908 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.908 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:05.908 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:00.299225+0000", 2023-11-05T08:34:05.908 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0058256949999999997 2023-11-05T08:34:05.909 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.909 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:05.909 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.909 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.909 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.910 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.16s3 56/45 ECSubWrite(tid=63, reqid=client.4691.0:369, at_version=56'20, trim_to=0'0, roll_forward_to=51'19))", 2023-11-05T08:34:05.910 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:09.293412+0000", 2023-11-05T08:34:05.910 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 56.549373893999999, 2023-11-05T08:34:05.910 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0049197820000000001, 2023-11-05T08:34:05.910 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:05.911 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:34:05.911 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:05.911 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.911 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:05.912 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.293412+0000", 2023-11-05T08:34:05.912 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.912 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.912 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.912 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:05.913 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.293412+0000", 2023-11-05T08:34:05.913 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.913 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.913 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.913 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:05.914 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.293413+0000", 2023-11-05T08:34:05.914 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.0299999999999998e-07 2023-11-05T08:34:05.914 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.914 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.915 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:05.915 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.293961+0000", 2023-11-05T08:34:05.915 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00054824799999999999 2023-11-05T08:34:05.915 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.915 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.916 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:05.916 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.293997+0000", 2023-11-05T08:34:05.916 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.5991999999999997e-05 2023-11-05T08:34:05.916 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.916 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.917 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:05.917 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.294022+0000", 2023-11-05T08:34:05.917 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.4865999999999999e-05 2023-11-05T08:34:05.917 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.918 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.918 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.918 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.294234+0000", 2023-11-05T08:34:05.918 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00021178399999999999 2023-11-05T08:34:05.918 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.919 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.919 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:05.919 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.294264+0000", 2023-11-05T08:34:05.919 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.0309000000000001e-05 2023-11-05T08:34:05.919 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.920 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.920 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:05.920 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.298238+0000", 2023-11-05T08:34:05.920 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0039737959999999999 2023-11-05T08:34:05.920 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.921 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.921 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:05.921 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.298332+0000", 2023-11-05T08:34:05.921 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.3984e-05 2023-11-05T08:34:05.921 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.922 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:05.922 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.922 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.922 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.923 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:374 3.14s0 3:292b6c2e:::smithi09039252-40:head [copy-from ver 2] snapc 15=[7] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2023-11-05T08:34:05.923 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:09.297419+0000", 2023-11-05T08:34:05.923 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 56.545367145999997, 2023-11-05T08:34:05.923 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.03004254, 2023-11-05T08:34:05.923 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:05.924 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:05.924 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:34:05.924 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:34:05.924 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:05.924 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 374 2023-11-05T08:34:05.925 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.925 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:05.925 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.925 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:05.925 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297419+0000", 2023-11-05T08:34:05.926 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.926 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.926 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.926 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:05.927 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297419+0000", 2023-11-05T08:34:05.927 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.927 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.927 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.927 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:05.928 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297422+0000", 2023-11-05T08:34:05.928 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.717e-06 2023-11-05T08:34:05.928 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.928 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.928 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:05.929 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297437+0000", 2023-11-05T08:34:05.929 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4793e-05 2023-11-05T08:34:05.929 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.929 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.930 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:05.930 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297468+0000", 2023-11-05T08:34:05.930 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.1519000000000001e-05 2023-11-05T08:34:05.930 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.930 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.931 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:05.931 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297490+0000", 2023-11-05T08:34:05.931 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.1651999999999999e-05 2023-11-05T08:34:05.931 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.931 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.932 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.932 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.300337+0000", 2023-11-05T08:34:05.932 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0028475219999999999 2023-11-05T08:34:05.932 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.932 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.933 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:34:05.933 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.300717+0000", 2023-11-05T08:34:05.933 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00037937600000000001 2023-11-05T08:34:05.933 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.934 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.934 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:05.934 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.319194+0000", 2023-11-05T08:34:05.934 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.018477303 2023-11-05T08:34:05.934 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.935 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.935 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:05.935 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.322493+0000", 2023-11-05T08:34:05.935 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0032987250000000002 2023-11-05T08:34:05.936 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.936 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.936 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:34:05.936 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.326714+0000", 2023-11-05T08:34:05.936 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0042208319999999999 2023-11-05T08:34:05.937 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.937 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.937 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:05.937 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.327462+0000", 2023-11-05T08:34:05.937 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00074810100000000004 2023-11-05T08:34:05.938 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.938 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:05.938 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.938 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.939 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.939 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:375 3.14s0 3:292b6c2e:::smithi09039252-40:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e56)", 2023-11-05T08:34:05.939 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:09.297530+0000", 2023-11-05T08:34:05.939 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 56.545256319000003, 2023-11-05T08:34:05.939 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.024433857999999999, 2023-11-05T08:34:05.940 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:05.940 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "started", 2023-11-05T08:34:05.940 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:34:05.940 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:34:05.941 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:05.941 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 375 2023-11-05T08:34:05.941 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.941 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:05.941 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.941 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:05.942 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297530+0000", 2023-11-05T08:34:05.942 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.942 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.942 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.943 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:05.943 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297530+0000", 2023-11-05T08:34:05.943 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.943 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.943 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.944 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:05.944 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297531+0000", 2023-11-05T08:34:05.944 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.48e-06 2023-11-05T08:34:05.944 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.945 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.945 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:05.945 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297540+0000", 2023-11-05T08:34:05.945 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.49e-06 2023-11-05T08:34:05.945 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.946 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.946 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:05.946 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297566+0000", 2023-11-05T08:34:05.946 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.6152e-05 2023-11-05T08:34:05.947 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.947 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.947 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:05.947 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.297583+0000", 2023-11-05T08:34:05.948 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.6676000000000001e-05 2023-11-05T08:34:05.948 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.948 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.948 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.948 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.301232+0000", 2023-11-05T08:34:05.949 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.003649209 2023-11-05T08:34:05.949 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.949 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.949 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for blocked object", 2023-11-05T08:34:05.949 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.301367+0000", 2023-11-05T08:34:05.950 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00013454600000000001 2023-11-05T08:34:05.950 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.950 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.950 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.951 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.321294+0000", 2023-11-05T08:34:05.951 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.019927261000000002 2023-11-05T08:34:05.951 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.951 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.952 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:34:05.952 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.321567+0000", 2023-11-05T08:34:05.952 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00027297299999999998 2023-11-05T08:34:05.952 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.952 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.953 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:05.953 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:09.321964+0000", 2023-11-05T08:34:05.953 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00039707099999999998 2023-11-05T08:34:05.953 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.954 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:05.954 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.954 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.954 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.954 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.10s4 62/45 ECSubWrite(tid=219, reqid=client.4691.0:424, at_version=62'11, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:34:05.955 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:15.391016+0000", 2023-11-05T08:34:05.955 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 50.451770160000002, 2023-11-05T08:34:05.955 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0049263429999999997, 2023-11-05T08:34:05.955 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:05.955 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:34:05.956 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:05.956 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.956 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:05.956 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.391016+0000", 2023-11-05T08:34:05.957 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.957 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.957 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.957 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:05.958 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.391016+0000", 2023-11-05T08:34:05.958 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.958 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.958 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.958 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:05.959 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.391017+0000", 2023-11-05T08:34:05.959 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.7e-07 2023-11-05T08:34:05.959 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.959 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.960 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:05.960 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.391866+0000", 2023-11-05T08:34:05.960 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00084949500000000002 2023-11-05T08:34:05.960 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.960 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.961 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:05.961 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.391906+0000", 2023-11-05T08:34:05.961 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.0302000000000001e-05 2023-11-05T08:34:05.961 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.962 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.962 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:05.962 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.391932+0000", 2023-11-05T08:34:05.962 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.5439000000000002e-05 2023-11-05T08:34:05.963 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.963 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.963 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.963 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.392053+0000", 2023-11-05T08:34:05.963 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00012140400000000001 2023-11-05T08:34:05.964 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.964 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.964 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:05.964 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.392067+0000", 2023-11-05T08:34:05.964 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4051e-05 2023-11-05T08:34:05.965 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.965 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.965 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:05.965 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.395819+0000", 2023-11-05T08:34:05.966 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0037515489999999999 2023-11-05T08:34:05.966 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.966 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.966 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:05.966 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.395943+0000", 2023-11-05T08:34:05.967 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000123733 2023-11-05T08:34:05.967 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.967 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:05.967 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.968 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.968 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.968 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.fs4 77/37 ECSubWrite(tid=130, reqid=client.4691.0:579, at_version=77'44, trim_to=0'0, roll_forward_to=70'42))", 2023-11-05T08:34:05.968 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:30.540343+0000", 2023-11-05T08:34:05.968 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 35.302443507, 2023-11-05T08:34:05.969 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0057579060000000001, 2023-11-05T08:34:05.969 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:05.969 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:34:05.969 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:05.969 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.970 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:05.970 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:30.540343+0000", 2023-11-05T08:34:05.970 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.970 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.971 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.971 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:05.971 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:30.540343+0000", 2023-11-05T08:34:05.971 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.971 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.972 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.972 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:05.972 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:30.540344+0000", 2023-11-05T08:34:05.972 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.5999999999999991e-07 2023-11-05T08:34:05.973 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.973 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.973 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:05.973 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:30.540957+0000", 2023-11-05T08:34:05.973 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00061291099999999995 2023-11-05T08:34:05.974 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.974 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.974 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:05.974 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:30.540985+0000", 2023-11-05T08:34:05.975 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.7999000000000001e-05 2023-11-05T08:34:05.975 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.975 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.975 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:05.976 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:30.541002+0000", 2023-11-05T08:34:05.976 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.7105999999999999e-05 2023-11-05T08:34:05.976 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.976 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.976 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.977 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:30.541176+0000", 2023-11-05T08:34:05.977 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00017453100000000001 2023-11-05T08:34:05.977 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.977 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.978 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:05.978 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:30.541198+0000", 2023-11-05T08:34:05.978 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.1773e-05 2023-11-05T08:34:05.978 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.978 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.979 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:05.979 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:30.546033+0000", 2023-11-05T08:34:05.979 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.004835092 2023-11-05T08:34:05.979 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.979 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.980 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:05.980 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:30.546101+0000", 2023-11-05T08:34:05.980 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.7533999999999994e-05 2023-11-05T08:34:05.980 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.981 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:05.981 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.981 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.981 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.981 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:635 3.19s0 3:99e86b02:::smithi09039252-36:head [copy-from ver 19] snapc 34=[34,2f,7] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2023-11-05T08:34:05.982 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:39.599051+0000", 2023-11-05T08:34:05.982 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 26.243735096000002, 2023-11-05T08:34:05.982 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.030549215000000001, 2023-11-05T08:34:05.982 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:05.982 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:05.983 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:34:05.983 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:34:05.983 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:05.983 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 635 2023-11-05T08:34:05.984 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.984 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:05.984 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.984 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:05.984 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:39.599051+0000", 2023-11-05T08:34:05.985 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.985 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.985 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.985 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:05.985 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:39.599051+0000", 2023-11-05T08:34:05.986 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:05.986 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.986 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.986 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:05.986 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:39.599054+0000", 2023-11-05T08:34:05.987 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.5749999999999999e-06 2023-11-05T08:34:05.987 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.987 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.987 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:05.987 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:39.599065+0000", 2023-11-05T08:34:05.988 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0978999999999999e-05 2023-11-05T08:34:05.988 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.988 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.988 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:05.988 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:39.599090+0000", 2023-11-05T08:34:05.989 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.4828e-05 2023-11-05T08:34:05.989 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.989 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.989 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:05.989 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:39.599101+0000", 2023-11-05T08:34:05.990 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.1639e-05 2023-11-05T08:34:05.990 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.990 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.990 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.990 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:39.599289+0000", 2023-11-05T08:34:05.991 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00018743000000000001 2023-11-05T08:34:05.991 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.991 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.991 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "op must wait for map", 2023-11-05T08:34:05.991 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:39.599308+0000", 2023-11-05T08:34:05.992 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.9296e-05 2023-11-05T08:34:05.992 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.992 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.992 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:05.993 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:39.603971+0000", 2023-11-05T08:34:05.993 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0046632130000000003 2023-11-05T08:34:05.993 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.993 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.993 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:34:05.994 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:39.604418+0000", 2023-11-05T08:34:05.994 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00044662899999999998 2023-11-05T08:34:05.994 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.994 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.995 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:05.995 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:39.620735+0000", 2023-11-05T08:34:05.995 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.016317017 2023-11-05T08:34:05.995 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.995 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.996 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:05.996 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:39.626257+0000", 2023-11-05T08:34:05.996 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0055224410000000003 2023-11-05T08:34:05.996 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.996 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.997 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:34:05.997 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:39.629516+0000", 2023-11-05T08:34:05.997 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0032589260000000001 2023-11-05T08:34:05.997 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.997 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.998 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:05.998 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:39.629600+0000", 2023-11-05T08:34:05.998 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.4241999999999997e-05 2023-11-05T08:34:05.998 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.998 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:05.999 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:05.999 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:05.999 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:05.999 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:636 3.19s0 3:99e86b02:::smithi09039252-36:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e86)", 2023-11-05T08:34:06.000 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:39.599126+0000", 2023-11-05T08:34:06.000 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 26.243659956999998, 2023-11-05T08:34:06.000 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.026471305000000001, 2023-11-05T08:34:06.000 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:06.001 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "started", 2023-11-05T08:34:06.001 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:34:06.001 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:34:06.001 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:06.001 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 636 2023-11-05T08:34:06.002 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.002 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:06.002 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.002 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:06.002 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:39.599126+0000", 2023-11-05T08:34:06.003 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.003 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.003 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.003 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:06.004 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:39.599126+0000", 2023-11-05T08:34:06.004 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.004 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.004 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.005 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:06.005 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:39.599127+0000", 2023-11-05T08:34:06.005 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.5899999999999996e-07 2023-11-05T08:34:06.005 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.005 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.006 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:06.006 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:39.599130+0000", 2023-11-05T08:34:06.006 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.942e-06 2023-11-05T08:34:06.006 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.007 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.007 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:06.007 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:39.599138+0000", 2023-11-05T08:34:06.007 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.8220000000000007e-06 2023-11-05T08:34:06.008 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.008 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.008 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:06.008 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:39.599144+0000", 2023-11-05T08:34:06.008 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.8130000000000001e-06 2023-11-05T08:34:06.009 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.009 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.009 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:06.009 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:39.599423+0000", 2023-11-05T08:34:06.010 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000279416 2023-11-05T08:34:06.010 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.010 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.010 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:34:06.011 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:39.599450+0000", 2023-11-05T08:34:06.011 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.6656e-05 2023-11-05T08:34:06.011 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.011 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.011 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:06.012 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:39.604981+0000", 2023-11-05T08:34:06.012 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0055315640000000001 2023-11-05T08:34:06.012 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.012 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.013 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for blocked object", 2023-11-05T08:34:06.013 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:39.605146+0000", 2023-11-05T08:34:06.013 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000165129 2023-11-05T08:34:06.013 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.014 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.014 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:06.014 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:39.625056+0000", 2023-11-05T08:34:06.014 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.019909894000000001 2023-11-05T08:34:06.014 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.015 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.015 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:34:06.015 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:39.625293+0000", 2023-11-05T08:34:06.015 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00023645400000000001 2023-11-05T08:34:06.015 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.016 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.016 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:06.016 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:39.625598+0000", 2023-11-05T08:34:06.016 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000305056 2023-11-05T08:34:06.016 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.017 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:06.017 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.017 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.017 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.018 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.10s4 87/45 ECSubWrite(tid=291, reqid=client.4691.0:638, at_version=87'49, trim_to=0'0, roll_forward_to=84'47))", 2023-11-05T08:34:06.018 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:40.616462+0000", 2023-11-05T08:34:06.018 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 25.226324814000002, 2023-11-05T08:34:06.018 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0044967710000000001, 2023-11-05T08:34:06.018 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:06.019 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:34:06.019 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:06.019 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.019 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:06.020 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:40.616462+0000", 2023-11-05T08:34:06.020 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.020 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.020 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.020 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:06.021 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:40.616462+0000", 2023-11-05T08:34:06.021 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.021 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.021 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.021 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:06.021 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:40.616462+0000", 2023-11-05T08:34:06.022 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.03e-07 2023-11-05T08:34:06.022 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.022 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.022 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:06.022 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:40.616818+0000", 2023-11-05T08:34:06.023 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000356308 2023-11-05T08:34:06.023 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.023 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.023 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:06.023 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:40.616856+0000", 2023-11-05T08:34:06.023 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.7592000000000002e-05 2023-11-05T08:34:06.024 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.024 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.024 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:06.024 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:40.616888+0000", 2023-11-05T08:34:06.024 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.2246000000000003e-05 2023-11-05T08:34:06.025 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.025 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.025 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:06.025 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:40.617065+0000", 2023-11-05T08:34:06.025 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00017741600000000001 2023-11-05T08:34:06.026 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.026 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.026 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:06.026 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:40.617104+0000", 2023-11-05T08:34:06.026 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.8868999999999997e-05 2023-11-05T08:34:06.027 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.027 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.027 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:06.027 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:40.620851+0000", 2023-11-05T08:34:06.027 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.003747106 2023-11-05T08:34:06.027 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.028 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.028 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:06.028 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:40.620958+0000", 2023-11-05T08:34:06.028 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00010683099999999999 2023-11-05T08:34:06.028 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.029 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:06.029 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.029 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.029 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.029 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.9s5 92/45 ECSubWrite(tid=98, reqid=client.4691.0:662, at_version=92'38, trim_to=0'0, roll_forward_to=68'37))", 2023-11-05T08:34:06.030 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:45.750062+0000", 2023-11-05T08:34:06.030 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 20.092724676, 2023-11-05T08:34:06.030 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0044797209999999999, 2023-11-05T08:34:06.030 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:06.030 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:34:06.030 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:06.031 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.031 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:06.031 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.750062+0000", 2023-11-05T08:34:06.031 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.031 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.032 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.032 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:06.032 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.750062+0000", 2023-11-05T08:34:06.032 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.032 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.032 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.033 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:06.033 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.750062+0000", 2023-11-05T08:34:06.033 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.2699999999999998e-07 2023-11-05T08:34:06.033 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.033 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.034 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:06.034 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.750635+0000", 2023-11-05T08:34:06.034 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00057248399999999995 2023-11-05T08:34:06.034 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.034 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.035 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:06.035 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.750666+0000", 2023-11-05T08:34:06.035 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.0874999999999998e-05 2023-11-05T08:34:06.035 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.035 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.035 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:06.036 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.750697+0000", 2023-11-05T08:34:06.036 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.0769000000000003e-05 2023-11-05T08:34:06.036 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.036 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.036 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:06.037 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.750871+0000", 2023-11-05T08:34:06.037 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00017414000000000001 2023-11-05T08:34:06.037 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.037 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.037 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:06.037 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.750902+0000", 2023-11-05T08:34:06.038 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.1359000000000002e-05 2023-11-05T08:34:06.038 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.038 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.038 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:06.038 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.754463+0000", 2023-11-05T08:34:06.039 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0035613659999999998 2023-11-05T08:34:06.039 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.039 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.039 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:06.039 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.754541+0000", 2023-11-05T08:34:06.039 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.7900999999999998e-05 2023-11-05T08:34:06.040 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.040 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:06.040 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.040 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.040 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.041 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:667 3.19s0 3:9ed96077:::smithi09039252-44:head [read 0~737280,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2023-11-05T08:34:06.041 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:45.760714+0000", 2023-11-05T08:34:06.041 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 20.082072158999999, 2023-11-05T08:34:06.041 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0047452789999999998, 2023-11-05T08:34:06.041 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:06.042 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "started", 2023-11-05T08:34:06.042 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:34:06.042 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:34:06.042 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:06.042 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 667 2023-11-05T08:34:06.042 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.043 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:06.043 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.043 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:06.043 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.760714+0000", 2023-11-05T08:34:06.043 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.043 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.044 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.044 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:06.044 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.760714+0000", 2023-11-05T08:34:06.044 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.044 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.044 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.045 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:06.045 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.760716+0000", 2023-11-05T08:34:06.045 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.317e-06 2023-11-05T08:34:06.045 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.045 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.046 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:06.046 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.760727+0000", 2023-11-05T08:34:06.046 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.078e-05 2023-11-05T08:34:06.046 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.046 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.047 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:06.047 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.760758+0000", 2023-11-05T08:34:06.047 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.1041999999999999e-05 2023-11-05T08:34:06.047 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.047 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.047 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:06.048 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.760773+0000", 2023-11-05T08:34:06.048 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.5133000000000001e-05 2023-11-05T08:34:06.048 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.048 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.048 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:06.049 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.760912+0000", 2023-11-05T08:34:06.049 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00013814 2023-11-05T08:34:06.049 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.049 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.049 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:34:06.050 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.761154+0000", 2023-11-05T08:34:06.050 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000241962 2023-11-05T08:34:06.050 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.050 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.050 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:06.050 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.765459+0000", 2023-11-05T08:34:06.051 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0043059049999999996 2023-11-05T08:34:06.051 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.051 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:06.051 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.052 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.052 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.052 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:668 3.19s0 3:9ed96077:::smithi09039252-44:head [read 0~737280,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2023-11-05T08:34:06.052 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:45.760807+0000", 2023-11-05T08:34:06.052 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 20.081978930999998, 2023-11-05T08:34:06.053 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0054248059999999999, 2023-11-05T08:34:06.053 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:06.053 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "started", 2023-11-05T08:34:06.053 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:34:06.053 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:34:06.054 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:06.054 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 668 2023-11-05T08:34:06.054 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.054 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:06.054 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.054 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:06.055 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.760807+0000", 2023-11-05T08:34:06.055 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.055 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.055 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.055 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:06.056 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.760807+0000", 2023-11-05T08:34:06.056 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.056 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.056 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.056 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:06.057 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.760808+0000", 2023-11-05T08:34:06.057 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.7000000000000003e-07 2023-11-05T08:34:06.057 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.057 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.057 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:06.058 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.760814+0000", 2023-11-05T08:34:06.058 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.48e-06 2023-11-05T08:34:06.058 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.058 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.058 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:06.058 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.760829+0000", 2023-11-05T08:34:06.059 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.5043e-05 2023-11-05T08:34:06.059 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.059 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.059 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:06.059 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.760838+0000", 2023-11-05T08:34:06.060 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.9099999999999994e-06 2023-11-05T08:34:06.060 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.060 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.060 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:06.060 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.761780+0000", 2023-11-05T08:34:06.061 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00094224800000000002 2023-11-05T08:34:06.061 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.061 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.061 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:34:06.061 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.761998+0000", 2023-11-05T08:34:06.061 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00021775200000000001 2023-11-05T08:34:06.062 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.062 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.062 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:06.062 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.766232+0000", 2023-11-05T08:34:06.062 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0042344030000000003 2023-11-05T08:34:06.063 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.063 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:06.063 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.063 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.063 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.063 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:669 3.19s0 3:9ed96077:::smithi09039252-44:head [read 0~737280,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2023-11-05T08:34:06.064 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:45.760923+0000", 2023-11-05T08:34:06.064 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 20.08186311, 2023-11-05T08:34:06.064 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0089358800000000002, 2023-11-05T08:34:06.064 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:06.064 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "started", 2023-11-05T08:34:06.065 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:34:06.065 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:34:06.065 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:06.065 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 669 2023-11-05T08:34:06.065 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.065 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:06.066 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.066 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:06.066 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.760923+0000", 2023-11-05T08:34:06.066 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.066 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.067 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.067 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:06.067 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.760923+0000", 2023-11-05T08:34:06.067 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.067 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.067 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.068 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:06.068 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.760924+0000", 2023-11-05T08:34:06.068 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.9599999999999998e-07 2023-11-05T08:34:06.068 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.068 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.069 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:06.069 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.760929+0000", 2023-11-05T08:34:06.069 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.7369999999999997e-06 2023-11-05T08:34:06.069 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.069 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.070 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:06.070 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.760941+0000", 2023-11-05T08:34:06.070 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.2077e-05 2023-11-05T08:34:06.070 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.070 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.071 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:06.071 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.760949+0000", 2023-11-05T08:34:06.071 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.2090000000000005e-06 2023-11-05T08:34:06.071 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.071 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.071 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:06.072 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.766311+0000", 2023-11-05T08:34:06.072 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0053615640000000001 2023-11-05T08:34:06.072 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.072 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.072 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:34:06.073 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.766489+0000", 2023-11-05T08:34:06.073 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00017866299999999999 2023-11-05T08:34:06.073 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.073 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.074 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:06.074 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.769859+0000", 2023-11-05T08:34:06.074 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0033698339999999999 2023-11-05T08:34:06.074 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.074 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:06.074 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.075 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.075 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.075 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.16s3 92/45 ECSubWrite(tid=164, reqid=client.4691.0:677, at_version=92'45, trim_to=0'0, roll_forward_to=92'43))", 2023-11-05T08:34:06.075 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:45.824808+0000", 2023-11-05T08:34:06.075 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 20.017978715000002, 2023-11-05T08:34:06.076 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.005200545, 2023-11-05T08:34:06.076 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:06.076 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:34:06.076 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:06.076 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.077 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:06.077 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.824808+0000", 2023-11-05T08:34:06.077 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.077 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.077 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.077 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:06.078 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.824808+0000", 2023-11-05T08:34:06.078 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.078 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.078 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.078 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:06.078 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.824810+0000", 2023-11-05T08:34:06.079 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.1299999999999999e-06 2023-11-05T08:34:06.079 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.079 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.079 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:06.079 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.825254+0000", 2023-11-05T08:34:06.080 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00044466599999999997 2023-11-05T08:34:06.080 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.080 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.080 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:06.080 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.825294+0000", 2023-11-05T08:34:06.080 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.9626000000000003e-05 2023-11-05T08:34:06.081 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.081 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.081 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:06.081 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.825318+0000", 2023-11-05T08:34:06.081 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.3842999999999999e-05 2023-11-05T08:34:06.082 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.082 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.082 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:06.082 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.825507+0000", 2023-11-05T08:34:06.082 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00018880599999999999 2023-11-05T08:34:06.083 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.083 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.083 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:06.083 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.825537+0000", 2023-11-05T08:34:06.083 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.0278999999999999e-05 2023-11-05T08:34:06.083 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.084 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.084 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:06.084 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.829932+0000", 2023-11-05T08:34:06.084 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0043953799999999999 2023-11-05T08:34:06.084 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.085 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.085 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:06.085 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.830008+0000", 2023-11-05T08:34:06.085 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.5815000000000005e-05 2023-11-05T08:34:06.085 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.085 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:06.086 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.086 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.086 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.086 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.10s4 97/45 ECSubWrite(tid=320, reqid=osd.13.0:320, at_version=97'60, trim_to=0'0, roll_forward_to=93'56))", 2023-11-05T08:34:06.086 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:50.667337+0000", 2023-11-05T08:34:06.087 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 15.17544919, 2023-11-05T08:34:06.087 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0065340420000000003, 2023-11-05T08:34:06.087 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:06.087 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:34:06.087 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:06.088 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.088 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:06.088 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:50.667337+0000", 2023-11-05T08:34:06.088 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.088 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.088 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.089 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:06.089 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:50.667337+0000", 2023-11-05T08:34:06.089 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.089 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.089 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.090 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:06.090 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:50.667338+0000", 2023-11-05T08:34:06.090 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.1900000000000002e-07 2023-11-05T08:34:06.090 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.090 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.090 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:06.091 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:50.667362+0000", 2023-11-05T08:34:06.091 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.3853e-05 2023-11-05T08:34:06.091 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.091 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.091 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:06.092 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:50.667372+0000", 2023-11-05T08:34:06.092 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.7880000000000001e-06 2023-11-05T08:34:06.092 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.092 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.092 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:06.093 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:50.667379+0000", 2023-11-05T08:34:06.093 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.2699999999999999e-06 2023-11-05T08:34:06.093 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.093 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.093 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:06.093 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:50.667446+0000", 2023-11-05T08:34:06.094 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.7234e-05 2023-11-05T08:34:06.094 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.094 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.094 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:06.094 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:50.667454+0000", 2023-11-05T08:34:06.095 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.0029999999999994e-06 2023-11-05T08:34:06.095 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.095 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.095 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:06.095 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:50.673831+0000", 2023-11-05T08:34:06.096 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0063773600000000003 2023-11-05T08:34:06.096 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.096 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.096 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:06.096 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:50.673871+0000", 2023-11-05T08:34:06.096 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.9814999999999997e-05 2023-11-05T08:34:06.097 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.097 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:06.097 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.097 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.097 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.097 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:802 3.19s0 3:99e86b02:::smithi09039252-36:head [copy-from ver 30] snapc 44=[44,42,41] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2023-11-05T08:34:06.098 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:55.746690+0000", 2023-11-05T08:34:06.098 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 10.096096341000001, 2023-11-05T08:34:06.098 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.027667417999999999, 2023-11-05T08:34:06.098 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:06.098 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:06.099 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:34:06.099 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:34:06.099 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:06.099 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 802 2023-11-05T08:34:06.099 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.099 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:06.100 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.100 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:06.100 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:55.746690+0000", 2023-11-05T08:34:06.100 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.100 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.100 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.101 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:06.101 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:55.746690+0000", 2023-11-05T08:34:06.101 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.101 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.101 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.102 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:06.102 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:55.746692+0000", 2023-11-05T08:34:06.102 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.2239999999999998e-06 2023-11-05T08:34:06.102 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.102 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.102 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:06.103 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:55.746707+0000", 2023-11-05T08:34:06.103 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4656e-05 2023-11-05T08:34:06.103 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.103 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.103 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:06.104 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:55.746736+0000", 2023-11-05T08:34:06.104 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.9374999999999999e-05 2023-11-05T08:34:06.104 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.104 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.104 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:06.104 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:55.746757+0000", 2023-11-05T08:34:06.105 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.086e-05 2023-11-05T08:34:06.105 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.105 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.105 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:06.105 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:55.746856+0000", 2023-11-05T08:34:06.105 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.8445999999999999e-05 2023-11-05T08:34:06.106 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.106 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.106 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "op must wait for map", 2023-11-05T08:34:06.106 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:55.746870+0000", 2023-11-05T08:34:06.106 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.3991999999999999e-05 2023-11-05T08:34:06.107 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.107 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.107 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:06.107 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:55.749773+0000", 2023-11-05T08:34:06.107 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.002903297 2023-11-05T08:34:06.107 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.108 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.108 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:34:06.108 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:55.749893+0000", 2023-11-05T08:34:06.108 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000120124 2023-11-05T08:34:06.108 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.108 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.109 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:06.109 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:55.764232+0000", 2023-11-05T08:34:06.109 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.014339167999999999 2023-11-05T08:34:06.109 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.109 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.110 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:06.110 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:55.767622+0000", 2023-11-05T08:34:06.110 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0033896400000000002 2023-11-05T08:34:06.110 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.110 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.110 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", 2023-11-05T08:34:06.111 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:55.774279+0000", 2023-11-05T08:34:06.111 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0066567359999999999 2023-11-05T08:34:06.111 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.111 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.111 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:06.112 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:55.774357+0000", 2023-11-05T08:34:06.112 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.8899999999999993e-05 2023-11-05T08:34:06.112 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.112 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:06.112 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.112 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.113 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.113 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "osd_op(client.4691.0:803 3.19s0 3:99e86b02:::smithi09039252-36:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e102)", 2023-11-05T08:34:06.113 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:55.746789+0000", 2023-11-05T08:34:06.113 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 10.095997158999999, 2023-11-05T08:34:06.113 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.019680515999999999, 2023-11-05T08:34:06.113 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:06.114 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "started", 2023-11-05T08:34:06.114 INFO:teuthology.orchestra.run.smithi002.stdout: "client_info": { 2023-11-05T08:34:06.114 INFO:teuthology.orchestra.run.smithi002.stdout: "client": "client.4691", 2023-11-05T08:34:06.114 INFO:teuthology.orchestra.run.smithi002.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:06.114 INFO:teuthology.orchestra.run.smithi002.stdout: "tid": 803 2023-11-05T08:34:06.114 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.115 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:06.115 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.115 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:06.115 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:55.746789+0000", 2023-11-05T08:34:06.115 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.115 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.116 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.116 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:06.116 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:55.746789+0000", 2023-11-05T08:34:06.116 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.116 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.117 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.117 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:06.117 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:55.746790+0000", 2023-11-05T08:34:06.117 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.4e-07 2023-11-05T08:34:06.117 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.118 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.118 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:06.118 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:55.746796+0000", 2023-11-05T08:34:06.118 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.7610000000000004e-06 2023-11-05T08:34:06.118 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.118 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.119 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:06.119 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:55.746809+0000", 2023-11-05T08:34:06.119 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.3339000000000001e-05 2023-11-05T08:34:06.119 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.119 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.120 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:06.120 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:55.746825+0000", 2023-11-05T08:34:06.120 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.5447e-05 2023-11-05T08:34:06.120 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.120 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.120 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:06.121 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:55.746964+0000", 2023-11-05T08:34:06.121 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00013898299999999999 2023-11-05T08:34:06.121 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.121 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.121 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:34:06.122 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:55.746972+0000", 2023-11-05T08:34:06.122 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.7460000000000006e-06 2023-11-05T08:34:06.122 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.122 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.122 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:06.122 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:55.750067+0000", 2023-11-05T08:34:06.123 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.003094903 2023-11-05T08:34:06.123 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.123 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.123 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting for blocked object", 2023-11-05T08:34:06.123 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:55.750116+0000", 2023-11-05T08:34:06.124 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.8927000000000001e-05 2023-11-05T08:34:06.124 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.124 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.124 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:06.124 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:55.766227+0000", 2023-11-05T08:34:06.124 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.016110731999999999 2023-11-05T08:34:06.125 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.125 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.125 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "started", 2023-11-05T08:34:06.125 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:55.766347+0000", 2023-11-05T08:34:06.125 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000119955 2023-11-05T08:34:06.126 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.126 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.126 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:06.126 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:55.766470+0000", 2023-11-05T08:34:06.126 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00012278300000000001 2023-11-05T08:34:06.126 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.127 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:06.127 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.127 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.127 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:06.127 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:34:06.128 INFO:tasks.rados.rados.0.smithi090.stdout:387: done (1 left) 2023-11-05T08:34:06.129 INFO:tasks.rados.rados.0.smithi090.stdout:388: done (0 left) 2023-11-05T08:34:06.129 INFO:tasks.rados.rados.0.smithi090.stdout:389: snap_remove snap 33 2023-11-05T08:34:06.129 INFO:teuthology.orchestra.run.smithi002.stderr:set 3.9 pg_upmap mapping to [15,11,8,1,6,3,13] 2023-11-05T08:34:06.129 INFO:teuthology.orchestra.run.smithi002.stderr:noscrub is set 2023-11-05T08:34:06.130 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2023-11-05T08:34:06.132 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:34:06.127+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:34:06.229 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-11-05T08:34:06.226+0000 7f72b2c5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 42525) UID: 0 2023-11-05T08:34:06.303 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:34:06.304 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [], 2023-11-05T08:34:06.304 INFO:teuthology.orchestra.run.smithi002.stdout: "num_ops": 0 2023-11-05T08:34:06.304 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:34:06.318 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2023-11-05T08:34:06.363 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:34:06.360+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:34:06.431 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:34:06.427+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:34:06.435 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:34:06.436 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [], 2023-11-05T08:34:06.436 INFO:teuthology.orchestra.run.smithi002.stdout: "complaint_time": 30, 2023-11-05T08:34:06.436 INFO:teuthology.orchestra.run.smithi002.stdout: "num_blocked_ops": 0 2023-11-05T08:34:06.436 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:34:06.450 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2023-11-05T08:34:06.536 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:34:06.532+0000 7fdc5cd58640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:34:06.621 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-05T08:34:06.622 INFO:teuthology.orchestra.run.smithi002.stdout: "size": 20, 2023-11-05T08:34:06.622 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 600, 2023-11-05T08:34:06.622 INFO:teuthology.orchestra.run.smithi002.stdout: "ops": [ 2023-11-05T08:34:06.623 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.623 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es6 41/30 ECSubWrite(tid=40, reqid=osd.2.0:40, at_version=41'27, trim_to=0'0, roll_forward_to=40'25))", 2023-11-05T08:34:06.623 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:54.120785+0000", 2023-11-05T08:34:06.624 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 72.497801445999997, 2023-11-05T08:34:06.624 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0057023600000000001, 2023-11-05T08:34:06.624 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:06.625 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:34:06.625 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:06.625 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.625 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:06.626 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120785+0000", 2023-11-05T08:34:06.626 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.626 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.626 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.627 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:06.627 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120785+0000", 2023-11-05T08:34:06.627 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.628 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.628 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.628 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:06.629 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120786+0000", 2023-11-05T08:34:06.629 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.0999999999999998e-07 2023-11-05T08:34:06.629 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.629 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.630 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:06.630 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120850+0000", 2023-11-05T08:34:06.630 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.4727000000000006e-05 2023-11-05T08:34:06.630 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.631 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.631 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:06.631 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120899+0000", 2023-11-05T08:34:06.631 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.8905e-05 2023-11-05T08:34:06.632 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.632 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.632 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:06.632 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.120925+0000", 2023-11-05T08:34:06.632 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.5831999999999999e-05 2023-11-05T08:34:06.633 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.633 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.633 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:06.633 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.121107+0000", 2023-11-05T08:34:06.634 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00018210899999999999 2023-11-05T08:34:06.634 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.634 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.634 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:06.635 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.121136+0000", 2023-11-05T08:34:06.635 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.9116e-05 2023-11-05T08:34:06.635 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.636 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.636 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:06.636 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.126383+0000", 2023-11-05T08:34:06.637 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0052464579999999999 2023-11-05T08:34:06.637 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.637 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.637 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:06.638 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:54.126487+0000", 2023-11-05T08:34:06.638 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000104503 2023-11-05T08:34:06.638 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.638 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:06.638 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.639 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.639 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.639 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es6 42/30 ECSubWrite(tid=43, reqid=osd.2.0:43, at_version=42'29, trim_to=0'0, roll_forward_to=41'27))", 2023-11-05T08:34:06.639 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:55.137493+0000", 2023-11-05T08:34:06.639 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 71.481093522999998, 2023-11-05T08:34:06.640 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0031512160000000001, 2023-11-05T08:34:06.640 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:06.640 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:34:06.640 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:06.640 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.640 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:06.641 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.137493+0000", 2023-11-05T08:34:06.641 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.641 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.641 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.641 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:06.642 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.137493+0000", 2023-11-05T08:34:06.642 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.642 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.643 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.643 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:06.643 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.137493+0000", 2023-11-05T08:34:06.643 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.7999999999999995e-07 2023-11-05T08:34:06.643 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.644 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.644 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:06.644 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.137559+0000", 2023-11-05T08:34:06.644 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.6098000000000003e-05 2023-11-05T08:34:06.644 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.645 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.645 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:06.645 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.137594+0000", 2023-11-05T08:34:06.645 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.4082000000000003e-05 2023-11-05T08:34:06.645 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.646 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.646 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:06.646 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.137620+0000", 2023-11-05T08:34:06.646 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.6271999999999999e-05 2023-11-05T08:34:06.646 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.646 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.647 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:06.647 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.137815+0000", 2023-11-05T08:34:06.647 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000195229 2023-11-05T08:34:06.647 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.647 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.648 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:06.648 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.137848+0000", 2023-11-05T08:34:06.648 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.2611999999999999e-05 2023-11-05T08:34:06.648 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.648 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.649 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:06.649 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.140476+0000", 2023-11-05T08:34:06.649 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0026284860000000002 2023-11-05T08:34:06.649 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.649 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.650 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:06.650 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:55.140644+0000", 2023-11-05T08:34:06.650 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00016785699999999999 2023-11-05T08:34:06.650 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.650 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:06.651 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.651 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.651 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.651 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.18s6 46/45 ECSubWrite(tid=195, reqid=osd.13.0:195, at_version=46'19, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:34:06.651 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:32:59.244820+0000", 2023-11-05T08:34:06.652 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 67.373766216999996, 2023-11-05T08:34:06.652 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0041897130000000003, 2023-11-05T08:34:06.652 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:06.652 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:34:06.652 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:06.653 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.653 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:06.653 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.244820+0000", 2023-11-05T08:34:06.653 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.653 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.654 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.654 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:06.654 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.244820+0000", 2023-11-05T08:34:06.654 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.655 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.655 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.655 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:06.655 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.244829+0000", 2023-11-05T08:34:06.655 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.8559999999999996e-06 2023-11-05T08:34:06.656 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.656 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.656 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:06.656 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.244875+0000", 2023-11-05T08:34:06.657 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.5837999999999997e-05 2023-11-05T08:34:06.657 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.657 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.657 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:06.658 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.244903+0000", 2023-11-05T08:34:06.658 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.8123000000000002e-05 2023-11-05T08:34:06.658 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.658 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.659 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:06.659 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.244919+0000", 2023-11-05T08:34:06.659 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.5973000000000001e-05 2023-11-05T08:34:06.660 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.660 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.661 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:06.661 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.245039+0000", 2023-11-05T08:34:06.661 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000120289 2023-11-05T08:34:06.661 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.662 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.662 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:06.662 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.245054+0000", 2023-11-05T08:34:06.662 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.491e-05 2023-11-05T08:34:06.663 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.663 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.663 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:06.664 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.248956+0000", 2023-11-05T08:34:06.664 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0039018859999999998 2023-11-05T08:34:06.664 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.664 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.665 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:06.665 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:32:59.249010+0000", 2023-11-05T08:34:06.665 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.3838000000000002e-05 2023-11-05T08:34:06.666 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.666 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:06.666 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.666 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.667 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.667 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es6 57/30 ECSubWrite(tid=60, reqid=client.4691.0:380, at_version=57'45, trim_to=0'0, roll_forward_to=49'43))", 2023-11-05T08:34:06.667 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:10.285649+0000", 2023-11-05T08:34:06.667 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 56.332936859999997, 2023-11-05T08:34:06.668 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0029323890000000001, 2023-11-05T08:34:06.668 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:06.668 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:34:06.668 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:06.668 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.668 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:06.669 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285649+0000", 2023-11-05T08:34:06.669 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.669 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.669 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.669 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:06.670 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285649+0000", 2023-11-05T08:34:06.670 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.670 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.670 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.670 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:06.671 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285650+0000", 2023-11-05T08:34:06.671 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.4300000000000003e-07 2023-11-05T08:34:06.671 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.671 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.671 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:06.672 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285727+0000", 2023-11-05T08:34:06.672 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.7088000000000005e-05 2023-11-05T08:34:06.672 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.672 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.672 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:06.672 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285762+0000", 2023-11-05T08:34:06.673 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.4808999999999998e-05 2023-11-05T08:34:06.673 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.673 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.673 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:06.673 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285788+0000", 2023-11-05T08:34:06.673 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.6075999999999999e-05 2023-11-05T08:34:06.674 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.674 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.674 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:06.674 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285958+0000", 2023-11-05T08:34:06.674 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00016993499999999999 2023-11-05T08:34:06.675 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.675 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.675 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:06.675 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.285977+0000", 2023-11-05T08:34:06.675 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.8692999999999999e-05 2023-11-05T08:34:06.676 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.676 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.676 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:06.676 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.288484+0000", 2023-11-05T08:34:06.676 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.002507085 2023-11-05T08:34:06.677 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.677 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.677 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:06.677 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:10.288582+0000", 2023-11-05T08:34:06.677 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.8060000000000006e-05 2023-11-05T08:34:06.678 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.678 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:06.678 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.678 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.678 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.679 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.18s6 62/45 ECSubWrite(tid=217, reqid=client.4691.0:423, at_version=62'30, trim_to=0'0, roll_forward_to=52'28))", 2023-11-05T08:34:06.679 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:15.372664+0000", 2023-11-05T08:34:06.679 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 51.245922247999999, 2023-11-05T08:34:06.679 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0040353610000000003, 2023-11-05T08:34:06.679 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:06.679 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:34:06.680 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:06.680 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.680 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:06.680 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.372664+0000", 2023-11-05T08:34:06.680 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.680 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.681 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.681 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:06.681 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.372664+0000", 2023-11-05T08:34:06.681 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.681 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.681 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.682 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:06.682 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.372664+0000", 2023-11-05T08:34:06.682 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.3099999999999999e-07 2023-11-05T08:34:06.682 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.682 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.682 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:06.683 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.372840+0000", 2023-11-05T08:34:06.683 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00017561599999999999 2023-11-05T08:34:06.683 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.683 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.683 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:06.684 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.372857+0000", 2023-11-05T08:34:06.684 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.7169000000000001e-05 2023-11-05T08:34:06.684 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.684 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.684 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:06.684 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.372870+0000", 2023-11-05T08:34:06.685 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.2847000000000001e-05 2023-11-05T08:34:06.685 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.685 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.686 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:06.686 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.373057+0000", 2023-11-05T08:34:06.687 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000186742 2023-11-05T08:34:06.687 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.687 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.687 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:06.688 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.373091+0000", 2023-11-05T08:34:06.688 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.4186000000000001e-05 2023-11-05T08:34:06.689 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.689 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.690 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:06.690 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.376614+0000", 2023-11-05T08:34:06.691 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0035234469999999999 2023-11-05T08:34:06.691 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.691 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.692 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:06.692 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:15.376699+0000", 2023-11-05T08:34:06.692 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.5023000000000002e-05 2023-11-05T08:34:06.693 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.693 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:06.693 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.694 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.694 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.694 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.bs3 66/37 ECSubWrite(tid=108, reqid=client.4691.0:496, at_version=66'23, trim_to=0'0, roll_forward_to=65'21))", 2023-11-05T08:34:06.695 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:19.400396+0000", 2023-11-05T08:34:06.695 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 47.218190591999999, 2023-11-05T08:34:06.696 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0042162620000000001, 2023-11-05T08:34:06.696 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:06.696 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:34:06.697 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:06.697 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.698 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:06.698 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.400396+0000", 2023-11-05T08:34:06.699 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.699 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.700 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.700 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:06.700 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.400396+0000", 2023-11-05T08:34:06.700 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.701 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.701 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.701 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:06.702 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.400397+0000", 2023-11-05T08:34:06.702 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.8700000000000004e-07 2023-11-05T08:34:06.702 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.702 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.703 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:06.703 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.400694+0000", 2023-11-05T08:34:06.703 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00029789299999999998 2023-11-05T08:34:06.703 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.703 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.703 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:06.704 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.400711+0000", 2023-11-05T08:34:06.704 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.6909999999999999e-05 2023-11-05T08:34:06.704 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.704 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.704 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:06.705 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.400722+0000", 2023-11-05T08:34:06.705 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0892e-05 2023-11-05T08:34:06.705 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.705 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.705 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:06.705 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.400819+0000", 2023-11-05T08:34:06.706 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.7171999999999997e-05 2023-11-05T08:34:06.706 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.706 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.706 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:06.706 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.400830+0000", 2023-11-05T08:34:06.707 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0825999999999999e-05 2023-11-05T08:34:06.707 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.707 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.707 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:06.708 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.404534+0000", 2023-11-05T08:34:06.708 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0037035570000000001 2023-11-05T08:34:06.708 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.708 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.709 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:06.709 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.404612+0000", 2023-11-05T08:34:06.709 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.8125000000000002e-05 2023-11-05T08:34:06.709 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.710 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:06.710 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.710 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.710 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.710 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es6 66/30 ECSubWrite(tid=88, reqid=osd.2.0:88, at_version=66'67, trim_to=0'0, roll_forward_to=65'63))", 2023-11-05T08:34:06.711 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:19.416196+0000", 2023-11-05T08:34:06.711 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 47.202389928999999, 2023-11-05T08:34:06.711 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0032309859999999999, 2023-11-05T08:34:06.711 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:06.711 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:34:06.712 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:06.712 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.712 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:06.712 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.416196+0000", 2023-11-05T08:34:06.712 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.713 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.713 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.713 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:06.714 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.416196+0000", 2023-11-05T08:34:06.714 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.714 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.715 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.715 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:06.715 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.416197+0000", 2023-11-05T08:34:06.716 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0079999999999999e-06 2023-11-05T08:34:06.716 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.717 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.717 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:06.718 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.416228+0000", 2023-11-05T08:34:06.718 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.0837000000000003e-05 2023-11-05T08:34:06.718 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.719 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.719 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:06.719 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.416246+0000", 2023-11-05T08:34:06.719 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.8289999999999999e-05 2023-11-05T08:34:06.719 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.720 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.720 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:06.720 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.416280+0000", 2023-11-05T08:34:06.720 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.3791000000000002e-05 2023-11-05T08:34:06.721 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.721 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.721 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:06.721 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.416373+0000", 2023-11-05T08:34:06.722 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.2650000000000002e-05 2023-11-05T08:34:06.722 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.722 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.722 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:06.722 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.416388+0000", 2023-11-05T08:34:06.722 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.471e-05 2023-11-05T08:34:06.723 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.723 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.723 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:06.723 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.419374+0000", 2023-11-05T08:34:06.723 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.002986388 2023-11-05T08:34:06.724 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.724 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.724 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:06.724 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:19.419427+0000", 2023-11-05T08:34:06.724 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.3312000000000001e-05 2023-11-05T08:34:06.724 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.725 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:06.725 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.725 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.725 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.726 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es6 91/30 ECSubWrite(tid=109, reqid=osd.2.0:109, at_version=91'86, trim_to=0'0, roll_forward_to=90'84))", 2023-11-05T08:34:06.726 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:45.091079+0000", 2023-11-05T08:34:06.726 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 21.527506769999999, 2023-11-05T08:34:06.726 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0025699590000000001, 2023-11-05T08:34:06.727 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:06.727 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:34:06.727 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:06.727 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.728 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:06.728 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.091079+0000", 2023-11-05T08:34:06.728 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.728 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.729 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.729 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:06.729 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.091079+0000", 2023-11-05T08:34:06.729 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.730 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.730 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.730 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:06.730 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.091080+0000", 2023-11-05T08:34:06.731 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.0699999999999998e-07 2023-11-05T08:34:06.731 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.731 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.732 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:06.732 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.091119+0000", 2023-11-05T08:34:06.732 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.8902999999999997e-05 2023-11-05T08:34:06.732 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.733 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.733 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:06.733 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.091154+0000", 2023-11-05T08:34:06.733 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.4816e-05 2023-11-05T08:34:06.734 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.734 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.734 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:06.735 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.091187+0000", 2023-11-05T08:34:06.735 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.3089000000000001e-05 2023-11-05T08:34:06.735 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.735 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.735 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:06.735 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.091370+0000", 2023-11-05T08:34:06.736 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00018351899999999999 2023-11-05T08:34:06.736 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.736 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.736 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:06.736 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.091409+0000", 2023-11-05T08:34:06.737 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.8346000000000002e-05 2023-11-05T08:34:06.737 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.737 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.737 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:06.737 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.093554+0000", 2023-11-05T08:34:06.738 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0021452870000000001 2023-11-05T08:34:06.738 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.738 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.738 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:06.738 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.093649+0000", 2023-11-05T08:34:06.738 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.5592000000000002e-05 2023-11-05T08:34:06.739 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.739 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:06.739 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.739 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.739 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.740 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es6 92/30 ECSubWrite(tid=111, reqid=osd.2.0:111, at_version=92'88, trim_to=0'0, roll_forward_to=91'86))", 2023-11-05T08:34:06.740 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:45.679765+0000", 2023-11-05T08:34:06.740 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 20.938821388000001, 2023-11-05T08:34:06.740 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0044167700000000004, 2023-11-05T08:34:06.740 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:06.741 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:34:06.741 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:06.741 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.741 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:06.741 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.679765+0000", 2023-11-05T08:34:06.742 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.742 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.742 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.742 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:06.742 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.679765+0000", 2023-11-05T08:34:06.742 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.743 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.743 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.743 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:06.743 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.679766+0000", 2023-11-05T08:34:06.743 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.0299999999999997e-07 2023-11-05T08:34:06.744 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.744 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.744 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:06.744 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.679835+0000", 2023-11-05T08:34:06.744 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.9105999999999999e-05 2023-11-05T08:34:06.745 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.745 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.745 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:06.745 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.679870+0000", 2023-11-05T08:34:06.745 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.5345000000000001e-05 2023-11-05T08:34:06.746 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.746 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.746 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:06.746 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.679896+0000", 2023-11-05T08:34:06.746 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.5533000000000001e-05 2023-11-05T08:34:06.746 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.747 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.747 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:06.747 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.680135+0000", 2023-11-05T08:34:06.747 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000239411 2023-11-05T08:34:06.747 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.748 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.748 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:06.748 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.680174+0000", 2023-11-05T08:34:06.748 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.8609000000000003e-05 2023-11-05T08:34:06.749 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.749 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.749 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:06.749 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.684095+0000", 2023-11-05T08:34:06.749 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0039209609999999997 2023-11-05T08:34:06.749 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.750 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.750 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:06.750 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.684182+0000", 2023-11-05T08:34:06.750 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.6902000000000002e-05 2023-11-05T08:34:06.750 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.751 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:06.751 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.751 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.751 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.751 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es6 92/30 ECSubWrite(tid=112, reqid=osd.2.0:111, at_version=0'0, trim_to=0'0, roll_forward_to=92'88))", 2023-11-05T08:34:06.752 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:45.687631+0000", 2023-11-05T08:34:06.752 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 20.930954993, 2023-11-05T08:34:06.752 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0028095389999999998, 2023-11-05T08:34:06.752 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:06.752 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:34:06.753 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:06.753 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.753 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:06.754 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.687631+0000", 2023-11-05T08:34:06.754 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.754 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.754 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.754 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:06.755 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.687631+0000", 2023-11-05T08:34:06.755 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.755 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.755 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.756 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:06.756 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.687634+0000", 2023-11-05T08:34:06.756 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.3e-06 2023-11-05T08:34:06.756 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.757 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.757 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:06.757 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.687664+0000", 2023-11-05T08:34:06.757 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.0249000000000001e-05 2023-11-05T08:34:06.758 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.758 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.758 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:06.758 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.687702+0000", 2023-11-05T08:34:06.758 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.8262000000000001e-05 2023-11-05T08:34:06.759 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.759 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.759 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:06.759 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.687728+0000", 2023-11-05T08:34:06.759 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.5567000000000001e-05 2023-11-05T08:34:06.760 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.760 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.760 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:06.760 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.687911+0000", 2023-11-05T08:34:06.760 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000183112 2023-11-05T08:34:06.760 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.761 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.761 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:06.761 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.687926+0000", 2023-11-05T08:34:06.761 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.499e-05 2023-11-05T08:34:06.761 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.762 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.762 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:06.762 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.690350+0000", 2023-11-05T08:34:06.762 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0024243849999999998 2023-11-05T08:34:06.762 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.763 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.763 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:06.763 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.690441+0000", 2023-11-05T08:34:06.763 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.0674000000000006e-05 2023-11-05T08:34:06.764 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.764 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:06.764 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.764 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.765 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.765 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es6 92/30 ECSubWrite(tid=116, reqid=client.4691.0:670, at_version=92'89, trim_to=0'0, roll_forward_to=92'88))", 2023-11-05T08:34:06.765 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:45.780269+0000", 2023-11-05T08:34:06.765 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 20.838316805000002, 2023-11-05T08:34:06.766 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0039783730000000003, 2023-11-05T08:34:06.766 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:06.766 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:34:06.766 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:06.767 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.767 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:06.767 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.780269+0000", 2023-11-05T08:34:06.768 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.768 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.768 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.768 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:06.769 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.780269+0000", 2023-11-05T08:34:06.769 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.769 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.769 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.770 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:06.770 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.780272+0000", 2023-11-05T08:34:06.770 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.373e-06 2023-11-05T08:34:06.770 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.771 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.771 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:06.771 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.781188+0000", 2023-11-05T08:34:06.771 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00091589999999999998 2023-11-05T08:34:06.771 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.772 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.772 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:06.772 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.781223+0000", 2023-11-05T08:34:06.772 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.5648999999999997e-05 2023-11-05T08:34:06.773 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.773 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.773 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:06.773 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.781241+0000", 2023-11-05T08:34:06.773 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.7748999999999999e-05 2023-11-05T08:34:06.774 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.774 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.774 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:06.774 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.781387+0000", 2023-11-05T08:34:06.774 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00014607699999999999 2023-11-05T08:34:06.775 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.775 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.775 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:06.775 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.781407+0000", 2023-11-05T08:34:06.775 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.0109999999999999e-05 2023-11-05T08:34:06.776 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.776 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.776 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:06.776 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.784167+0000", 2023-11-05T08:34:06.776 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0027593069999999999 2023-11-05T08:34:06.776 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.777 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.777 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:06.777 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.784248+0000", 2023-11-05T08:34:06.777 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.1207999999999999e-05 2023-11-05T08:34:06.777 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.778 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:06.778 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.778 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.778 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.778 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es6 92/30 ECSubWrite(tid=118, reqid=client.4691.0:671, at_version=92'90, trim_to=0'0, roll_forward_to=92'89))", 2023-11-05T08:34:06.779 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:45.788756+0000", 2023-11-05T08:34:06.779 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 20.829829852, 2023-11-05T08:34:06.779 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.004254674, 2023-11-05T08:34:06.779 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:06.779 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:34:06.780 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:06.780 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.780 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:06.780 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.788756+0000", 2023-11-05T08:34:06.780 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.781 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.781 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.781 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:06.781 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.788756+0000", 2023-11-05T08:34:06.781 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.781 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.782 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.782 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:06.782 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.788758+0000", 2023-11-05T08:34:06.782 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.6029999999999999e-06 2023-11-05T08:34:06.782 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.783 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.783 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:06.783 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.789682+0000", 2023-11-05T08:34:06.783 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00092436600000000003 2023-11-05T08:34:06.783 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.784 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.784 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:06.784 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.789713+0000", 2023-11-05T08:34:06.784 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.0127000000000001e-05 2023-11-05T08:34:06.785 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.785 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.785 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:06.785 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.789728+0000", 2023-11-05T08:34:06.786 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.5911999999999999e-05 2023-11-05T08:34:06.786 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.786 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.786 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:06.787 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.789872+0000", 2023-11-05T08:34:06.787 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00014347399999999999 2023-11-05T08:34:06.787 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.787 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.788 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:06.788 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.789890+0000", 2023-11-05T08:34:06.788 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.7632999999999999e-05 2023-11-05T08:34:06.788 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.788 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.789 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:06.789 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.792931+0000", 2023-11-05T08:34:06.789 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.003041218 2023-11-05T08:34:06.789 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.790 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.790 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:06.790 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.793011+0000", 2023-11-05T08:34:06.790 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.0340999999999995e-05 2023-11-05T08:34:06.790 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.791 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:06.791 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.791 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.791 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.791 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.es6 92/30 ECSubWrite(tid=119, reqid=client.4691.0:672, at_version=92'91, trim_to=0'0, roll_forward_to=92'89))", 2023-11-05T08:34:06.792 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:45.792577+0000", 2023-11-05T08:34:06.792 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 20.826009331000002, 2023-11-05T08:34:06.792 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0030989049999999999, 2023-11-05T08:34:06.792 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:06.792 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:34:06.792 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:06.793 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.793 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:06.793 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.792577+0000", 2023-11-05T08:34:06.793 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.793 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.794 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.794 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:06.794 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.792577+0000", 2023-11-05T08:34:06.795 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.795 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.795 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.795 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:06.796 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.792579+0000", 2023-11-05T08:34:06.796 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.2330000000000001e-06 2023-11-05T08:34:06.796 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.796 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.797 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:06.797 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.792946+0000", 2023-11-05T08:34:06.797 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00036645500000000001 2023-11-05T08:34:06.797 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.798 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.798 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:06.798 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.792974+0000", 2023-11-05T08:34:06.798 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.8005999999999999e-05 2023-11-05T08:34:06.799 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.799 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.799 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:06.799 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.792991+0000", 2023-11-05T08:34:06.800 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.7646e-05 2023-11-05T08:34:06.800 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.800 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.800 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:06.801 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.793136+0000", 2023-11-05T08:34:06.801 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000144487 2023-11-05T08:34:06.801 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.801 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.802 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:06.802 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.793153+0000", 2023-11-05T08:34:06.802 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.7062999999999999e-05 2023-11-05T08:34:06.802 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.802 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.802 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:06.803 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.795609+0000", 2023-11-05T08:34:06.803 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0024557609999999999 2023-11-05T08:34:06.803 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.803 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.803 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:06.804 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.795676+0000", 2023-11-05T08:34:06.804 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.7254000000000004e-05 2023-11-05T08:34:06.804 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.804 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:06.804 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.805 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.805 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.805 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.bs3 92/37 ECSubWrite(tid=128, reqid=client.4691.0:683, at_version=92'29, trim_to=0'0, roll_forward_to=74'28))", 2023-11-05T08:34:06.805 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:45.867508+0000", 2023-11-05T08:34:06.805 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 20.751078054000001, 2023-11-05T08:34:06.806 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0036355440000000001, 2023-11-05T08:34:06.806 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:06.806 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:34:06.806 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:06.806 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.807 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:06.807 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.867508+0000", 2023-11-05T08:34:06.807 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.807 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.807 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.808 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:06.808 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.867508+0000", 2023-11-05T08:34:06.808 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.808 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.808 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.809 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:06.809 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.867509+0000", 2023-11-05T08:34:06.809 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.2699999999999999e-06 2023-11-05T08:34:06.809 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.809 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.810 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:06.810 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.868082+0000", 2023-11-05T08:34:06.810 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00057286999999999995 2023-11-05T08:34:06.810 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.810 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.811 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:06.811 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.868126+0000", 2023-11-05T08:34:06.811 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.3846e-05 2023-11-05T08:34:06.811 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.811 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.811 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:06.812 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.868157+0000", 2023-11-05T08:34:06.812 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.0389e-05 2023-11-05T08:34:06.812 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.812 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.812 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:06.813 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.868342+0000", 2023-11-05T08:34:06.813 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000185763 2023-11-05T08:34:06.813 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.813 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.813 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:06.814 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.868359+0000", 2023-11-05T08:34:06.814 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.6852999999999999e-05 2023-11-05T08:34:06.814 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.814 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.814 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:06.815 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.871046+0000", 2023-11-05T08:34:06.815 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0026865719999999999 2023-11-05T08:34:06.815 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.815 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.815 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:06.815 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.871144+0000", 2023-11-05T08:34:06.816 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.7980999999999996e-05 2023-11-05T08:34:06.816 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.816 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:06.816 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.816 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.817 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.817 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.bs3 92/37 ECSubWrite(tid=129, reqid=client.4691.0:684, at_version=92'30, trim_to=0'0, roll_forward_to=74'28))", 2023-11-05T08:34:06.817 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:45.872966+0000", 2023-11-05T08:34:06.817 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 20.745619934, 2023-11-05T08:34:06.817 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.003476804, 2023-11-05T08:34:06.818 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:06.818 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:34:06.818 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:06.818 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.818 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:06.819 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.872966+0000", 2023-11-05T08:34:06.819 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.819 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.819 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.820 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:06.820 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.872966+0000", 2023-11-05T08:34:06.820 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.820 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.821 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.821 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:06.821 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.872968+0000", 2023-11-05T08:34:06.821 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.5400000000000001e-06 2023-11-05T08:34:06.821 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.822 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.822 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:06.822 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.873209+0000", 2023-11-05T08:34:06.822 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000241427 2023-11-05T08:34:06.823 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.823 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.823 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:06.823 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.873241+0000", 2023-11-05T08:34:06.824 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.1853000000000001e-05 2023-11-05T08:34:06.824 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.824 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.824 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:06.825 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.873262+0000", 2023-11-05T08:34:06.825 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.0772999999999999e-05 2023-11-05T08:34:06.825 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.825 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.826 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:06.826 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.873458+0000", 2023-11-05T08:34:06.826 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00019627199999999999 2023-11-05T08:34:06.826 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.827 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.827 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:06.827 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.873485+0000", 2023-11-05T08:34:06.828 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.6916000000000001e-05 2023-11-05T08:34:06.828 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.828 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.828 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:06.829 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.876336+0000", 2023-11-05T08:34:06.829 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.002851152 2023-11-05T08:34:06.829 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.829 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.830 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:06.830 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.876443+0000", 2023-11-05T08:34:06.830 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000106871 2023-11-05T08:34:06.830 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.831 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:06.831 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.831 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.831 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.831 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.bs3 92/37 ECSubWrite(tid=130, reqid=client.4691.0:685, at_version=92'31, trim_to=0'0, roll_forward_to=92'29))", 2023-11-05T08:34:06.831 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:45.879138+0000", 2023-11-05T08:34:06.832 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 20.739447784999999, 2023-11-05T08:34:06.832 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0029156410000000001, 2023-11-05T08:34:06.832 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:06.832 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:34:06.832 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:06.833 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.833 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:06.833 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.879138+0000", 2023-11-05T08:34:06.833 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.833 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.834 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.834 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:06.834 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.879138+0000", 2023-11-05T08:34:06.834 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.834 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.835 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.835 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:06.835 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.879140+0000", 2023-11-05T08:34:06.835 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4699999999999999e-06 2023-11-05T08:34:06.835 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.836 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.836 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:06.836 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.879247+0000", 2023-11-05T08:34:06.836 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000107518 2023-11-05T08:34:06.836 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.837 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.837 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:06.837 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.879288+0000", 2023-11-05T08:34:06.837 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 4.0222000000000002e-05 2023-11-05T08:34:06.837 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.838 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.838 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:06.838 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.879306+0000", 2023-11-05T08:34:06.838 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.8039000000000001e-05 2023-11-05T08:34:06.838 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.838 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.839 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:06.839 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.879471+0000", 2023-11-05T08:34:06.839 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00016571700000000001 2023-11-05T08:34:06.839 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.839 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.840 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:06.840 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.879503+0000", 2023-11-05T08:34:06.840 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.1813000000000002e-05 2023-11-05T08:34:06.840 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.840 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.841 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:06.841 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.881974+0000", 2023-11-05T08:34:06.841 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0024711640000000001 2023-11-05T08:34:06.841 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.841 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.841 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:06.842 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:45.882054+0000", 2023-11-05T08:34:06.842 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.9697999999999995e-05 2023-11-05T08:34:06.842 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.842 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:06.842 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.843 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.843 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.843 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.18s6 97/45 ECSubWrite(tid=317, reqid=osd.13.0:317, at_version=97'42, trim_to=0'0, roll_forward_to=68'40))", 2023-11-05T08:34:06.843 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:50.666460+0000", 2023-11-05T08:34:06.843 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 15.952126402999999, 2023-11-05T08:34:06.844 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0034138340000000001, 2023-11-05T08:34:06.844 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:06.844 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:34:06.844 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:06.844 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.845 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:06.845 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:50.666460+0000", 2023-11-05T08:34:06.845 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.845 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.845 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.845 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:06.846 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:50.666460+0000", 2023-11-05T08:34:06.846 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.846 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.846 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.846 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:06.847 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:50.666461+0000", 2023-11-05T08:34:06.847 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 6.8999999999999996e-07 2023-11-05T08:34:06.847 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.847 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.847 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:06.848 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:50.666534+0000", 2023-11-05T08:34:06.848 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.3762000000000003e-05 2023-11-05T08:34:06.848 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.848 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.848 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:06.848 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:50.666569+0000", 2023-11-05T08:34:06.849 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.4993000000000002e-05 2023-11-05T08:34:06.849 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.849 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.849 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:06.850 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:50.666600+0000", 2023-11-05T08:34:06.850 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.0306000000000001e-05 2023-11-05T08:34:06.850 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.850 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.851 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:06.851 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:50.666785+0000", 2023-11-05T08:34:06.851 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.000185403 2023-11-05T08:34:06.851 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.852 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.852 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:06.852 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:50.666813+0000", 2023-11-05T08:34:06.852 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.8449000000000002e-05 2023-11-05T08:34:06.853 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.853 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.853 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:06.853 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:50.669817+0000", 2023-11-05T08:34:06.854 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.003003361 2023-11-05T08:34:06.854 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.854 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.854 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:06.855 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:50.669874+0000", 2023-11-05T08:34:06.855 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.6870000000000003e-05 2023-11-05T08:34:06.855 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.855 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:06.856 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.856 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.856 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.857 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.18s6 97/45 ECSubWrite(tid=319, reqid=osd.13.0:319, at_version=97'44, trim_to=0'0, roll_forward_to=68'40))", 2023-11-05T08:34:06.857 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:50.667377+0000", 2023-11-05T08:34:06.857 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 15.951209348000001, 2023-11-05T08:34:06.857 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0051578689999999998, 2023-11-05T08:34:06.858 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:06.858 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:34:06.858 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:06.858 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.858 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:06.859 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:50.667377+0000", 2023-11-05T08:34:06.859 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.859 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.859 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.859 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:06.860 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:50.667377+0000", 2023-11-05T08:34:06.860 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.860 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.860 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.860 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:06.861 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:50.667378+0000", 2023-11-05T08:34:06.861 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.4330000000000001e-06 2023-11-05T08:34:06.861 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.861 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.862 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:06.862 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:50.667438+0000", 2023-11-05T08:34:06.862 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.9898999999999997e-05 2023-11-05T08:34:06.862 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.863 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.863 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:06.863 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:50.667467+0000", 2023-11-05T08:34:06.863 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.8373000000000001e-05 2023-11-05T08:34:06.864 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.864 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.864 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:06.865 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:50.667488+0000", 2023-11-05T08:34:06.865 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.1199000000000002e-05 2023-11-05T08:34:06.865 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.865 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.865 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:06.866 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:50.667929+0000", 2023-11-05T08:34:06.866 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00044106800000000003 2023-11-05T08:34:06.866 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.866 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.866 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:06.867 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:50.667966+0000", 2023-11-05T08:34:06.867 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.6854999999999999e-05 2023-11-05T08:34:06.867 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.867 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.867 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:06.868 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:50.672461+0000", 2023-11-05T08:34:06.868 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0044949780000000002 2023-11-05T08:34:06.868 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.868 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.868 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:06.869 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:50.672535+0000", 2023-11-05T08:34:06.869 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 7.4064000000000001e-05 2023-11-05T08:34:06.869 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.869 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:06.869 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.869 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.870 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.870 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.18s6 106/45 ECSubWrite(tid=369, reqid=osd.13.0:369, at_version=106'61, trim_to=0'0, roll_forward_to=105'59))", 2023-11-05T08:34:06.870 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:33:59.934861+0000", 2023-11-05T08:34:06.870 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 6.6837256509999996, 2023-11-05T08:34:06.870 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.003372917, 2023-11-05T08:34:06.871 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:06.871 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:34:06.871 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:06.871 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.871 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:06.872 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:59.934861+0000", 2023-11-05T08:34:06.872 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.872 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.872 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.872 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:06.873 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:59.934861+0000", 2023-11-05T08:34:06.873 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.873 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.873 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.873 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:06.873 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:59.934861+0000", 2023-11-05T08:34:06.874 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.8e-07 2023-11-05T08:34:06.874 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.874 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.874 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:06.874 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:59.934891+0000", 2023-11-05T08:34:06.875 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.0352e-05 2023-11-05T08:34:06.875 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.875 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.875 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:06.876 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:59.934907+0000", 2023-11-05T08:34:06.876 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.5673000000000001e-05 2023-11-05T08:34:06.876 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.876 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.876 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:06.876 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:59.934920+0000", 2023-11-05T08:34:06.877 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.3565e-05 2023-11-05T08:34:06.877 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.877 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.877 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:06.877 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:59.935021+0000", 2023-11-05T08:34:06.878 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00010043399999999999 2023-11-05T08:34:06.878 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.878 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.878 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:06.878 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:59.935033+0000", 2023-11-05T08:34:06.879 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.1989e-05 2023-11-05T08:34:06.879 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.879 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.879 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:06.879 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:59.938174+0000", 2023-11-05T08:34:06.880 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.003140842 2023-11-05T08:34:06.880 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.880 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.880 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:06.880 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:33:59.938234+0000", 2023-11-05T08:34:06.881 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 5.9882e-05 2023-11-05T08:34:06.881 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.881 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:06.881 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.881 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.882 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.882 INFO:teuthology.orchestra.run.smithi002.stdout: "description": "MOSDECSubOpWrite(3.bs3 111/37 ECSubWrite(tid=185, reqid=client.4691.0:965, at_version=111'60, trim_to=0'0, roll_forward_to=108'58))", 2023-11-05T08:34:06.882 INFO:teuthology.orchestra.run.smithi002.stdout: "initiated_at": "2023-11-05T08:34:04.842148+0000", 2023-11-05T08:34:06.882 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 1.7764384209999999, 2023-11-05T08:34:06.883 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.002492036, 2023-11-05T08:34:06.883 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2023-11-05T08:34:06.883 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", 2023-11-05T08:34:06.883 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ 2023-11-05T08:34:06.884 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.884 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "initiated", 2023-11-05T08:34:06.884 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:34:04.842148+0000", 2023-11-05T08:34:06.884 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.884 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.885 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.885 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "header_read", 2023-11-05T08:34:06.885 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:34:04.842148+0000", 2023-11-05T08:34:06.885 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0 2023-11-05T08:34:06.886 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.886 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.886 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2023-11-05T08:34:06.886 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:34:04.842148+0000", 2023-11-05T08:34:06.887 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 2.2499999999999999e-07 2023-11-05T08:34:06.887 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.887 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.887 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "all_read", 2023-11-05T08:34:06.888 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:34:04.842182+0000", 2023-11-05T08:34:06.888 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.3930999999999998e-05 2023-11-05T08:34:06.888 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.888 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.889 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "dispatched", 2023-11-05T08:34:06.889 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:34:04.842192+0000", 2023-11-05T08:34:06.889 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.0101999999999999e-05 2023-11-05T08:34:06.890 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.890 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.890 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", 2023-11-05T08:34:06.890 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:34:04.842201+0000", 2023-11-05T08:34:06.891 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.3160000000000003e-06 2023-11-05T08:34:06.891 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.891 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.891 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "reached_pg", 2023-11-05T08:34:06.892 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:34:04.842289+0000", 2023-11-05T08:34:06.892 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 8.7212000000000005e-05 2023-11-05T08:34:06.892 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.892 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.893 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", 2023-11-05T08:34:06.893 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:34:04.842302+0000", 2023-11-05T08:34:06.893 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 1.3478999999999999e-05 2023-11-05T08:34:06.893 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.894 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.894 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", 2023-11-05T08:34:06.894 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:34:04.844602+0000", 2023-11-05T08:34:06.894 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0023001369999999998 2023-11-05T08:34:06.894 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-05T08:34:06.894 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-05T08:34:06.895 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "done", 2023-11-05T08:34:06.895 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2023-11-05T08:34:04.844640+0000", 2023-11-05T08:34:06.895 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 3.7633999999999999e-05 2023-11-05T08:34:06.895 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.895 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:06.896 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.896 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-05T08:34:06.896 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-05T08:34:06.896 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-05T08:34:06.897 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-05T08:34:06.898 INFO:tasks.rados.rados.0.smithi090.stdout:389: done (0 left) 2023-11-05T08:34:06.898 INFO:tasks.rados.rados.0.smithi090.stdout:390: snap_create 2023-11-05T08:34:06.899 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-11-05T08:34:06.900 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:34:06.892+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:34:06.999 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:34:06.994+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:34:07.029 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-11-05T08:34:07.030 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2023-11-05T08:34:07.030 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 0 2023-11-05T08:34:07.031 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-11-05T08:34:07.044 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-11-05T08:34:07.083 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-11-05T08:34:07.099 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:34:07.095+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:34:07.103 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1: {} 2023-11-05T08:34:07.104 INFO:teuthology.orchestra.run.smithi002.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-11-05T08:34:07.124 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2: {} 2023-11-05T08:34:07.125 INFO:teuthology.orchestra.run.smithi002.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-11-05T08:34:07.145 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3: {} 2023-11-05T08:34:07.146 INFO:teuthology.orchestra.run.smithi002.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-11-05T08:34:07.165 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4: {} 2023-11-05T08:34:07.166 INFO:teuthology.orchestra.run.smithi002.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-11-05T08:34:07.172 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:34:07.172 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:34:07.186 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5: {} 2023-11-05T08:34:07.186 INFO:teuthology.orchestra.run.smithi002.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-11-05T08:34:07.187 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-11-05T08:34:07.200 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-11-05T08:34:07.198+0000 7f72b2c5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 42525) UID: 0 2023-11-05T08:34:07.209 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6: {} 2023-11-05T08:34:07.209 INFO:teuthology.orchestra.run.smithi002.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-11-05T08:34:07.232 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7: {} 2023-11-05T08:34:07.233 INFO:teuthology.orchestra.run.smithi002.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-11-05T08:34:07.255 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8: {} 2023-11-05T08:34:07.256 INFO:teuthology.orchestra.run.smithi002.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-11-05T08:34:07.276 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9: {} 2023-11-05T08:34:07.277 INFO:teuthology.orchestra.run.smithi002.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-11-05T08:34:07.300 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10: {} 2023-11-05T08:34:07.301 INFO:teuthology.orchestra.run.smithi002.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-11-05T08:34:07.301 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:34:07.295+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:34:07.324 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11: {} 2023-11-05T08:34:07.324 INFO:teuthology.orchestra.run.smithi002.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-11-05T08:34:07.345 INFO:teuthology.orchestra.run.smithi002.stdout:osd.12: {} 2023-11-05T08:34:07.345 INFO:teuthology.orchestra.run.smithi002.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-11-05T08:34:07.356 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:34:07.356 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:34:07.369 INFO:teuthology.orchestra.run.smithi002.stdout:osd.13: {} 2023-11-05T08:34:07.369 INFO:teuthology.orchestra.run.smithi002.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-11-05T08:34:07.371 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-11-05T08:34:07.394 INFO:teuthology.orchestra.run.smithi002.stdout:osd.14: {} 2023-11-05T08:34:07.395 INFO:teuthology.orchestra.run.smithi002.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-11-05T08:34:07.396 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-05T08:34:07.392+0000 7ff6794da640 0 -- 172.21.15.2:0/2806976941 >> [v2:172.21.15.90:6824/739151808,v1:172.21.15.90:6825/739151808] conn(0x7ff67421fdc0 msgr2=0x7ff6742221b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-11-05T08:34:07.401 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-11-05T08:34:07.398+0000 7f72b2c5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 42525) UID: 0 2023-11-05T08:34:07.436 INFO:teuthology.orchestra.run.smithi002.stdout:osd.15: {} 2023-11-05T08:34:07.437 INFO:teuthology.orchestra.run.smithi002.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-11-05T08:34:07.443 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-05T08:34:07.443 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-05T08:34:07.501 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-05T08:34:07.543 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:34:07.544 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:34:07.558 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-11-05T08:34:07.606 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:34:07.603+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:34:07.690 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:34:07.690 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:34:07.705 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-11-05T08:34:07.705 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:34:07.699+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:34:07.805 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:34:07.799+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:34:07.880 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:34:07.881 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:34:07.883 INFO:tasks.rados.rados.0.smithi090.stdout:390: done (0 left) 2023-11-05T08:34:07.883 INFO:tasks.rados.rados.0.smithi090.stdout:391: read oid 25 snap -1 2023-11-05T08:34:07.884 INFO:tasks.rados.rados.0.smithi090.stdout:391: expect deleted 2023-11-05T08:34:07.884 INFO:tasks.rados.rados.0.smithi090.stdout:392: read oid 14 snap -1 2023-11-05T08:34:07.884 INFO:tasks.rados.rados.0.smithi090.stdout:392: expect (ObjNum 3 snap 0 seq_num 3) 2023-11-05T08:34:07.884 INFO:tasks.rados.rados.0.smithi090.stdout:393: copy_from oid 44 from oid 20 current snap is 41 2023-11-05T08:34:07.885 INFO:tasks.rados.rados.0.smithi090.stdout:394: setattr oid 22 current snap is 41 2023-11-05T08:34:07.889 INFO:tasks.rados.rados.0.smithi090.stdout:391: done (3 left) 2023-11-05T08:34:07.890 INFO:tasks.rados.rados.0.smithi090.stdout:395: snap_remove snap 40 2023-11-05T08:34:07.897 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2023-11-05T08:34:07.942 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:34:07.939+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:34:08.008 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:34:08.004+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:34:08.060 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:34:08.061 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:34:08.075 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2023-11-05T08:34:08.109 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:34:08.104+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:34:08.132 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-11-05T08:34:08.202 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:34:08.202 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:34:08.210 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:34:08.204+0000 7fdc5cd58640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:34:08.216 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2023-11-05T08:34:08.311 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:34:08.304+0000 7fda6d004640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:34:08.345 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:34:08.345 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:34:08.362 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2023-11-05T08:34:08.412 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:34:08.410+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:34:08.484 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:34:08.484 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:34:08.498 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2023-11-05T08:34:08.543 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:34:08.539+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:34:08.614 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:34:08.610+0000 7fc10ec39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:34:08.631 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:34:08.631 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:34:08.646 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2023-11-05T08:34:08.715 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:34:08.712+0000 7fdc5cd58640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:34:08.816 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:34:08.814+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:34:08.822 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2023-11-05T08:34:08.822 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:34:08.839 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-11-05T08:34:08.884 INFO:teuthology.orchestra.run.smithi002.stderr:nodeep-scrub is set 2023-11-05T08:34:08.890 INFO:tasks.rados.rados.0.smithi090.stdout:395: done (3 left) 2023-11-05T08:34:08.891 INFO:tasks.rados.rados.0.smithi090.stdout:396: read oid 13 snap -1 2023-11-05T08:34:08.891 INFO:tasks.rados.rados.0.smithi090.stdout:396: expect (ObjNum 119 snap 32 seq_num 119) 2023-11-05T08:34:08.891 INFO:tasks.rados.rados.0.smithi090.stdout:397: snap_remove snap 39 2023-11-05T08:34:08.917 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~73s 2023-11-05T08:34:08.918 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:34:08.911+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:34:08.978 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:34:08.979 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:34:08.994 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-11-05T08:34:09.040 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:34:09.036+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:34:09.115 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:34:09.119 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:34:09.120 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:34:09.116+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:34:09.129 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-11-05T08:34:09.221 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:34:09.216+0000 7fa106acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:34:09.298 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:34:09.298 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:34:09.312 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-11-05T08:34:09.356 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:34:09.352+0000 7f226237e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:34:09.422 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:34:09.416+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:34:09.445 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-05T08:34:09.477 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:34:09.478 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:34:09.491 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-11-05T08:34:09.522 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-05T08:34:09.586 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-11-05T08:34:09.608 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1: {} 2023-11-05T08:34:09.608 INFO:teuthology.orchestra.run.smithi002.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-11-05T08:34:09.624 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:34:09.620+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:34:09.629 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2: {} 2023-11-05T08:34:09.629 INFO:teuthology.orchestra.run.smithi002.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-11-05T08:34:09.651 INFO:tasks.rados.rados.0.smithi090.stdout:397: done (4 left) 2023-11-05T08:34:09.651 INFO:tasks.rados.rados.0.smithi090.stdout:398: copy_from oid 49 from oid 15 current snap is 41 2023-11-05T08:34:09.651 INFO:tasks.rados.rados.0.smithi090.stdout:399: snap_remove snap 38 2023-11-05T08:34:09.658 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3: {} 2023-11-05T08:34:09.658 INFO:teuthology.orchestra.run.smithi002.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-11-05T08:34:09.664 INFO:teuthology.orchestra.run.smithi069.stdout:ERROR: (22) Invalid argument 2023-11-05T08:34:09.665 INFO:teuthology.orchestra.run.smithi069.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:34:09.677 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4: {} 2023-11-05T08:34:09.677 INFO:teuthology.orchestra.run.smithi002.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-11-05T08:34:09.680 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-11-05T08:34:09.698 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5: {} 2023-11-05T08:34:09.699 INFO:teuthology.orchestra.run.smithi002.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-11-05T08:34:09.719 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6: {} 2023-11-05T08:34:09.720 INFO:teuthology.orchestra.run.smithi002.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-11-05T08:34:09.725 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:34:09.719+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:34:09.739 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7: {} 2023-11-05T08:34:09.740 INFO:teuthology.orchestra.run.smithi002.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-11-05T08:34:09.757 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8: {} 2023-11-05T08:34:09.757 INFO:teuthology.orchestra.run.smithi002.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-11-05T08:34:09.779 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9: {} 2023-11-05T08:34:09.780 INFO:teuthology.orchestra.run.smithi002.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-11-05T08:34:09.801 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:34:09.802 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 20, 2023-11-05T08:34:09.802 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 600, 2023-11-05T08:34:09.802 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [ 2023-11-05T08:34:09.802 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.803 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.as5 42/33 ECSubWrite(tid=38, reqid=osd.7.0:38, at_version=42'16, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:34:09.803 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:55.127656+0000", 2023-11-05T08:34:09.803 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 74.669310730000007, 2023-11-05T08:34:09.803 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.016413512000000002, 2023-11-05T08:34:09.803 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:09.803 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:09.804 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:09.804 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.804 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:09.804 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127656+0000", 2023-11-05T08:34:09.804 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:09.804 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.805 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.805 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:09.805 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127656+0000", 2023-11-05T08:34:09.805 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:09.805 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.806 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.806 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:09.806 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127656+0000", 2023-11-05T08:34:09.806 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.2699999999999999e-07 2023-11-05T08:34:09.806 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.807 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.807 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:09.807 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127732+0000", 2023-11-05T08:34:09.807 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.5978999999999999e-05 2023-11-05T08:34:09.807 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.808 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.808 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:09.808 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127763+0000", 2023-11-05T08:34:09.808 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0672e-05 2023-11-05T08:34:09.808 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.809 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.809 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:09.809 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127785+0000", 2023-11-05T08:34:09.809 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2059e-05 2023-11-05T08:34:09.809 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.810 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.810 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:09.810 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127975+0000", 2023-11-05T08:34:09.810 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000189946 2023-11-05T08:34:09.810 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.811 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.811 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:09.811 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127999+0000", 2023-11-05T08:34:09.811 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3555999999999999e-05 2023-11-05T08:34:09.811 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.812 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.812 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:09.812 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.143967+0000", 2023-11-05T08:34:09.812 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.015968781000000001 2023-11-05T08:34:09.812 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.812 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.813 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:09.813 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.144069+0000", 2023-11-05T08:34:09.813 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000101992 2023-11-05T08:34:09.813 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:09.813 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:09.814 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:09.814 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.814 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.814 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.as5 42/33 ECSubWrite(tid=39, reqid=osd.7.0:39, at_version=42'18, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:34:09.814 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:55.127880+0000", 2023-11-05T08:34:09.815 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 74.669086245000003, 2023-11-05T08:34:09.815 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.019355170000000001, 2023-11-05T08:34:09.815 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:09.815 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:09.815 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:09.816 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.816 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:09.816 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127880+0000", 2023-11-05T08:34:09.816 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:09.816 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.817 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.817 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:09.817 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127880+0000", 2023-11-05T08:34:09.817 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:09.818 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.818 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.818 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:09.818 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127882+0000", 2023-11-05T08:34:09.818 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.187e-06 2023-11-05T08:34:09.818 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.819 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.819 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:09.819 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127934+0000", 2023-11-05T08:34:09.819 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.2553000000000003e-05 2023-11-05T08:34:09.819 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.820 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.820 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:09.820 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127961+0000", 2023-11-05T08:34:09.820 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6999e-05 2023-11-05T08:34:09.820 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.821 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.821 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:09.821 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127983+0000", 2023-11-05T08:34:09.821 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1778999999999999e-05 2023-11-05T08:34:09.821 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.821 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.822 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:09.822 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.129155+0000", 2023-11-05T08:34:09.822 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001172022 2023-11-05T08:34:09.822 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.822 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.823 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:09.823 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.129180+0000", 2023-11-05T08:34:09.823 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4841000000000002e-05 2023-11-05T08:34:09.823 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.823 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.823 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:09.824 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.147146+0000", 2023-11-05T08:34:09.824 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.017965990000000001 2023-11-05T08:34:09.824 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.824 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.824 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:09.825 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.147236+0000", 2023-11-05T08:34:09.825 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.9798999999999999e-05 2023-11-05T08:34:09.825 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:09.825 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:09.825 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:09.825 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.826 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.826 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.es5 42/30 ECSubWrite(tid=43, reqid=osd.2.0:43, at_version=42'29, trim_to=0'0, roll_forward_to=41'27))", 2023-11-05T08:34:09.826 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:55.134988+0000", 2023-11-05T08:34:09.826 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 74.661978429000001, 2023-11-05T08:34:09.826 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.012343942, 2023-11-05T08:34:09.827 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:09.827 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:09.827 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:09.827 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.827 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:09.827 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.134988+0000", 2023-11-05T08:34:09.828 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:09.828 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.828 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.828 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:09.828 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.134988+0000", 2023-11-05T08:34:09.829 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:09.829 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.829 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.829 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:09.829 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.134989+0000", 2023-11-05T08:34:09.829 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.6000000000000003e-07 2023-11-05T08:34:09.830 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.830 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.830 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:09.830 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.135039+0000", 2023-11-05T08:34:09.830 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.9879999999999997e-05 2023-11-05T08:34:09.831 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.831 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.831 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:09.831 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.135069+0000", 2023-11-05T08:34:09.831 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0445000000000001e-05 2023-11-05T08:34:09.831 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.832 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.832 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:09.832 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.135094+0000", 2023-11-05T08:34:09.832 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4646000000000001e-05 2023-11-05T08:34:09.832 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.833 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.833 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:09.833 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.135758+0000", 2023-11-05T08:34:09.833 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00066366799999999996 2023-11-05T08:34:09.833 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.834 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.834 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:09.834 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.135780+0000", 2023-11-05T08:34:09.834 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2509000000000001e-05 2023-11-05T08:34:09.834 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.835 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.835 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:09.835 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.147237+0000", 2023-11-05T08:34:09.835 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.011456401999999999 2023-11-05T08:34:09.835 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.835 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.836 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:09.836 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.147332+0000", 2023-11-05T08:34:09.836 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.5631999999999995e-05 2023-11-05T08:34:09.836 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:09.836 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:09.837 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:09.837 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.837 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.837 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.bs4 66/37 ECSubWrite(tid=108, reqid=client.4691.0:496, at_version=66'23, trim_to=0'0, roll_forward_to=65'21))", 2023-11-05T08:34:09.837 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:19.399298+0000", 2023-11-05T08:34:09.837 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 50.397668262000003, 2023-11-05T08:34:09.838 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.012016639000000001, 2023-11-05T08:34:09.838 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:09.838 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:09.838 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:09.838 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.839 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:09.839 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.399298+0000", 2023-11-05T08:34:09.839 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:09.839 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.839 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.840 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:09.840 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.399298+0000", 2023-11-05T08:34:09.840 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:09.840 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.840 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.840 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:09.841 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.399299+0000", 2023-11-05T08:34:09.841 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.4399999999999999e-07 2023-11-05T08:34:09.841 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.841 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.841 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:09.842 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.399993+0000", 2023-11-05T08:34:09.842 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00069384999999999998 2023-11-05T08:34:09.842 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.842 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.842 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:09.843 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.400030+0000", 2023-11-05T08:34:09.843 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.6508999999999999e-05 2023-11-05T08:34:09.843 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.843 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.843 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:09.844 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.400046+0000", 2023-11-05T08:34:09.844 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6266e-05 2023-11-05T08:34:09.844 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.844 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.844 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:09.844 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.400213+0000", 2023-11-05T08:34:09.845 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00016678099999999999 2023-11-05T08:34:09.845 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.845 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.845 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:09.845 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.400240+0000", 2023-11-05T08:34:09.846 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6979999999999999e-05 2023-11-05T08:34:09.846 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.846 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.846 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:09.846 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.411240+0000", 2023-11-05T08:34:09.846 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.010999978000000001 2023-11-05T08:34:09.847 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.847 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.847 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:09.847 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.411315+0000", 2023-11-05T08:34:09.847 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.5531000000000006e-05 2023-11-05T08:34:09.848 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:09.848 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:09.848 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:09.848 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.848 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.848 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.18s1 97/45 ECSubWrite(tid=317, reqid=osd.13.0:317, at_version=97'42, trim_to=0'0, roll_forward_to=68'40))", 2023-11-05T08:34:09.849 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.668427+0000", 2023-11-05T08:34:09.849 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 19.128540045000001, 2023-11-05T08:34:09.849 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.014768801, 2023-11-05T08:34:09.849 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:09.849 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:09.850 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:09.850 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.850 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:09.850 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668427+0000", 2023-11-05T08:34:09.850 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:09.850 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.851 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.851 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:09.851 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668427+0000", 2023-11-05T08:34:09.851 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:09.851 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.852 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.852 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:09.852 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668429+0000", 2023-11-05T08:34:09.852 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.916e-06 2023-11-05T08:34:09.852 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.852 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.853 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:09.853 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668490+0000", 2023-11-05T08:34:09.853 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.0977999999999998e-05 2023-11-05T08:34:09.853 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.853 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.854 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:09.854 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668527+0000", 2023-11-05T08:34:09.854 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.7905999999999999e-05 2023-11-05T08:34:09.854 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.854 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.855 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:09.855 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668556+0000", 2023-11-05T08:34:09.855 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8605000000000001e-05 2023-11-05T08:34:09.855 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.855 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.855 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:09.856 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668745+0000", 2023-11-05T08:34:09.856 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00018850099999999999 2023-11-05T08:34:09.856 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.856 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.856 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:09.857 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668776+0000", 2023-11-05T08:34:09.857 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.1134999999999999e-05 2023-11-05T08:34:09.857 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.857 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.857 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:09.858 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.683106+0000", 2023-11-05T08:34:09.858 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.01433067 2023-11-05T08:34:09.858 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.858 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.858 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:09.859 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.683195+0000", 2023-11-05T08:34:09.859 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.9090000000000003e-05 2023-11-05T08:34:09.859 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:09.859 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:09.859 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:09.860 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.860 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.860 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.10s1 97/45 ECSubWrite(tid=318, reqid=osd.13.0:318, at_version=97'58, trim_to=0'0, roll_forward_to=93'56))", 2023-11-05T08:34:09.860 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.668619+0000", 2023-11-05T08:34:09.860 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 19.128347892000001, 2023-11-05T08:34:09.860 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.014652379, 2023-11-05T08:34:09.861 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:09.861 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:09.861 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:09.861 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.861 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:09.862 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668619+0000", 2023-11-05T08:34:09.862 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:09.862 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.862 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.862 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:09.863 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668619+0000", 2023-11-05T08:34:09.863 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:09.863 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.863 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.863 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:09.863 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668620+0000", 2023-11-05T08:34:09.864 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.017e-06 2023-11-05T08:34:09.864 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.864 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.864 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:09.864 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668710+0000", 2023-11-05T08:34:09.865 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.0587000000000005e-05 2023-11-05T08:34:09.865 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.865 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.865 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:09.865 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668736+0000", 2023-11-05T08:34:09.865 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6098999999999999e-05 2023-11-05T08:34:09.866 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.866 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.866 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:09.866 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668756+0000", 2023-11-05T08:34:09.866 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9599e-05 2023-11-05T08:34:09.867 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.867 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.867 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:09.867 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.669838+0000", 2023-11-05T08:34:09.867 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0010821240000000001 2023-11-05T08:34:09.868 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.868 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.868 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:09.868 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.669863+0000", 2023-11-05T08:34:09.868 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4496000000000001e-05 2023-11-05T08:34:09.868 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.869 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.869 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:09.869 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.683214+0000", 2023-11-05T08:34:09.869 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.013350879 2023-11-05T08:34:09.869 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.870 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.870 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:09.870 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.683271+0000", 2023-11-05T08:34:09.870 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.7578000000000003e-05 2023-11-05T08:34:09.870 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:09.871 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:09.871 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:09.871 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.871 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.871 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.10s5 97/45 ECSubWrite(tid=318, reqid=osd.13.0:318, at_version=97'58, trim_to=0'0, roll_forward_to=93'56))", 2023-11-05T08:34:09.872 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.668786+0000", 2023-11-05T08:34:09.872 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 19.128180731, 2023-11-05T08:34:09.872 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.014548782, 2023-11-05T08:34:09.872 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:09.872 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:09.872 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:09.873 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.873 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:09.873 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668786+0000", 2023-11-05T08:34:09.873 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:09.873 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.874 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.874 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:09.874 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668786+0000", 2023-11-05T08:34:09.874 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:09.875 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.875 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.875 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:09.875 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668787+0000", 2023-11-05T08:34:09.875 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.4e-07 2023-11-05T08:34:09.876 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.876 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.876 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:09.876 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668844+0000", 2023-11-05T08:34:09.876 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.7680999999999999e-05 2023-11-05T08:34:09.876 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.877 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.877 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:09.877 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668869+0000", 2023-11-05T08:34:09.877 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.41e-05 2023-11-05T08:34:09.877 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.878 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.878 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:09.878 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668891+0000", 2023-11-05T08:34:09.878 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2259000000000002e-05 2023-11-05T08:34:09.878 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.879 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.879 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:09.879 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.670415+0000", 2023-11-05T08:34:09.879 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0015243019999999999 2023-11-05T08:34:09.879 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.880 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.880 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:09.880 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.670492+0000", 2023-11-05T08:34:09.880 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.6576999999999995e-05 2023-11-05T08:34:09.880 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.881 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.881 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:09.881 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.683277+0000", 2023-11-05T08:34:09.881 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.012785179000000001 2023-11-05T08:34:09.881 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.882 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.882 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:09.882 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.683335+0000", 2023-11-05T08:34:09.882 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.7843999999999997e-05 2023-11-05T08:34:09.882 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:09.883 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:09.883 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:09.883 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.883 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.883 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.10s1 97/45 ECSubWrite(tid=320, reqid=osd.13.0:320, at_version=97'60, trim_to=0'0, roll_forward_to=93'56))", 2023-11-05T08:34:09.883 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.668922+0000", 2023-11-05T08:34:09.884 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 19.128044622000001, 2023-11-05T08:34:09.884 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.015049735999999999, 2023-11-05T08:34:09.884 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:09.884 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:09.884 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:09.885 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.885 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:09.885 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668922+0000", 2023-11-05T08:34:09.885 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:09.885 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.886 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.886 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:09.886 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668922+0000", 2023-11-05T08:34:09.886 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:09.886 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.886 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.887 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:09.887 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668923+0000", 2023-11-05T08:34:09.887 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.6300000000000005e-07 2023-11-05T08:34:09.887 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.887 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.887 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:09.888 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668964+0000", 2023-11-05T08:34:09.888 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.0878999999999999e-05 2023-11-05T08:34:09.888 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.888 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.888 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:09.889 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.668986+0000", 2023-11-05T08:34:09.889 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2166000000000001e-05 2023-11-05T08:34:09.889 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.889 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.889 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:09.889 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.669003+0000", 2023-11-05T08:34:09.890 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6892999999999999e-05 2023-11-05T08:34:09.890 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.890 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.890 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:09.890 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.673668+0000", 2023-11-05T08:34:09.891 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0046656019999999996 2023-11-05T08:34:09.891 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.891 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.891 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:09.891 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.673693+0000", 2023-11-05T08:34:09.891 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4816000000000001e-05 2023-11-05T08:34:09.892 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.892 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.892 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:09.892 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.683900+0000", 2023-11-05T08:34:09.892 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.010206814999999999 2023-11-05T08:34:09.893 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.893 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.893 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:09.893 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.683972+0000", 2023-11-05T08:34:09.893 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.1902000000000004e-05 2023-11-05T08:34:09.893 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:09.894 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:09.894 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:09.894 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.894 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.894 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.18s1 97/45 ECSubWrite(tid=319, reqid=osd.13.0:319, at_version=97'44, trim_to=0'0, roll_forward_to=68'40))", 2023-11-05T08:34:09.895 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.669028+0000", 2023-11-05T08:34:09.895 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 19.127938569000001, 2023-11-05T08:34:09.895 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.015017037, 2023-11-05T08:34:09.895 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:09.895 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:09.895 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:09.896 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.896 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:09.896 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.669028+0000", 2023-11-05T08:34:09.896 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:09.896 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.897 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.897 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:09.897 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.669028+0000", 2023-11-05T08:34:09.897 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:09.897 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.897 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.898 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:09.898 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.669029+0000", 2023-11-05T08:34:09.898 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.7700000000000004e-07 2023-11-05T08:34:09.898 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.898 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.899 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:09.899 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.669081+0000", 2023-11-05T08:34:09.899 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.1851000000000002e-05 2023-11-05T08:34:09.899 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.899 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.899 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:09.900 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.669103+0000", 2023-11-05T08:34:09.900 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2146000000000001e-05 2023-11-05T08:34:09.900 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.900 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.900 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:09.901 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.669120+0000", 2023-11-05T08:34:09.901 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6629999999999998e-05 2023-11-05T08:34:09.901 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.901 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.901 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:09.902 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.673823+0000", 2023-11-05T08:34:09.902 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0047034039999999996 2023-11-05T08:34:09.902 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.902 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.902 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:09.902 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.673850+0000", 2023-11-05T08:34:09.903 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6672000000000001e-05 2023-11-05T08:34:09.903 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.903 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.903 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:09.903 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.683983+0000", 2023-11-05T08:34:09.904 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.010133222000000001 2023-11-05T08:34:09.904 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.904 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.904 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:09.904 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.684045+0000", 2023-11-05T08:34:09.904 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.2335000000000005e-05 2023-11-05T08:34:09.905 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:09.905 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:09.905 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:09.905 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.905 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.906 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.10s5 97/45 ECSubWrite(tid=320, reqid=osd.13.0:320, at_version=97'60, trim_to=0'0, roll_forward_to=93'56))", 2023-11-05T08:34:09.906 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.669144+0000", 2023-11-05T08:34:09.906 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 19.127823013, 2023-11-05T08:34:09.906 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.019507072, 2023-11-05T08:34:09.906 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:09.907 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:09.907 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:09.907 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.907 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:09.907 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.669144+0000", 2023-11-05T08:34:09.907 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:09.908 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.908 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.908 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:09.908 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.669144+0000", 2023-11-05T08:34:09.908 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:09.909 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.909 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.909 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:09.909 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.669144+0000", 2023-11-05T08:34:09.909 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.6300000000000005e-07 2023-11-05T08:34:09.909 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.910 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.910 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:09.910 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.669194+0000", 2023-11-05T08:34:09.910 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.9372000000000001e-05 2023-11-05T08:34:09.910 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.911 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.911 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:09.911 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.669215+0000", 2023-11-05T08:34:09.911 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.158e-05 2023-11-05T08:34:09.911 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.912 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.912 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:09.912 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.669231+0000", 2023-11-05T08:34:09.912 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5892e-05 2023-11-05T08:34:09.912 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.913 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.913 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:09.913 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.678029+0000", 2023-11-05T08:34:09.913 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0087977669999999997 2023-11-05T08:34:09.913 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.914 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.914 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:09.914 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.678056+0000", 2023-11-05T08:34:09.914 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7219000000000001e-05 2023-11-05T08:34:09.914 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.914 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.915 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:09.915 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.688571+0000", 2023-11-05T08:34:09.915 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.010514931999999999 2023-11-05T08:34:09.915 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.915 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.915 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:09.916 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.688651+0000", 2023-11-05T08:34:09.916 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.9647000000000005e-05 2023-11-05T08:34:09.916 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:09.916 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:09.916 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:09.916 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.917 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.917 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.11s5 97/45 ECSubWrite(tid=19, reqid=osd.10.0:19, at_version=97'29, trim_to=0'0, roll_forward_to=94'27))", 2023-11-05T08:34:09.917 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.670156+0000", 2023-11-05T08:34:09.917 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 19.12681031, 2023-11-05T08:34:09.917 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.01292214, 2023-11-05T08:34:09.918 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:09.918 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:09.918 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:09.918 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.918 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:09.919 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.670156+0000", 2023-11-05T08:34:09.919 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:09.919 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.919 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.919 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:09.919 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.670156+0000", 2023-11-05T08:34:09.920 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:09.920 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.920 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.920 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:09.920 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.670158+0000", 2023-11-05T08:34:09.921 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9360000000000002e-06 2023-11-05T08:34:09.921 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.921 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.921 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:09.921 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.670220+0000", 2023-11-05T08:34:09.921 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.1587999999999994e-05 2023-11-05T08:34:09.922 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.922 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.922 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:09.922 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.670252+0000", 2023-11-05T08:34:09.922 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2252000000000002e-05 2023-11-05T08:34:09.923 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.923 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.923 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:09.923 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.670274+0000", 2023-11-05T08:34:09.923 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1549999999999999e-05 2023-11-05T08:34:09.924 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.924 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.924 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:09.924 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.670436+0000", 2023-11-05T08:34:09.924 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00016249 2023-11-05T08:34:09.925 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.925 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.925 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:09.925 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.670461+0000", 2023-11-05T08:34:09.925 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5096000000000002e-05 2023-11-05T08:34:09.926 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.926 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.926 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:09.926 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.683006+0000", 2023-11-05T08:34:09.926 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.012544504 2023-11-05T08:34:09.927 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.927 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.927 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:09.927 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.683078+0000", 2023-11-05T08:34:09.927 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.2724000000000004e-05 2023-11-05T08:34:09.928 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:09.928 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:09.928 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:09.928 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.928 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.928 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.es5 97/30 ECSubWrite(tid=128, reqid=osd.2.0:128, at_version=97'100, trim_to=0'0, roll_forward_to=94'96))", 2023-11-05T08:34:09.929 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.673930+0000", 2023-11-05T08:34:09.929 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 19.123036213999999, 2023-11-05T08:34:09.929 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.014705367, 2023-11-05T08:34:09.929 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:09.929 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:09.930 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:09.930 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.930 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:09.930 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.673930+0000", 2023-11-05T08:34:09.930 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:09.931 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.931 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.931 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:09.933 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.673930+0000", 2023-11-05T08:34:09.933 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:09.934 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.934 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.934 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:09.934 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.673932+0000", 2023-11-05T08:34:09.934 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.4789999999999999e-06 2023-11-05T08:34:09.935 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.935 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.935 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:09.935 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.673982+0000", 2023-11-05T08:34:09.935 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.0355999999999997e-05 2023-11-05T08:34:09.935 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.936 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.936 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:09.936 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.674009+0000", 2023-11-05T08:34:09.936 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6392000000000001e-05 2023-11-05T08:34:09.936 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.936 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.937 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:09.937 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.674029+0000", 2023-11-05T08:34:09.937 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0716e-05 2023-11-05T08:34:09.937 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.937 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.937 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:09.938 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.674184+0000", 2023-11-05T08:34:09.938 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00015448099999999999 2023-11-05T08:34:09.938 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.938 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.938 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:09.939 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.674208+0000", 2023-11-05T08:34:09.939 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4533e-05 2023-11-05T08:34:09.939 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.939 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.939 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:09.939 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.688543+0000", 2023-11-05T08:34:09.940 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.01433482 2023-11-05T08:34:09.940 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.940 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.940 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:09.940 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.688636+0000", 2023-11-05T08:34:09.940 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.2590000000000006e-05 2023-11-05T08:34:09.941 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:09.941 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:09.941 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:09.941 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.941 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.942 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.10s1 97/45 ECSubWrite(tid=321, reqid=osd.13.0:320, at_version=0'0, trim_to=0'0, roll_forward_to=97'60))", 2023-11-05T08:34:09.942 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.699620+0000", 2023-11-05T08:34:09.942 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 19.097346946999998, 2023-11-05T08:34:09.942 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.013813372000000001, 2023-11-05T08:34:09.942 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:09.943 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:09.943 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:09.943 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.943 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:09.943 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.699620+0000", 2023-11-05T08:34:09.943 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:09.944 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.944 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.944 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:09.944 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.699620+0000", 2023-11-05T08:34:09.944 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:09.945 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.945 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.945 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:09.945 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.699621+0000", 2023-11-05T08:34:09.945 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.7460000000000001e-06 2023-11-05T08:34:09.946 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.946 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.946 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:09.946 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.699658+0000", 2023-11-05T08:34:09.946 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.6412000000000003e-05 2023-11-05T08:34:09.946 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.947 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.947 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:09.947 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.699691+0000", 2023-11-05T08:34:09.947 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2656e-05 2023-11-05T08:34:09.947 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.948 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.948 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:09.948 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.699715+0000", 2023-11-05T08:34:09.948 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4148999999999999e-05 2023-11-05T08:34:09.948 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.948 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.949 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:09.949 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.700026+0000", 2023-11-05T08:34:09.949 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00031170900000000002 2023-11-05T08:34:09.949 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.949 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.949 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:09.950 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.700051+0000", 2023-11-05T08:34:09.950 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4556e-05 2023-11-05T08:34:09.950 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.950 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.951 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:09.951 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.713354+0000", 2023-11-05T08:34:09.951 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.01330345 2023-11-05T08:34:09.951 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.951 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.951 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:09.952 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.713433+0000", 2023-11-05T08:34:09.952 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.8694000000000002e-05 2023-11-05T08:34:09.952 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:09.952 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:09.952 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:09.953 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.953 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.953 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.10s5 97/45 ECSubWrite(tid=321, reqid=osd.13.0:320, at_version=0'0, trim_to=0'0, roll_forward_to=97'60))", 2023-11-05T08:34:09.953 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.699776+0000", 2023-11-05T08:34:09.953 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 19.097190402999999, 2023-11-05T08:34:09.953 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.013719832, 2023-11-05T08:34:09.954 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:09.954 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:09.954 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:09.954 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.954 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:09.954 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.699776+0000", 2023-11-05T08:34:09.955 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:09.955 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.955 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.955 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:09.955 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.699776+0000", 2023-11-05T08:34:09.956 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:09.956 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.956 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.956 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:09.956 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.699777+0000", 2023-11-05T08:34:09.956 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.8700000000000004e-07 2023-11-05T08:34:09.957 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.957 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.957 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:09.957 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.699806+0000", 2023-11-05T08:34:09.957 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8626e-05 2023-11-05T08:34:09.957 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.958 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.958 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:09.958 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.699828+0000", 2023-11-05T08:34:09.958 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2602000000000001e-05 2023-11-05T08:34:09.958 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.959 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.959 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:09.959 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.699846+0000", 2023-11-05T08:34:09.959 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.7266999999999999e-05 2023-11-05T08:34:09.959 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.960 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.960 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:09.960 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.700434+0000", 2023-11-05T08:34:09.960 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00058856999999999996 2023-11-05T08:34:09.960 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.960 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.961 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:09.961 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.700467+0000", 2023-11-05T08:34:09.961 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2611999999999999e-05 2023-11-05T08:34:09.961 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.961 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.962 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:09.962 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.713448+0000", 2023-11-05T08:34:09.962 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.012980818 2023-11-05T08:34:09.962 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.962 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.962 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:09.963 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.713496+0000", 2023-11-05T08:34:09.963 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.8350000000000003e-05 2023-11-05T08:34:09.963 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:09.963 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:09.963 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:09.964 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.964 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.964 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.18s1 97/45 ECSubWrite(tid=322, reqid=osd.13.0:319, at_version=0'0, trim_to=0'0, roll_forward_to=97'44))", 2023-11-05T08:34:09.964 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.699885+0000", 2023-11-05T08:34:09.964 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 19.097081616000001, 2023-11-05T08:34:09.965 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.013657423, 2023-11-05T08:34:09.965 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:09.965 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:09.965 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:09.965 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.966 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:09.966 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.699885+0000", 2023-11-05T08:34:09.966 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:09.966 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.966 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.967 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:09.967 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.699885+0000", 2023-11-05T08:34:09.967 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:09.967 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.967 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.967 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:09.968 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.699886+0000", 2023-11-05T08:34:09.968 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.8599999999999997e-07 2023-11-05T08:34:09.968 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.968 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.968 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:09.969 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.699911+0000", 2023-11-05T08:34:09.969 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5355999999999999e-05 2023-11-05T08:34:09.969 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.969 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.969 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:09.969 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.699936+0000", 2023-11-05T08:34:09.970 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5035999999999999e-05 2023-11-05T08:34:09.970 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.970 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.970 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:09.970 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.699954+0000", 2023-11-05T08:34:09.971 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.7535999999999999e-05 2023-11-05T08:34:09.971 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.971 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.971 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:09.971 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.701485+0000", 2023-11-05T08:34:09.972 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001530886 2023-11-05T08:34:09.972 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.972 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.972 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:09.972 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.701507+0000", 2023-11-05T08:34:09.972 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2175999999999999e-05 2023-11-05T08:34:09.973 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.973 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.973 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:09.973 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.713500+0000", 2023-11-05T08:34:09.973 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.011992674 2023-11-05T08:34:09.974 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.974 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.974 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:09.974 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.713542+0000", 2023-11-05T08:34:09.974 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.2972999999999997e-05 2023-11-05T08:34:09.974 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:09.975 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:09.975 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:09.975 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.975 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.975 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.11s5 97/45 ECSubWrite(tid=21, reqid=osd.10.0:20, at_version=0'0, trim_to=0'0, roll_forward_to=97'31))", 2023-11-05T08:34:09.976 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.700121+0000", 2023-11-05T08:34:09.976 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 19.096845387999998, 2023-11-05T08:34:09.976 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.015358910999999999, 2023-11-05T08:34:09.976 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:09.976 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:09.976 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:09.977 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.977 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:09.977 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.700121+0000", 2023-11-05T08:34:09.977 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:09.977 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.977 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.978 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:09.978 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.700121+0000", 2023-11-05T08:34:09.978 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:09.978 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.978 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.979 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:09.979 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.700123+0000", 2023-11-05T08:34:09.979 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.657e-06 2023-11-05T08:34:09.979 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.979 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.979 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:09.980 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.700150+0000", 2023-11-05T08:34:09.980 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7268999999999999e-05 2023-11-05T08:34:09.980 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.980 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.980 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:09.980 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.703027+0000", 2023-11-05T08:34:09.981 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.002876736 2023-11-05T08:34:09.981 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.981 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.981 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:09.981 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.703056+0000", 2023-11-05T08:34:09.981 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.9065e-05 2023-11-05T08:34:09.982 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.982 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.982 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:09.982 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.703188+0000", 2023-11-05T08:34:09.982 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000132008 2023-11-05T08:34:09.983 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.983 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.983 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:09.983 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.703208+0000", 2023-11-05T08:34:09.983 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0387999999999999e-05 2023-11-05T08:34:09.983 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.984 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.984 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:09.984 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.715366+0000", 2023-11-05T08:34:09.984 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.012157340000000001 2023-11-05T08:34:09.984 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.985 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.985 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:09.985 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.715480+0000", 2023-11-05T08:34:09.985 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000114448 2023-11-05T08:34:09.985 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:09.985 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:09.986 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:09.986 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.986 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.986 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.10s1 97/45 ECSubWrite(tid=324, reqid=osd.13.0:324, at_version=97'62, trim_to=0'0, roll_forward_to=97'60))", 2023-11-05T08:34:09.986 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.701788+0000", 2023-11-05T08:34:09.987 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 19.095178536999999, 2023-11-05T08:34:09.987 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.013789007000000001, 2023-11-05T08:34:09.987 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:09.987 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:09.987 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:09.987 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.988 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:09.988 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.701788+0000", 2023-11-05T08:34:09.988 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:09.988 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.988 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.989 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:09.989 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.701788+0000", 2023-11-05T08:34:09.989 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:09.989 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.989 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.990 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:09.990 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.701790+0000", 2023-11-05T08:34:09.990 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9400000000000001e-06 2023-11-05T08:34:09.990 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.990 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.991 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:09.991 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.701853+0000", 2023-11-05T08:34:09.991 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.2471000000000005e-05 2023-11-05T08:34:09.991 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.991 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.991 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:09.992 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.701888+0000", 2023-11-05T08:34:09.992 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.5009000000000003e-05 2023-11-05T08:34:09.992 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.992 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.992 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:09.993 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.701911+0000", 2023-11-05T08:34:09.993 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3136e-05 2023-11-05T08:34:09.993 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.993 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.993 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:09.993 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.702544+0000", 2023-11-05T08:34:09.994 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00063307700000000001 2023-11-05T08:34:09.994 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.994 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.994 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:09.994 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.702565+0000", 2023-11-05T08:34:09.995 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1291e-05 2023-11-05T08:34:09.995 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.995 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.995 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:09.995 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.715516+0000", 2023-11-05T08:34:09.995 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.012951275 2023-11-05T08:34:09.996 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.996 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.996 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:09.996 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.715577+0000", 2023-11-05T08:34:09.996 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.0807999999999998e-05 2023-11-05T08:34:09.997 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:09.997 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:09.997 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:09.997 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:09.997 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.997 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.10s5 97/45 ECSubWrite(tid=324, reqid=osd.13.0:324, at_version=97'62, trim_to=0'0, roll_forward_to=97'60))", 2023-11-05T08:34:09.998 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.701963+0000", 2023-11-05T08:34:09.998 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 19.095004143000001, 2023-11-05T08:34:09.998 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.013656227999999999, 2023-11-05T08:34:09.998 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:09.998 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:09.999 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:09.999 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:09.999 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:09.999 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.701963+0000", 2023-11-05T08:34:09.999 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.000 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.000 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.000 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:10.000 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.701963+0000", 2023-11-05T08:34:10.000 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.001 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.001 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.001 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:10.001 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.701964+0000", 2023-11-05T08:34:10.001 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.026e-06 2023-11-05T08:34:10.002 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.002 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.002 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:10.002 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.702057+0000", 2023-11-05T08:34:10.002 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.3160999999999998e-05 2023-11-05T08:34:10.003 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.003 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.003 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:10.003 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.702082+0000", 2023-11-05T08:34:10.003 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5534999999999999e-05 2023-11-05T08:34:10.003 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.004 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.004 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:10.004 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.702101+0000", 2023-11-05T08:34:10.004 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8490000000000001e-05 2023-11-05T08:34:10.004 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.005 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.005 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:10.005 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.702846+0000", 2023-11-05T08:34:10.005 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00074520999999999995 2023-11-05T08:34:10.005 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.005 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.006 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:10.006 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.702869+0000", 2023-11-05T08:34:10.006 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3394e-05 2023-11-05T08:34:10.006 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.006 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.007 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:10.007 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.715584+0000", 2023-11-05T08:34:10.007 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.012714723000000001 2023-11-05T08:34:10.007 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.007 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.008 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:10.008 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.715619+0000", 2023-11-05T08:34:10.008 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.4688999999999999e-05 2023-11-05T08:34:10.008 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.008 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:10.008 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.009 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.009 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.009 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.18s1 97/45 ECSubWrite(tid=323, reqid=osd.13.0:323, at_version=97'46, trim_to=0'0, roll_forward_to=97'44))", 2023-11-05T08:34:10.009 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.702129+0000", 2023-11-05T08:34:10.009 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 19.094837539, 2023-11-05T08:34:10.010 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.016379057999999998, 2023-11-05T08:34:10.010 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:10.010 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:10.010 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:10.010 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.011 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:10.011 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.702129+0000", 2023-11-05T08:34:10.011 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.011 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.011 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.012 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:10.012 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.702129+0000", 2023-11-05T08:34:10.012 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.012 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.012 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.012 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:10.013 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.702130+0000", 2023-11-05T08:34:10.013 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.4300000000000002e-07 2023-11-05T08:34:10.013 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.013 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.013 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:10.014 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.702175+0000", 2023-11-05T08:34:10.014 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.5062000000000003e-05 2023-11-05T08:34:10.014 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.014 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.014 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:10.015 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.702198+0000", 2023-11-05T08:34:10.015 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3036000000000001e-05 2023-11-05T08:34:10.015 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.015 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.015 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:10.016 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.702218+0000", 2023-11-05T08:34:10.016 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9949000000000002e-05 2023-11-05T08:34:10.016 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.016 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.016 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:10.017 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.707796+0000", 2023-11-05T08:34:10.017 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0055775260000000002 2023-11-05T08:34:10.017 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.017 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.017 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:10.017 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.707818+0000", 2023-11-05T08:34:10.018 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2385e-05 2023-11-05T08:34:10.018 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.018 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.018 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:10.018 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.718457+0000", 2023-11-05T08:34:10.019 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.010639431 2023-11-05T08:34:10.019 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.019 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.019 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:10.019 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.718508+0000", 2023-11-05T08:34:10.020 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.0825999999999998e-05 2023-11-05T08:34:10.020 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.020 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:10.020 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.020 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.020 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.021 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.es5 97/30 ECSubWrite(tid=129, reqid=osd.2.0:128, at_version=0'0, trim_to=0'0, roll_forward_to=97'100))", 2023-11-05T08:34:10.021 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.704114+0000", 2023-11-05T08:34:10.021 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 19.092852951000001, 2023-11-05T08:34:10.021 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.013277532999999999, 2023-11-05T08:34:10.021 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:10.022 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:10.022 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:10.022 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.022 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:10.023 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.704114+0000", 2023-11-05T08:34:10.023 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.023 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.023 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.023 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:10.023 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.704114+0000", 2023-11-05T08:34:10.024 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.024 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.024 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.024 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:10.024 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.704115+0000", 2023-11-05T08:34:10.025 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.68e-06 2023-11-05T08:34:10.025 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.025 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.025 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:10.025 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.704139+0000", 2023-11-05T08:34:10.025 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3238999999999999e-05 2023-11-05T08:34:10.026 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.026 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.026 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:10.026 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.704167+0000", 2023-11-05T08:34:10.026 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8591000000000001e-05 2023-11-05T08:34:10.027 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.027 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.027 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:10.027 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.704186+0000", 2023-11-05T08:34:10.027 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9031000000000001e-05 2023-11-05T08:34:10.028 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.028 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.028 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:10.028 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.705305+0000", 2023-11-05T08:34:10.028 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0011183740000000001 2023-11-05T08:34:10.029 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.029 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.029 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:10.029 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.705329+0000", 2023-11-05T08:34:10.029 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4006e-05 2023-11-05T08:34:10.030 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.030 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.030 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:10.030 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.717332+0000", 2023-11-05T08:34:10.030 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.012003131 2023-11-05T08:34:10.031 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.031 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.031 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:10.031 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.717391+0000", 2023-11-05T08:34:10.031 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.9481000000000002e-05 2023-11-05T08:34:10.032 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.032 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:10.032 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.032 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.032 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:10.033 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:34:10.034 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10: {} 2023-11-05T08:34:10.034 INFO:teuthology.orchestra.run.smithi002.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-11-05T08:34:10.035 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11: {} 2023-11-05T08:34:10.035 INFO:teuthology.orchestra.run.smithi002.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-11-05T08:34:10.036 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2023-11-05T08:34:10.036 INFO:teuthology.orchestra.run.smithi002.stdout:osd.12: {} 2023-11-05T08:34:10.037 INFO:teuthology.orchestra.run.smithi002.stderr:osd.12: osd_enable_op_tracker = 'true' 2023-11-05T08:34:10.037 INFO:teuthology.orchestra.run.smithi002.stdout:osd.13: {} 2023-11-05T08:34:10.038 INFO:teuthology.orchestra.run.smithi002.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-11-05T08:34:10.038 INFO:teuthology.orchestra.run.smithi002.stdout:osd.14: {} 2023-11-05T08:34:10.039 INFO:teuthology.orchestra.run.smithi002.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-11-05T08:34:10.039 INFO:teuthology.orchestra.run.smithi002.stdout:osd.15: {} 2023-11-05T08:34:10.040 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:34:10.031+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:34:10.040 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-05T08:34:10.041 INFO:teuthology.orchestra.run.smithi002.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-11-05T08:34:10.041 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-05T08:34:10.135 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:34:10.131+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:34:10.169 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:34:10.169 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2023-11-05T08:34:10.169 INFO:teuthology.orchestra.run.smithi069.stdout: "num_ops": 0 2023-11-05T08:34:10.169 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:34:10.183 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2023-11-05T08:34:10.236 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:34:10.234+0000 7ffbcd342640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:34:10.303 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:34:10.304 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2023-11-05T08:34:10.304 INFO:teuthology.orchestra.run.smithi069.stdout: "complaint_time": 30, 2023-11-05T08:34:10.304 INFO:teuthology.orchestra.run.smithi069.stdout: "num_blocked_ops": 0 2023-11-05T08:34:10.304 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:34:10.318 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2023-11-05T08:34:10.336 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-11-05T08:34:10.334+0000 7f72b2c5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 42525) UID: 0 2023-11-05T08:34:10.437 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:34:10.434+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:34:10.443 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:34:10.443 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 20, 2023-11-05T08:34:10.444 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 600, 2023-11-05T08:34:10.444 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [ 2023-11-05T08:34:10.444 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.444 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.ds3 36/30 ECSubWrite(tid=24, reqid=osd.7.0:23, at_version=0'0, trim_to=0'0, roll_forward_to=36'14))", 2023-11-05T08:34:10.445 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:49.089743+0000", 2023-11-05T08:34:10.445 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 81.350264588000002, 2023-11-05T08:34:10.445 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0062502979999999996, 2023-11-05T08:34:10.445 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:10.445 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:10.446 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:10.446 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.446 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:10.446 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089743+0000", 2023-11-05T08:34:10.446 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.447 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.447 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.447 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:10.447 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089743+0000", 2023-11-05T08:34:10.447 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.448 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.448 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.448 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:10.448 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089745+0000", 2023-11-05T08:34:10.448 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.034e-06 2023-11-05T08:34:10.449 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.449 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.449 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:10.449 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089768+0000", 2023-11-05T08:34:10.449 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3195000000000001e-05 2023-11-05T08:34:10.450 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.450 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.450 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:10.450 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089791+0000", 2023-11-05T08:34:10.450 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2804999999999999e-05 2023-11-05T08:34:10.451 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.451 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.451 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:10.451 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.089843+0000", 2023-11-05T08:34:10.451 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.27e-05 2023-11-05T08:34:10.452 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.452 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.452 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:10.452 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.090167+0000", 2023-11-05T08:34:10.452 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00032410700000000002 2023-11-05T08:34:10.453 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.453 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.453 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:10.453 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.090192+0000", 2023-11-05T08:34:10.453 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4819000000000001e-05 2023-11-05T08:34:10.453 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.454 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.454 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:10.454 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.095900+0000", 2023-11-05T08:34:10.454 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0057078540000000001 2023-11-05T08:34:10.454 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.455 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.455 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:10.455 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:49.095994+0000", 2023-11-05T08:34:10.455 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.3783999999999995e-05 2023-11-05T08:34:10.455 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.456 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:10.456 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.456 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.456 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.456 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.11s4 41/40 ECSubWrite(tid=65, reqid=client.4691.0:287, at_version=41'13, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:34:10.457 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.173904+0000", 2023-11-05T08:34:10.457 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 76.266104373999994, 2023-11-05T08:34:10.457 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0062957869999999997, 2023-11-05T08:34:10.457 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:10.457 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:10.457 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:10.458 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.458 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:10.458 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.173904+0000", 2023-11-05T08:34:10.458 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.458 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.459 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.459 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:10.459 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.173904+0000", 2023-11-05T08:34:10.459 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.459 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.460 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.460 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:10.460 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.173905+0000", 2023-11-05T08:34:10.460 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.68e-06 2023-11-05T08:34:10.460 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.460 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.461 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:10.461 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.174701+0000", 2023-11-05T08:34:10.461 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00079553000000000002 2023-11-05T08:34:10.461 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.461 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.461 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:10.462 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.174739+0000", 2023-11-05T08:34:10.462 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.7605999999999999e-05 2023-11-05T08:34:10.462 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.462 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.462 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:10.463 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.174758+0000", 2023-11-05T08:34:10.463 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9403999999999999e-05 2023-11-05T08:34:10.463 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.463 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.463 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:10.463 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.174938+0000", 2023-11-05T08:34:10.464 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00018046800000000001 2023-11-05T08:34:10.464 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.464 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.464 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:10.464 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.174959+0000", 2023-11-05T08:34:10.464 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0471000000000001e-05 2023-11-05T08:34:10.465 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.465 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.465 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:10.465 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.180140+0000", 2023-11-05T08:34:10.465 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0051811080000000002 2023-11-05T08:34:10.466 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.466 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.466 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:10.466 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.180199+0000", 2023-11-05T08:34:10.466 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.9519999999999999e-05 2023-11-05T08:34:10.467 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.467 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:10.467 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.467 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.467 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.467 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.9s6 52/45 ECSubWrite(tid=75, reqid=client.4691.0:341, at_version=52'24, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:34:10.468 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:05.258033+0000", 2023-11-05T08:34:10.468 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 65.181974710000006, 2023-11-05T08:34:10.468 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0062192000000000002, 2023-11-05T08:34:10.468 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:10.468 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:10.469 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:10.469 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.469 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:10.469 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.258033+0000", 2023-11-05T08:34:10.469 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.469 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.470 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.470 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:10.470 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.258033+0000", 2023-11-05T08:34:10.470 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.470 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.471 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.471 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:10.471 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.258034+0000", 2023-11-05T08:34:10.471 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.1299999999999999e-07 2023-11-05T08:34:10.471 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.472 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.472 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:10.472 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.258165+0000", 2023-11-05T08:34:10.472 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000131221 2023-11-05T08:34:10.472 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.472 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.473 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:10.473 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.258202+0000", 2023-11-05T08:34:10.473 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.6276000000000003e-05 2023-11-05T08:34:10.473 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.473 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.474 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:10.474 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.258225+0000", 2023-11-05T08:34:10.474 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3204999999999999e-05 2023-11-05T08:34:10.474 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.474 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.475 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:10.475 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.258480+0000", 2023-11-05T08:34:10.475 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00025507299999999998 2023-11-05T08:34:10.475 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.475 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.476 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:10.476 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.258512+0000", 2023-11-05T08:34:10.476 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2008999999999998e-05 2023-11-05T08:34:10.476 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.476 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.477 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:10.477 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.264164+0000", 2023-11-05T08:34:10.477 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0056518360000000004 2023-11-05T08:34:10.477 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.477 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.478 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:10.478 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.264253+0000", 2023-11-05T08:34:10.478 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.8766999999999999e-05 2023-11-05T08:34:10.478 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.478 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:10.479 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.479 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.479 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.479 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:481 3.11s0 3:88a306be:::smithi09039252-16:7 [read 0~262144,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2023-11-05T08:34:10.479 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:18.555634+0000", 2023-11-05T08:34:10.480 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 51.884373754000002, 2023-11-05T08:34:10.480 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0065275120000000001, 2023-11-05T08:34:10.480 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:10.480 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2023-11-05T08:34:10.480 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:34:10.480 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:34:10.481 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:10.481 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 481 2023-11-05T08:34:10.481 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.481 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:10.481 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.482 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:10.482 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555634+0000", 2023-11-05T08:34:10.482 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.482 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.482 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.482 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:10.483 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555634+0000", 2023-11-05T08:34:10.483 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.483 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.483 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.483 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:10.484 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555636+0000", 2023-11-05T08:34:10.484 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2860000000000001e-06 2023-11-05T08:34:10.484 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.484 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.484 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:10.484 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555642+0000", 2023-11-05T08:34:10.485 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.5810000000000002e-06 2023-11-05T08:34:10.485 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.485 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.485 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:10.485 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555658+0000", 2023-11-05T08:34:10.485 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5875999999999998e-05 2023-11-05T08:34:10.486 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.486 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.486 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:10.486 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555667+0000", 2023-11-05T08:34:10.486 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.9530000000000005e-06 2023-11-05T08:34:10.487 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.487 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.487 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:10.487 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555826+0000", 2023-11-05T08:34:10.487 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000158834 2023-11-05T08:34:10.487 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.488 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.488 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:34:10.488 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.556071+0000", 2023-11-05T08:34:10.488 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000245441 2023-11-05T08:34:10.488 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.489 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.489 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:10.489 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.562162+0000", 2023-11-05T08:34:10.489 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0060905409999999997 2023-11-05T08:34:10.489 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.490 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:10.490 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.490 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.490 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.490 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:482 3.11s0 3:88a306be:::smithi09039252-16:7 [read 0~262144,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2023-11-05T08:34:10.491 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:18.555691+0000", 2023-11-05T08:34:10.491 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 51.884316888999997, 2023-11-05T08:34:10.491 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.006823389, 2023-11-05T08:34:10.491 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:10.491 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2023-11-05T08:34:10.492 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:34:10.492 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:34:10.492 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:10.492 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 482 2023-11-05T08:34:10.492 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.492 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:10.493 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.493 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:10.493 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555691+0000", 2023-11-05T08:34:10.493 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.493 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.494 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.494 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:10.494 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555691+0000", 2023-11-05T08:34:10.494 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.494 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.495 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.495 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:10.495 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555692+0000", 2023-11-05T08:34:10.495 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.6199999999999998e-07 2023-11-05T08:34:10.495 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.495 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.496 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:10.496 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555695+0000", 2023-11-05T08:34:10.496 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.7220000000000001e-06 2023-11-05T08:34:10.496 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.496 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.496 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:10.497 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555703+0000", 2023-11-05T08:34:10.497 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.9689999999999994e-06 2023-11-05T08:34:10.497 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.497 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.497 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:10.498 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555709+0000", 2023-11-05T08:34:10.498 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.6420000000000004e-06 2023-11-05T08:34:10.498 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.498 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.498 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:10.498 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.556434+0000", 2023-11-05T08:34:10.499 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00072455700000000002 2023-11-05T08:34:10.499 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.499 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.499 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:34:10.499 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.556542+0000", 2023-11-05T08:34:10.499 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0001081 2023-11-05T08:34:10.500 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.500 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.500 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:10.500 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.562515+0000", 2023-11-05T08:34:10.500 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0059728369999999999 2023-11-05T08:34:10.500 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.501 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:10.501 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.501 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.501 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.501 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:483 3.11s0 3:88a306be:::smithi09039252-16:7 [read 0~262144,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2023-11-05T08:34:10.501 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:18.555880+0000", 2023-11-05T08:34:10.502 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 51.884127972000002, 2023-11-05T08:34:10.502 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0070107930000000004, 2023-11-05T08:34:10.502 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:10.502 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2023-11-05T08:34:10.502 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:34:10.503 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:34:10.503 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:10.503 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 483 2023-11-05T08:34:10.503 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.503 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:10.503 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.504 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:10.504 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555880+0000", 2023-11-05T08:34:10.504 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.504 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.504 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.504 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:10.505 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555880+0000", 2023-11-05T08:34:10.505 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.505 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.505 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.505 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:10.505 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555886+0000", 2023-11-05T08:34:10.506 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.1129999999999997e-06 2023-11-05T08:34:10.506 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.506 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.506 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:10.506 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.555900+0000", 2023-11-05T08:34:10.507 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3769e-05 2023-11-05T08:34:10.507 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.507 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.507 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:10.507 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.556103+0000", 2023-11-05T08:34:10.508 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00020306400000000001 2023-11-05T08:34:10.508 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.508 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.508 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:10.508 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.556130+0000", 2023-11-05T08:34:10.508 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6466e-05 2023-11-05T08:34:10.509 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.509 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.509 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:10.509 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.558343+0000", 2023-11-05T08:34:10.509 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.002213207 2023-11-05T08:34:10.509 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.510 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.510 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:34:10.510 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.558458+0000", 2023-11-05T08:34:10.510 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000115773 2023-11-05T08:34:10.510 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.511 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.511 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:10.511 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:18.562891+0000", 2023-11-05T08:34:10.511 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0044324009999999999 2023-11-05T08:34:10.511 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.512 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:10.512 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.512 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.512 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.512 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.16s6 66/45 ECSubWrite(tid=112, reqid=osd.11.0:112, at_version=66'34, trim_to=0'0, roll_forward_to=65'32))", 2023-11-05T08:34:10.512 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:19.401233+0000", 2023-11-05T08:34:10.513 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 51.038774811000003, 2023-11-05T08:34:10.513 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.013005030000000001, 2023-11-05T08:34:10.513 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:10.513 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:10.513 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:10.514 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.514 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:10.514 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.401233+0000", 2023-11-05T08:34:10.514 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.514 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.514 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.515 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:10.515 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.401233+0000", 2023-11-05T08:34:10.515 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.515 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.515 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.516 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:10.516 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.401234+0000", 2023-11-05T08:34:10.516 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.1999999999999999e-07 2023-11-05T08:34:10.516 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.516 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.517 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:10.517 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.401290+0000", 2023-11-05T08:34:10.517 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.5544999999999998e-05 2023-11-05T08:34:10.517 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.517 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.517 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:10.518 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.401316+0000", 2023-11-05T08:34:10.518 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6889e-05 2023-11-05T08:34:10.518 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.518 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.518 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:10.519 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.401343+0000", 2023-11-05T08:34:10.519 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6239000000000001e-05 2023-11-05T08:34:10.519 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.519 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.519 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:10.520 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.401551+0000", 2023-11-05T08:34:10.520 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00020812399999999999 2023-11-05T08:34:10.520 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.520 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.520 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:10.520 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.401581+0000", 2023-11-05T08:34:10.521 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0508999999999999e-05 2023-11-05T08:34:10.521 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.521 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.521 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:10.521 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.414133+0000", 2023-11-05T08:34:10.522 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.012551639999999999 2023-11-05T08:34:10.522 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.522 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.522 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:10.522 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:19.414238+0000", 2023-11-05T08:34:10.522 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000105364 2023-11-05T08:34:10.523 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.523 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:10.523 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.523 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.523 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.524 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.1s4 97/40 ECSubWrite(tid=140, reqid=osd.4.0:140, at_version=97'28, trim_to=0'0, roll_forward_to=66'26))", 2023-11-05T08:34:10.524 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.664721+0000", 2023-11-05T08:34:10.524 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 19.775287528, 2023-11-05T08:34:10.524 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.021386919000000001, 2023-11-05T08:34:10.524 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:10.524 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:10.525 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:10.525 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.525 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:10.525 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.664721+0000", 2023-11-05T08:34:10.525 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.525 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.526 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.526 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:10.526 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.664721+0000", 2023-11-05T08:34:10.526 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.526 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.527 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.527 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:10.527 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.664721+0000", 2023-11-05T08:34:10.527 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.8999999999999995e-07 2023-11-05T08:34:10.527 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.527 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.528 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:10.528 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.664771+0000", 2023-11-05T08:34:10.528 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.0002e-05 2023-11-05T08:34:10.528 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.528 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.528 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:10.529 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.664806+0000", 2023-11-05T08:34:10.529 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.4285000000000001e-05 2023-11-05T08:34:10.529 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.529 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.529 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:10.530 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.664857+0000", 2023-11-05T08:34:10.530 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.1476e-05 2023-11-05T08:34:10.530 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.530 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.530 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:10.530 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.669736+0000", 2023-11-05T08:34:10.531 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0048794049999999999 2023-11-05T08:34:10.531 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.531 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.531 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:10.531 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.669763+0000", 2023-11-05T08:34:10.531 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6571999999999999e-05 2023-11-05T08:34:10.532 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.532 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.532 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:10.532 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.686021+0000", 2023-11-05T08:34:10.532 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.016257882000000001 2023-11-05T08:34:10.532 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.533 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.533 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:10.533 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.686107+0000", 2023-11-05T08:34:10.533 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.6507000000000004e-05 2023-11-05T08:34:10.533 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.533 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:10.534 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.534 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.534 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.534 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 97/45 ECSubWriteReply(tid=19, last_complete=97'29, committed=1, applied=1))", 2023-11-05T08:34:10.534 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.672139+0000", 2023-11-05T08:34:10.535 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 19.767868580999998, 2023-11-05T08:34:10.535 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.01364305, 2023-11-05T08:34:10.535 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:10.535 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:10.535 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:10.536 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.536 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:10.536 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.672139+0000", 2023-11-05T08:34:10.536 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.536 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.537 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.537 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:10.537 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.672139+0000", 2023-11-05T08:34:10.537 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.537 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.537 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.538 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:10.538 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.672141+0000", 2023-11-05T08:34:10.538 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.793e-06 2023-11-05T08:34:10.538 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.538 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.539 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:10.539 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.672151+0000", 2023-11-05T08:34:10.539 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.6630000000000005e-06 2023-11-05T08:34:10.539 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.540 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.540 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:10.540 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.672181+0000", 2023-11-05T08:34:10.540 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0165e-05 2023-11-05T08:34:10.540 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.541 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.541 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:10.541 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.673687+0000", 2023-11-05T08:34:10.541 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001505473 2023-11-05T08:34:10.541 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.541 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.542 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:10.542 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.685718+0000", 2023-11-05T08:34:10.542 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.012031714000000001 2023-11-05T08:34:10.542 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.542 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.543 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:10.543 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.685783+0000", 2023-11-05T08:34:10.543 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.4241999999999999e-05 2023-11-05T08:34:10.543 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.543 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:10.544 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.544 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.544 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.544 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 97/45 ECSubWriteReply(tid=19, last_complete=97'29, committed=1, applied=1))", 2023-11-05T08:34:10.544 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.674303+0000", 2023-11-05T08:34:10.544 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 19.765704924000001, 2023-11-05T08:34:10.545 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.011878712, 2023-11-05T08:34:10.545 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:10.545 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:10.545 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:10.545 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.546 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:10.546 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.674303+0000", 2023-11-05T08:34:10.546 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.546 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.546 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.547 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:10.547 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.674303+0000", 2023-11-05T08:34:10.547 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.547 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.548 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.548 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:10.548 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.674305+0000", 2023-11-05T08:34:10.548 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.773e-06 2023-11-05T08:34:10.548 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.548 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.549 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:10.549 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.674314+0000", 2023-11-05T08:34:10.549 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.1759999999999999e-06 2023-11-05T08:34:10.549 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.549 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.550 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:10.550 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.674338+0000", 2023-11-05T08:34:10.550 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4366e-05 2023-11-05T08:34:10.550 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.550 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.551 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:10.551 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.674357+0000", 2023-11-05T08:34:10.551 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8953e-05 2023-11-05T08:34:10.551 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.551 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.552 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:10.552 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.685891+0000", 2023-11-05T08:34:10.552 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.011533933999999999 2023-11-05T08:34:10.552 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.552 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.552 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:10.553 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.686182+0000", 2023-11-05T08:34:10.553 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00029051000000000001 2023-11-05T08:34:10.553 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.553 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:10.553 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.554 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.554 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.554 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.13s6 97/45 ECSubWrite(tid=168, reqid=osd.11.0:168, at_version=97'25, trim_to=0'0, roll_forward_to=92'23))", 2023-11-05T08:34:10.554 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.675582+0000", 2023-11-05T08:34:10.554 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 19.764426297, 2023-11-05T08:34:10.555 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.012261456, 2023-11-05T08:34:10.555 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:10.555 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:10.555 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:10.555 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.556 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:10.556 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.675582+0000", 2023-11-05T08:34:10.556 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.556 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.556 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.557 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:10.557 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.675582+0000", 2023-11-05T08:34:10.557 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.557 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.557 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.557 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:10.558 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.675584+0000", 2023-11-05T08:34:10.558 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0959999999999999e-06 2023-11-05T08:34:10.558 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.558 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.558 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:10.559 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.675648+0000", 2023-11-05T08:34:10.559 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.4271999999999997e-05 2023-11-05T08:34:10.559 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.559 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.559 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:10.560 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.675679+0000", 2023-11-05T08:34:10.560 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0839000000000001e-05 2023-11-05T08:34:10.560 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.560 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.560 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:10.560 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.675701+0000", 2023-11-05T08:34:10.561 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2019e-05 2023-11-05T08:34:10.561 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.561 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.561 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:10.561 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.675855+0000", 2023-11-05T08:34:10.561 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00015368099999999999 2023-11-05T08:34:10.562 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.562 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.562 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:10.562 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.675881+0000", 2023-11-05T08:34:10.562 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6159000000000001e-05 2023-11-05T08:34:10.563 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.563 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.563 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:10.563 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.687753+0000", 2023-11-05T08:34:10.563 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.01187205 2023-11-05T08:34:10.564 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.564 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.564 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:10.564 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.687843+0000", 2023-11-05T08:34:10.564 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.0340000000000006e-05 2023-11-05T08:34:10.565 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.565 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:10.565 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.565 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.565 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.565 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.16s6 97/45 ECSubWrite(tid=169, reqid=osd.11.0:169, at_version=97'48, trim_to=0'0, roll_forward_to=92'46))", 2023-11-05T08:34:10.566 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.675789+0000", 2023-11-05T08:34:10.566 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 19.764218798000002, 2023-11-05T08:34:10.566 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.011724231999999999, 2023-11-05T08:34:10.566 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:10.566 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:10.567 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:10.567 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.567 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:10.567 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.675789+0000", 2023-11-05T08:34:10.567 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.568 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.568 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.568 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:10.568 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.675789+0000", 2023-11-05T08:34:10.568 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.568 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.569 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.569 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:10.569 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.675791+0000", 2023-11-05T08:34:10.569 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2699999999999999e-06 2023-11-05T08:34:10.569 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.570 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.570 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:10.570 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.675833+0000", 2023-11-05T08:34:10.570 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.2852000000000002e-05 2023-11-05T08:34:10.570 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.570 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.571 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:10.571 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.675859+0000", 2023-11-05T08:34:10.571 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5360000000000001e-05 2023-11-05T08:34:10.571 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.571 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.572 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:10.572 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.675884+0000", 2023-11-05T08:34:10.572 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4915000000000001e-05 2023-11-05T08:34:10.572 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.572 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.572 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:10.573 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.676018+0000", 2023-11-05T08:34:10.573 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00013400200000000001 2023-11-05T08:34:10.573 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.573 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.573 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:10.574 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.676038+0000", 2023-11-05T08:34:10.574 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0610000000000001e-05 2023-11-05T08:34:10.574 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.574 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.574 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:10.574 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.687410+0000", 2023-11-05T08:34:10.575 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.011371576 2023-11-05T08:34:10.575 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.575 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.575 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:10.575 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.687513+0000", 2023-11-05T08:34:10.576 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00010364700000000001 2023-11-05T08:34:10.576 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.576 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:10.576 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.576 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.577 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.577 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 97/45 ECSubWriteReply(tid=19, last_complete=97'29, committed=1, applied=1))", 2023-11-05T08:34:10.577 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.677080+0000", 2023-11-05T08:34:10.577 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 19.762927877999999, 2023-11-05T08:34:10.577 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0092669310000000008, 2023-11-05T08:34:10.577 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:10.578 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:10.578 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:10.578 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.578 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:10.578 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.677080+0000", 2023-11-05T08:34:10.579 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.579 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.579 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.579 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:10.579 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.677080+0000", 2023-11-05T08:34:10.580 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.580 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.580 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.580 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:10.580 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.677082+0000", 2023-11-05T08:34:10.580 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.607e-06 2023-11-05T08:34:10.581 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.581 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.581 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:10.581 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.677091+0000", 2023-11-05T08:34:10.581 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.2059999999999996e-06 2023-11-05T08:34:10.582 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.582 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.582 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:10.582 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.677120+0000", 2023-11-05T08:34:10.582 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.9488999999999998e-05 2023-11-05T08:34:10.582 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.583 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.583 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:10.583 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.677140+0000", 2023-11-05T08:34:10.583 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9959999999999999e-05 2023-11-05T08:34:10.583 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.584 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.584 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:10.584 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.686281+0000", 2023-11-05T08:34:10.584 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0091408340000000005 2023-11-05T08:34:10.584 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.585 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.585 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:10.585 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.686347+0000", 2023-11-05T08:34:10.585 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.5834999999999996e-05 2023-11-05T08:34:10.585 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.585 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:10.586 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.586 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.586 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.586 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.17s2 97/96 ECSubWrite(tid=56, reqid=osd.5.0:56, at_version=97'23, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:34:10.586 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.680845+0000", 2023-11-05T08:34:10.587 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 19.759163371, 2023-11-05T08:34:10.587 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.011722811, 2023-11-05T08:34:10.587 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:10.587 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:10.587 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:10.588 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.588 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:10.588 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.680845+0000", 2023-11-05T08:34:10.588 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.588 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.588 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.589 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:10.589 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.680845+0000", 2023-11-05T08:34:10.589 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.589 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.589 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.590 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:10.590 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.680847+0000", 2023-11-05T08:34:10.590 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4959999999999999e-06 2023-11-05T08:34:10.590 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.590 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.591 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:10.591 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.680903+0000", 2023-11-05T08:34:10.591 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.5685e-05 2023-11-05T08:34:10.591 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.591 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.591 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:10.592 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.680938+0000", 2023-11-05T08:34:10.592 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.5098999999999997e-05 2023-11-05T08:34:10.592 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.592 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.592 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:10.593 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.680964+0000", 2023-11-05T08:34:10.593 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5735999999999999e-05 2023-11-05T08:34:10.593 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.593 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.593 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:10.594 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.681116+0000", 2023-11-05T08:34:10.594 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00015194399999999999 2023-11-05T08:34:10.594 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.594 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.594 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:10.595 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.681142+0000", 2023-11-05T08:34:10.595 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6733e-05 2023-11-05T08:34:10.595 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.595 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.595 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:10.595 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.692435+0000", 2023-11-05T08:34:10.596 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.011292224999999999 2023-11-05T08:34:10.596 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.596 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.596 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:10.596 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.692568+0000", 2023-11-05T08:34:10.597 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000132893 2023-11-05T08:34:10.597 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.597 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:10.597 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.597 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.598 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.598 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.17s2 97/96 ECSubWrite(tid=57, reqid=osd.5.0:57, at_version=97'25, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:34:10.598 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.684852+0000", 2023-11-05T08:34:10.598 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 19.755156469999999, 2023-11-05T08:34:10.598 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.012092159, 2023-11-05T08:34:10.599 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:10.599 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:10.599 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:10.599 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.599 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:10.599 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.684852+0000", 2023-11-05T08:34:10.600 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.600 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.600 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.600 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:10.600 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.684852+0000", 2023-11-05T08:34:10.601 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.601 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.601 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.601 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:10.601 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.684853+0000", 2023-11-05T08:34:10.601 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8059999999999999e-06 2023-11-05T08:34:10.602 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.602 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.602 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:10.602 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.684905+0000", 2023-11-05T08:34:10.602 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.1715000000000002e-05 2023-11-05T08:34:10.603 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.603 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.603 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:10.603 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.684936+0000", 2023-11-05T08:34:10.603 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0913e-05 2023-11-05T08:34:10.603 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.604 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.604 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:10.604 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.684958+0000", 2023-11-05T08:34:10.604 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2232e-05 2023-11-05T08:34:10.604 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.604 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.605 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:10.605 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.686136+0000", 2023-11-05T08:34:10.605 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001178194 2023-11-05T08:34:10.605 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.605 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.605 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:10.606 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.686160+0000", 2023-11-05T08:34:10.606 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3462e-05 2023-11-05T08:34:10.606 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.606 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.606 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:10.607 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.696852+0000", 2023-11-05T08:34:10.607 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.010692313 2023-11-05T08:34:10.607 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.607 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.607 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:10.607 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.696944+0000", 2023-11-05T08:34:10.608 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.1524e-05 2023-11-05T08:34:10.608 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.608 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:10.608 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.608 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.608 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.609 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.1s4 97/40 ECSubWrite(tid=141, reqid=osd.4.0:140, at_version=0'0, trim_to=0'0, roll_forward_to=97'28))", 2023-11-05T08:34:10.609 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.687254+0000", 2023-11-05T08:34:10.609 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 19.752754160999999, 2023-11-05T08:34:10.609 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0099101720000000001, 2023-11-05T08:34:10.609 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:10.609 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:10.610 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:10.610 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.610 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:10.610 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.687254+0000", 2023-11-05T08:34:10.610 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.611 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.611 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.611 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:10.611 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.687254+0000", 2023-11-05T08:34:10.611 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.611 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.612 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.612 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:10.612 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.687256+0000", 2023-11-05T08:34:10.612 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9199999999999998e-06 2023-11-05T08:34:10.612 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.612 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.613 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:10.613 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.687284+0000", 2023-11-05T08:34:10.613 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7688999999999999e-05 2023-11-05T08:34:10.613 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.613 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.613 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:10.614 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.687316+0000", 2023-11-05T08:34:10.614 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2212000000000003e-05 2023-11-05T08:34:10.614 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.614 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.614 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:10.615 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.687338+0000", 2023-11-05T08:34:10.615 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2025999999999999e-05 2023-11-05T08:34:10.615 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.615 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.615 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:10.615 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.687703+0000", 2023-11-05T08:34:10.616 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00036512399999999998 2023-11-05T08:34:10.616 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.616 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.616 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:10.616 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.687727+0000", 2023-11-05T08:34:10.617 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4453000000000001e-05 2023-11-05T08:34:10.617 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.617 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.617 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:10.617 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.696960+0000", 2023-11-05T08:34:10.618 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0092326390000000008 2023-11-05T08:34:10.618 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.618 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.618 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:10.618 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.697164+0000", 2023-11-05T08:34:10.619 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00020410900000000001 2023-11-05T08:34:10.619 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.619 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:10.619 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.619 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.619 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.620 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.16s6 97/45 ECSubWrite(tid=170, reqid=osd.11.0:169, at_version=0'0, trim_to=0'0, roll_forward_to=97'48))", 2023-11-05T08:34:10.620 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.694898+0000", 2023-11-05T08:34:10.620 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 19.745110079, 2023-11-05T08:34:10.620 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0067628150000000001, 2023-11-05T08:34:10.620 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:10.621 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:10.621 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:10.621 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.621 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:10.621 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.694898+0000", 2023-11-05T08:34:10.622 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.622 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.622 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.622 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:10.622 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.694898+0000", 2023-11-05T08:34:10.622 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.623 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.623 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.623 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:10.623 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.694900+0000", 2023-11-05T08:34:10.623 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.553e-06 2023-11-05T08:34:10.623 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.624 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.624 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:10.624 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.694921+0000", 2023-11-05T08:34:10.624 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1933000000000002e-05 2023-11-05T08:34:10.624 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.625 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.625 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:10.625 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.694965+0000", 2023-11-05T08:34:10.625 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.3738e-05 2023-11-05T08:34:10.625 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.626 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.626 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:10.626 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.694990+0000", 2023-11-05T08:34:10.626 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4686e-05 2023-11-05T08:34:10.626 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.627 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.627 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:10.627 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.695152+0000", 2023-11-05T08:34:10.627 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00016206200000000001 2023-11-05T08:34:10.627 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.627 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.628 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:10.628 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.695182+0000", 2023-11-05T08:34:10.628 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.9632e-05 2023-11-05T08:34:10.628 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.628 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.629 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:10.629 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.701577+0000", 2023-11-05T08:34:10.629 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0063955909999999999 2023-11-05T08:34:10.629 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.629 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.629 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:10.630 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.701661+0000", 2023-11-05T08:34:10.630 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.3620000000000002e-05 2023-11-05T08:34:10.630 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.630 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:10.630 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.630 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.631 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.631 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.13s6 97/45 ECSubWrite(tid=171, reqid=osd.11.0:168, at_version=0'0, trim_to=0'0, roll_forward_to=97'25))", 2023-11-05T08:34:10.631 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:50.709051+0000", 2023-11-05T08:34:10.631 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 19.730957262, 2023-11-05T08:34:10.631 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0072026249999999998, 2023-11-05T08:34:10.631 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:10.632 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:10.632 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:10.632 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.632 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:10.632 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.709051+0000", 2023-11-05T08:34:10.632 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.633 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.633 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.633 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:10.633 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.709051+0000", 2023-11-05T08:34:10.633 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.633 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.634 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.634 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:10.634 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.709052+0000", 2023-11-05T08:34:10.634 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5370000000000001e-06 2023-11-05T08:34:10.634 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.634 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.635 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:10.635 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.709081+0000", 2023-11-05T08:34:10.635 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8407000000000001e-05 2023-11-05T08:34:10.635 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.635 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.636 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:10.636 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.709109+0000", 2023-11-05T08:34:10.636 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8302000000000001e-05 2023-11-05T08:34:10.636 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.636 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.637 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:10.637 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.709129+0000", 2023-11-05T08:34:10.637 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.0145000000000001e-05 2023-11-05T08:34:10.637 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.637 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.638 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:10.638 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.709325+0000", 2023-11-05T08:34:10.638 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00019577900000000001 2023-11-05T08:34:10.638 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.638 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.638 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:10.639 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.709348+0000", 2023-11-05T08:34:10.639 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3144999999999999e-05 2023-11-05T08:34:10.639 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.639 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.639 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:10.640 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.716173+0000", 2023-11-05T08:34:10.640 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.006824681 2023-11-05T08:34:10.640 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.640 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.640 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:10.641 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:50.716253+0000", 2023-11-05T08:34:10.641 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.0629000000000003e-05 2023-11-05T08:34:10.641 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.641 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:10.641 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.641 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.642 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.642 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.fs5 101/37 ECSubWrite(tid=172, reqid=client.4691.0:775, at_version=101'59, trim_to=0'0, roll_forward_to=90'57))", 2023-11-05T08:34:10.642 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:54.762525+0000", 2023-11-05T08:34:10.642 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 15.677483147, 2023-11-05T08:34:10.642 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0084539679999999992, 2023-11-05T08:34:10.643 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:10.643 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:10.643 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:10.643 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.643 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:10.643 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.762525+0000", 2023-11-05T08:34:10.644 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.644 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.644 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.644 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:10.644 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.762525+0000", 2023-11-05T08:34:10.644 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.645 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.645 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.645 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:10.645 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.762526+0000", 2023-11-05T08:34:10.645 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.6300000000000004e-07 2023-11-05T08:34:10.646 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.646 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.646 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:10.646 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.762901+0000", 2023-11-05T08:34:10.646 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00037554800000000002 2023-11-05T08:34:10.646 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.647 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.647 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:10.647 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.762930+0000", 2023-11-05T08:34:10.647 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8586e-05 2023-11-05T08:34:10.647 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.648 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.648 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:10.648 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.762953+0000", 2023-11-05T08:34:10.648 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2966999999999999e-05 2023-11-05T08:34:10.648 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.649 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.649 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:10.649 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.763124+0000", 2023-11-05T08:34:10.649 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00017125699999999999 2023-11-05T08:34:10.649 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.650 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.650 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:10.650 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.763155+0000", 2023-11-05T08:34:10.650 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.1415999999999999e-05 2023-11-05T08:34:10.650 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.651 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.651 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:10.651 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.770896+0000", 2023-11-05T08:34:10.651 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0077405260000000002 2023-11-05T08:34:10.651 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.652 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.652 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:10.652 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.770979+0000", 2023-11-05T08:34:10.652 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.2905000000000007e-05 2023-11-05T08:34:10.652 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.652 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:10.653 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.653 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.653 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.653 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.fs5 101/37 ECSubWrite(tid=175, reqid=client.4691.0:778, at_version=101'62, trim_to=0'0, roll_forward_to=101'59))", 2023-11-05T08:34:10.653 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:54.775586+0000", 2023-11-05T08:34:10.654 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 15.664422058, 2023-11-05T08:34:10.654 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0063911539999999996, 2023-11-05T08:34:10.654 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:10.654 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:10.654 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:10.655 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.655 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:10.655 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.775586+0000", 2023-11-05T08:34:10.655 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.655 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.655 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.656 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:10.656 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.775586+0000", 2023-11-05T08:34:10.656 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:10.656 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.656 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.657 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:10.657 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.775587+0000", 2023-11-05T08:34:10.657 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.367e-06 2023-11-05T08:34:10.657 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.657 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.658 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:10.658 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.777730+0000", 2023-11-05T08:34:10.658 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0021425369999999999 2023-11-05T08:34:10.658 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.658 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.659 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:10.659 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.777766+0000", 2023-11-05T08:34:10.659 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.6056000000000002e-05 2023-11-05T08:34:10.659 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.659 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.660 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:10.660 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.777789+0000", 2023-11-05T08:34:10.660 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2969e-05 2023-11-05T08:34:10.660 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.660 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.661 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:10.661 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.777942+0000", 2023-11-05T08:34:10.661 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000152669 2023-11-05T08:34:10.661 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.661 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.661 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:10.662 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.777967+0000", 2023-11-05T08:34:10.662 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5686000000000001e-05 2023-11-05T08:34:10.662 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.662 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.662 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:10.663 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.781906+0000", 2023-11-05T08:34:10.663 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.003938932 2023-11-05T08:34:10.663 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:10.663 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:10.663 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:10.664 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.781977+0000", 2023-11-05T08:34:10.664 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.0938000000000004e-05 2023-11-05T08:34:10.664 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.664 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:10.664 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.664 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:10.665 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:10.665 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:34:10.665 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-05T08:34:10.666 INFO:tasks.rados.rados.0.smithi090.stdout:399: done (5 left) 2023-11-05T08:34:10.667 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-11-05T08:34:10.667 INFO:tasks.rados.rados.0.smithi090.stdout:392: expect (ObjNum 3 snap 0 seq_num 3) 2023-11-05T08:34:10.668 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2023-11-05T08:34:10.682 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 22 v 38 (ObjNum 21 snap 0 seq_num 21) dirty exists 2023-11-05T08:34:10.682 INFO:tasks.rados.rados.0.smithi090.stdout:393: finishing copy_from racing read to smithi09039252-44 2023-11-05T08:34:10.683 INFO:tasks.rados.rados.0.smithi090.stdout:393: finishing copy_from to smithi09039252-44 2023-11-05T08:34:10.683 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 44 v 58 (ObjNum 141 snap 38 seq_num 141) dirty exists 2023-11-05T08:34:10.683 INFO:tasks.rados.rados.0.smithi090.stdout:396: expect (ObjNum 119 snap 32 seq_num 119) 2023-11-05T08:34:10.767 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:34:10.763+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:34:10.808 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:34:10.808 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2023-11-05T08:34:10.808 INFO:teuthology.orchestra.run.smithi069.stdout: "num_ops": 0 2023-11-05T08:34:10.809 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:34:10.823 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2023-11-05T08:34:10.868 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:34:10.863+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:34:10.918 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-11-05T08:34:10.941 INFO:tasks.rados.rados.0.smithi090.stdout:398: finishing copy_from to smithi09039252-49 2023-11-05T08:34:10.941 INFO:tasks.rados.rados.0.smithi090.stdout:398: got expected ENOENT (src dne) 2023-11-05T08:34:10.941 INFO:tasks.rados.rados.0.smithi090.stdout:398: finishing copy_from racing read to smithi09039252-49 2023-11-05T08:34:10.942 INFO:tasks.rados.rados.0.smithi090.stdout:392: done (4 left) 2023-11-05T08:34:10.942 INFO:tasks.rados.rados.0.smithi090.stdout:393: done (3 left) 2023-11-05T08:34:10.942 INFO:tasks.rados.rados.0.smithi090.stdout:394: done (2 left) 2023-11-05T08:34:10.942 INFO:tasks.rados.rados.0.smithi090.stdout:396: done (1 left) 2023-11-05T08:34:10.943 INFO:tasks.rados.rados.0.smithi090.stdout:398: done (0 left) 2023-11-05T08:34:10.943 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:34:10.943 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [], 2023-11-05T08:34:10.943 INFO:teuthology.orchestra.run.smithi069.stdout: "complaint_time": 30, 2023-11-05T08:34:10.943 INFO:teuthology.orchestra.run.smithi069.stdout: "num_blocked_ops": 0 2023-11-05T08:34:10.944 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:34:10.949 INFO:tasks.rados.rados.0.smithi090.stderr:0 errors. 2023-11-05T08:34:10.949 INFO:tasks.rados.rados.0.smithi090.stderr: 2023-11-05T08:34:10.958 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2023-11-05T08:34:10.959 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:34:11.004 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:34:11.000+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:34:11.070 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:34:11.063+0000 7f55c564d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:34:11.080 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2023-11-05T08:34:11.080 INFO:teuthology.orchestra.run.smithi069.stdout: "size": 20, 2023-11-05T08:34:11.081 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 600, 2023-11-05T08:34:11.081 INFO:teuthology.orchestra.run.smithi069.stdout: "ops": [ 2023-11-05T08:34:11.081 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.081 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.as4 41/33 ECSubWrite(tid=35, reqid=osd.7.0:35, at_version=41'12, trim_to=0'0, roll_forward_to=40'10))", 2023-11-05T08:34:11.081 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.128988+0000", 2023-11-05T08:34:11.082 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 76.947523478999997, 2023-11-05T08:34:11.082 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0097032620000000007, 2023-11-05T08:34:11.082 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:11.082 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:11.082 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:11.083 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.083 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:11.083 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.128988+0000", 2023-11-05T08:34:11.083 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.083 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.084 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.084 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:11.084 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.128988+0000", 2023-11-05T08:34:11.084 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.084 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.084 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.085 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:11.085 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.128989+0000", 2023-11-05T08:34:11.085 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.3600000000000003e-07 2023-11-05T08:34:11.085 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.085 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.086 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:11.086 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.129043+0000", 2023-11-05T08:34:11.086 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.4163999999999998e-05 2023-11-05T08:34:11.086 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.086 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.087 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:11.087 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.129080+0000", 2023-11-05T08:34:11.087 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.7144999999999998e-05 2023-11-05T08:34:11.087 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.087 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.088 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:11.088 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.129104+0000", 2023-11-05T08:34:11.088 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3538999999999999e-05 2023-11-05T08:34:11.088 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.088 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.089 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:11.089 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.129248+0000", 2023-11-05T08:34:11.089 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00014404500000000001 2023-11-05T08:34:11.089 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.089 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.089 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:11.090 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.129273+0000", 2023-11-05T08:34:11.090 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5814999999999999e-05 2023-11-05T08:34:11.090 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.090 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.090 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:11.091 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.138576+0000", 2023-11-05T08:34:11.091 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0093020840000000004 2023-11-05T08:34:11.091 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.091 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.091 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:11.091 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.138691+0000", 2023-11-05T08:34:11.092 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000115834 2023-11-05T08:34:11.092 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.092 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:11.092 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.092 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.092 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.093 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.as4 41/33 ECSubWrite(tid=36, reqid=osd.7.0:36, at_version=41'14, trim_to=0'0, roll_forward_to=40'10))", 2023-11-05T08:34:11.093 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.131320+0000", 2023-11-05T08:34:11.093 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 76.945191597999994, 2023-11-05T08:34:11.093 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0086819870000000004, 2023-11-05T08:34:11.093 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:11.094 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:11.094 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:11.094 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.094 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:11.094 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131320+0000", 2023-11-05T08:34:11.095 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.095 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.095 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.095 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:11.095 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131320+0000", 2023-11-05T08:34:11.096 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.096 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.096 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.096 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:11.096 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131322+0000", 2023-11-05T08:34:11.096 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2500000000000001e-06 2023-11-05T08:34:11.097 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.097 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.097 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:11.097 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131375+0000", 2023-11-05T08:34:11.097 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.3041000000000003e-05 2023-11-05T08:34:11.098 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.098 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.098 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:11.098 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131422+0000", 2023-11-05T08:34:11.099 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.6526999999999997e-05 2023-11-05T08:34:11.099 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.099 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.099 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:11.099 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131445+0000", 2023-11-05T08:34:11.099 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3672e-05 2023-11-05T08:34:11.100 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.100 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.100 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:11.100 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131641+0000", 2023-11-05T08:34:11.100 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00019531299999999999 2023-11-05T08:34:11.101 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.101 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.101 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:11.101 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.131666+0000", 2023-11-05T08:34:11.101 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5122000000000001e-05 2023-11-05T08:34:11.101 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.102 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.102 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:11.102 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.139945+0000", 2023-11-05T08:34:11.102 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0082790889999999999 2023-11-05T08:34:11.102 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.103 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.103 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:11.103 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.140002+0000", 2023-11-05T08:34:11.103 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.6972999999999998e-05 2023-11-05T08:34:11.103 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.103 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:11.104 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.104 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.104 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.104 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:294 3.9s0 3:96f693b2:::smithi09039252-13:head [read 0~1064960,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2023-11-05T08:34:11.104 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.184100+0000", 2023-11-05T08:34:11.104 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 76.892411441999997, 2023-11-05T08:34:11.105 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0099915090000000008, 2023-11-05T08:34:11.105 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:11.105 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2023-11-05T08:34:11.105 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:34:11.105 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:34:11.106 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:11.106 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 294 2023-11-05T08:34:11.106 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.106 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:11.106 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.106 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:11.107 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184100+0000", 2023-11-05T08:34:11.107 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.107 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.107 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.107 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:11.108 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184100+0000", 2023-11-05T08:34:11.108 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.108 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.108 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.108 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:11.108 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184101+0000", 2023-11-05T08:34:11.109 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2100000000000001e-06 2023-11-05T08:34:11.109 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.109 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.109 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:11.109 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184109+0000", 2023-11-05T08:34:11.110 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.5630000000000003e-06 2023-11-05T08:34:11.110 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.110 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.110 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:11.110 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184126+0000", 2023-11-05T08:34:11.111 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6674e-05 2023-11-05T08:34:11.111 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.111 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.111 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:11.111 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184138+0000", 2023-11-05T08:34:11.111 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.2306e-05 2023-11-05T08:34:11.112 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.112 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.112 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:11.112 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.189352+0000", 2023-11-05T08:34:11.112 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0052140629999999997 2023-11-05T08:34:11.112 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.113 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.113 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:34:11.113 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.189596+0000", 2023-11-05T08:34:11.113 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00024365899999999999 2023-11-05T08:34:11.113 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.114 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.114 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:11.114 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.194092+0000", 2023-11-05T08:34:11.114 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0044960340000000003 2023-11-05T08:34:11.114 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.115 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:11.115 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.115 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.115 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.115 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:295 3.9s0 3:96f693b2:::smithi09039252-13:head [read 0~1064960,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2023-11-05T08:34:11.115 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:54.184155+0000", 2023-11-05T08:34:11.116 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 76.892356637000006, 2023-11-05T08:34:11.116 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.011309207999999999, 2023-11-05T08:34:11.116 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:11.116 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2023-11-05T08:34:11.116 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:34:11.117 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:34:11.117 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:11.117 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 295 2023-11-05T08:34:11.117 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.117 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:11.118 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.118 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:11.118 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184155+0000", 2023-11-05T08:34:11.118 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.119 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.119 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.119 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:11.119 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184155+0000", 2023-11-05T08:34:11.119 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.120 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.120 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.120 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:11.120 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184156+0000", 2023-11-05T08:34:11.120 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0139999999999999e-06 2023-11-05T08:34:11.120 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.121 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.121 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:11.121 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184162+0000", 2023-11-05T08:34:11.121 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.5330000000000002e-06 2023-11-05T08:34:11.121 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.122 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.122 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:11.122 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184176+0000", 2023-11-05T08:34:11.122 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.486e-05 2023-11-05T08:34:11.122 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.123 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.123 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:11.123 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.184188+0000", 2023-11-05T08:34:11.123 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1246e-05 2023-11-05T08:34:11.123 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.124 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.124 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:11.124 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.190012+0000", 2023-11-05T08:34:11.124 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0058240920000000003 2023-11-05T08:34:11.124 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.125 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.125 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:34:11.125 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.190143+0000", 2023-11-05T08:34:11.125 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000130984 2023-11-05T08:34:11.125 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.125 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.126 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:11.126 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:54.195464+0000", 2023-11-05T08:34:11.126 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0053214789999999996 2023-11-05T08:34:11.126 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.126 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:11.127 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.127 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.127 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.127 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.as4 42/33 ECSubWrite(tid=38, reqid=osd.7.0:38, at_version=42'16, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:34:11.127 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:55.127593+0000", 2023-11-05T08:34:11.128 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 75.948918418999995, 2023-11-05T08:34:11.128 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.014145875, 2023-11-05T08:34:11.128 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:11.128 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:11.128 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:11.129 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.129 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:11.129 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127593+0000", 2023-11-05T08:34:11.129 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.129 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.130 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.130 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:11.130 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127593+0000", 2023-11-05T08:34:11.130 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.130 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.131 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.131 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:11.131 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127594+0000", 2023-11-05T08:34:11.131 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.1399999999999996e-07 2023-11-05T08:34:11.131 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.132 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.132 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:11.132 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127658+0000", 2023-11-05T08:34:11.132 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 6.3590000000000006e-05 2023-11-05T08:34:11.132 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.133 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.133 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:11.133 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127690+0000", 2023-11-05T08:34:11.133 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2447999999999998e-05 2023-11-05T08:34:11.133 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.134 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.134 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:11.134 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127716+0000", 2023-11-05T08:34:11.134 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.5931999999999998e-05 2023-11-05T08:34:11.134 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.135 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.135 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:11.135 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127955+0000", 2023-11-05T08:34:11.135 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00023882100000000001 2023-11-05T08:34:11.135 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.136 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.136 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:11.136 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127983+0000", 2023-11-05T08:34:11.136 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7928000000000001e-05 2023-11-05T08:34:11.136 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.136 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.137 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:11.137 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.141638+0000", 2023-11-05T08:34:11.137 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.013655244 2023-11-05T08:34:11.137 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.137 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.138 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:11.138 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.141739+0000", 2023-11-05T08:34:11.138 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000101098 2023-11-05T08:34:11.138 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.138 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:11.139 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.139 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.139 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.139 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.as4 42/33 ECSubWrite(tid=39, reqid=osd.7.0:39, at_version=42'18, trim_to=0'0, roll_forward_to=41'14))", 2023-11-05T08:34:11.139 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:32:55.127804+0000", 2023-11-05T08:34:11.140 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 75.948707596000006, 2023-11-05T08:34:11.140 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.017037818, 2023-11-05T08:34:11.140 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:11.140 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:11.140 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:11.141 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.141 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:11.141 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127804+0000", 2023-11-05T08:34:11.141 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.141 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.142 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.142 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:11.142 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127804+0000", 2023-11-05T08:34:11.142 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.142 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.143 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.143 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:11.143 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127805+0000", 2023-11-05T08:34:11.143 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.026e-06 2023-11-05T08:34:11.143 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.144 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.144 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:11.144 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.127848+0000", 2023-11-05T08:34:11.144 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.3195000000000002e-05 2023-11-05T08:34:11.144 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.144 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.145 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:11.145 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.133513+0000", 2023-11-05T08:34:11.145 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.005664741 2023-11-05T08:34:11.145 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.145 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.146 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:11.146 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.133539+0000", 2023-11-05T08:34:11.146 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6389000000000001e-05 2023-11-05T08:34:11.146 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.146 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.147 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:11.147 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.135275+0000", 2023-11-05T08:34:11.147 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001735349 2023-11-05T08:34:11.147 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.147 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.147 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:11.148 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.135312+0000", 2023-11-05T08:34:11.148 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.6965000000000003e-05 2023-11-05T08:34:11.148 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.148 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.148 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:11.149 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.144746+0000", 2023-11-05T08:34:11.149 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0094338319999999996 2023-11-05T08:34:11.149 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.149 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.149 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:11.149 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:32:55.144842+0000", 2023-11-05T08:34:11.150 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 9.6321000000000002e-05 2023-11-05T08:34:11.150 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.150 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:11.150 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.150 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.151 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.151 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:341 3.9s0 3:96f693b2:::smithi09039252-13:head [create,setxattr ]ScjNOsReWmcrsXtQckiNfVsraic]liXUgaZfrPb (12280) in=40b,setxattr RP^fql]PXPbbV]`\\UXhZUcc\\s\\eVVh]On^oUhNgZ (593) in=40b,setxattr ejajVrghj]sk_[VhasYoUSNhlO_PNopbXh\\iYad\\ (1963) in=40b,setxattr ocopbYhpUShgXjpVTtXhgrZ^fhTrWUZnka_sOlQ] (1506) in=40b,setxattr drSfWQp[TZZaai[S^\\ecpepRjetPTZ[Qn[slVhsn (1437) in=40b,setxattr qfWkbgV_hlqXWl`QOe]pOPUPZspbtU__NVPUUltj (877) in=40b,setxattr ]Vrng_\\anafncgktlQh_okb\\i]chR]P]jeoaR\\h] (1858) in=40b,setxattr ]W_abhUq]ocdkOt[ZkpjgPX^drZi]OcUWNRonRa\\ (455) in=40b,setxattr WiZeSWPfsqgYXk`d\\gOeSecbpakdsPTXZfZhn_k^ (875) in=40b,setxattr kqn^akbbamUTRXRhQS^i`lQYomhV_P`j\\ahTj]dk (1894) in=40b,setxattr TUcrapOXk_aZjTSdOT`rYcU_YYstnVgNTojk`lXZ (805) in=40b,setxattr a`RfekoeadQgiYossXre`rV^qNeUNiNf\\lSXhohq (15648) in=40b,setxattr VigYdf[rkVcbX`_aXcZrljeilZZV\\O[gpOhof]\\P (1473) in=40b,setxattr Qrfl^hc[_sp[p`bcN[Ne\\tg^]rgrOlRqfQ_VWNt\\ (1261) in=40b,setxattr iQVqR\\bssbQRQf^a`UrVROfgpPZlnstjPUaUnWWt (665) in=40b,setxattr aO^mjdTNk[qYOgYsgTigbbP`]mhe`WmaNUO[^ele (979) in=40b,setxattr cfQeZUernkePqgPcSrokkcnPqZ^Osc^^ng[ilga_ (16840) in=40b,setxattr fdbVa_]fdVq_`kcpaWlQSgY^VgakSqVhj`qqdj[m (1858) in=40b,setxattr tXieTi_TgP_dVipjmQcU_QVRqNdboYhRWgNrpVhm (1858) in=40b,setxattr ]YTcidkg\\][ZtajYPqmOji_UPVPZ`j^]tnNRefcd (2876) in=40b] snapc 10=[7] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:34:11.151 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:05.255132+0000", 2023-11-05T08:34:11.151 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 65.821379172999997, 2023-11-05T08:34:11.151 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.01486664, 2023-11-05T08:34:11.152 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:11.152 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:11.152 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:34:11.152 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:34:11.152 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:11.152 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 341 2023-11-05T08:34:11.153 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.153 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:11.153 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.153 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:11.153 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255132+0000", 2023-11-05T08:34:11.153 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.154 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.154 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.154 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:11.154 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255132+0000", 2023-11-05T08:34:11.154 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.155 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.155 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.155 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:11.155 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255136+0000", 2023-11-05T08:34:11.155 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.5159999999999999e-06 2023-11-05T08:34:11.156 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.156 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.156 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:11.156 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255423+0000", 2023-11-05T08:34:11.156 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00028735799999999998 2023-11-05T08:34:11.157 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.157 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.157 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:11.157 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255462+0000", 2023-11-05T08:34:11.157 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.8512e-05 2023-11-05T08:34:11.158 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.158 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.158 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:11.158 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255499+0000", 2023-11-05T08:34:11.158 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.7531999999999999e-05 2023-11-05T08:34:11.159 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.159 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.159 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:11.159 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255651+0000", 2023-11-05T08:34:11.159 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00015121700000000001 2023-11-05T08:34:11.159 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.160 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.160 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:34:11.160 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.256051+0000", 2023-11-05T08:34:11.160 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00040081699999999998 2023-11-05T08:34:11.160 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.160 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.161 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:11.161 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.257968+0000", 2023-11-05T08:34:11.161 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001917052 2023-11-05T08:34:11.161 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.161 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.162 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:11.162 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.265314+0000", 2023-11-05T08:34:11.162 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0073454469999999997 2023-11-05T08:34:11.162 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.162 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.163 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:34:11.163 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.269385+0000", 2023-11-05T08:34:11.163 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0040709839999999997 2023-11-05T08:34:11.163 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.163 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.163 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:11.164 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.269999+0000", 2023-11-05T08:34:11.164 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00061420500000000005 2023-11-05T08:34:11.164 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.164 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:11.164 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.165 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.165 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.165 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:344 3.9s0 3:936041dc:::smithi09039252-3:head [copy-from ver 4] snapc 10=[7] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:34:11.165 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:05.255538+0000", 2023-11-05T08:34:11.165 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 65.820973163000005, 2023-11-05T08:34:11.165 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.032906886000000003, 2023-11-05T08:34:11.166 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:11.166 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:11.166 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:34:11.166 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:34:11.166 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:11.167 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 344 2023-11-05T08:34:11.167 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.167 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:11.167 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.167 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:11.168 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255538+0000", 2023-11-05T08:34:11.168 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.168 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.168 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.168 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:11.169 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255538+0000", 2023-11-05T08:34:11.169 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.169 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.169 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.169 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:11.170 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255540+0000", 2023-11-05T08:34:11.170 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5060000000000001e-06 2023-11-05T08:34:11.170 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.170 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.170 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:11.171 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255550+0000", 2023-11-05T08:34:11.171 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0063e-05 2023-11-05T08:34:11.171 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.171 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.171 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:11.172 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255589+0000", 2023-11-05T08:34:11.172 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.9245000000000001e-05 2023-11-05T08:34:11.172 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.172 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.172 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:11.172 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255606+0000", 2023-11-05T08:34:11.173 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6307000000000002e-05 2023-11-05T08:34:11.173 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.173 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.173 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:11.173 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.259641+0000", 2023-11-05T08:34:11.174 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.004035421 2023-11-05T08:34:11.174 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.174 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.174 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:34:11.174 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.260112+0000", 2023-11-05T08:34:11.174 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00047097799999999999 2023-11-05T08:34:11.175 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.175 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.175 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:11.175 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.280466+0000", 2023-11-05T08:34:11.175 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.020353850999999999 2023-11-05T08:34:11.176 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.176 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.176 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:11.176 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.286378+0000", 2023-11-05T08:34:11.176 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0059124490000000002 2023-11-05T08:34:11.176 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.177 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.177 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:34:11.177 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.288092+0000", 2023-11-05T08:34:11.177 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0017133300000000001 2023-11-05T08:34:11.177 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.177 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.178 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:11.178 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.288445+0000", 2023-11-05T08:34:11.178 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000353736 2023-11-05T08:34:11.178 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.178 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:11.179 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.179 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.179 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.179 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:345 3.9s0 3:936041dc:::smithi09039252-3:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:34:11.179 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:05.255630+0000", 2023-11-05T08:34:11.180 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 65.820881702999998, 2023-11-05T08:34:11.180 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.029547922000000001, 2023-11-05T08:34:11.180 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:11.180 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2023-11-05T08:34:11.180 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:34:11.180 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:34:11.181 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:11.181 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 345 2023-11-05T08:34:11.181 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.181 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:11.181 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.181 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:11.182 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255630+0000", 2023-11-05T08:34:11.182 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.182 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.182 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.182 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:11.183 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255630+0000", 2023-11-05T08:34:11.183 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.183 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.183 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.183 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:11.184 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255631+0000", 2023-11-05T08:34:11.184 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.519e-06 2023-11-05T08:34:11.184 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.184 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.184 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:11.185 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255640+0000", 2023-11-05T08:34:11.185 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.9360000000000005e-06 2023-11-05T08:34:11.185 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.185 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.185 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:11.186 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255659+0000", 2023-11-05T08:34:11.186 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.9089999999999998e-05 2023-11-05T08:34:11.186 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.186 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.186 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:11.187 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.255675+0000", 2023-11-05T08:34:11.187 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.5133000000000001e-05 2023-11-05T08:34:11.187 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.187 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.187 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:11.188 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.260602+0000", 2023-11-05T08:34:11.188 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0049270449999999997 2023-11-05T08:34:11.188 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.188 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.188 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for blocked object", 2023-11-05T08:34:11.189 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.260717+0000", 2023-11-05T08:34:11.189 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000115182 2023-11-05T08:34:11.189 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.189 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.189 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:11.189 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.284014+0000", 2023-11-05T08:34:11.190 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.023297577999999999 2023-11-05T08:34:11.190 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.190 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.190 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:34:11.191 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.284370+0000", 2023-11-05T08:34:11.191 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000355306 2023-11-05T08:34:11.191 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.191 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.191 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:11.191 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:05.285178+0000", 2023-11-05T08:34:11.192 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00080813300000000005 2023-11-05T08:34:11.192 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.192 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:11.192 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.192 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.192 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.193 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.as4 61/45 ECSubWrite(tid=57, reqid=client.4691.0:405, at_version=61'23, trim_to=0'0, roll_forward_to=52'21))", 2023-11-05T08:34:11.193 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:14.366205+0000", 2023-11-05T08:34:11.193 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 56.710306222, 2023-11-05T08:34:11.193 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0087065779999999995, 2023-11-05T08:34:11.193 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:11.193 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:11.194 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:11.194 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.194 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:11.194 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.366205+0000", 2023-11-05T08:34:11.194 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.195 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.195 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.195 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:11.195 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.366205+0000", 2023-11-05T08:34:11.195 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.196 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.196 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.196 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:11.196 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.366206+0000", 2023-11-05T08:34:11.196 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.3000000000000001e-07 2023-11-05T08:34:11.196 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.197 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.197 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:11.197 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.366951+0000", 2023-11-05T08:34:11.197 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00074537699999999998 2023-11-05T08:34:11.197 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.197 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.198 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:11.198 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.366978+0000", 2023-11-05T08:34:11.198 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6721999999999999e-05 2023-11-05T08:34:11.198 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.198 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.199 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:11.199 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.366994+0000", 2023-11-05T08:34:11.199 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6189999999999999e-05 2023-11-05T08:34:11.199 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.199 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.200 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:11.200 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.367186+0000", 2023-11-05T08:34:11.200 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000191387 2023-11-05T08:34:11.200 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.200 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.200 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:11.201 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.367213+0000", 2023-11-05T08:34:11.201 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7169e-05 2023-11-05T08:34:11.201 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.201 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.201 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:11.202 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.374835+0000", 2023-11-05T08:34:11.202 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0076217589999999997 2023-11-05T08:34:11.202 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.202 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.202 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:11.203 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:14.374912+0000", 2023-11-05T08:34:11.203 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.7443999999999999e-05 2023-11-05T08:34:11.203 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.203 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:11.203 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.204 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.204 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.204 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:662 3.9s0 3:96f693b2:::smithi09039252-13:head [append 0~704512] snapc 38=[7] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2023-11-05T08:34:11.204 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:45.739998+0000", 2023-11-05T08:34:11.204 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 25.336513381, 2023-11-05T08:34:11.204 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.018542187000000002, 2023-11-05T08:34:11.205 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:11.205 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:11.205 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:34:11.205 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:34:11.205 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:11.205 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 662 2023-11-05T08:34:11.206 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.206 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:11.206 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.206 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:11.206 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.739998+0000", 2023-11-05T08:34:11.207 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.207 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.207 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.207 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:11.207 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.739998+0000", 2023-11-05T08:34:11.207 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.208 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.208 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.208 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:11.208 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.740001+0000", 2023-11-05T08:34:11.208 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.2229999999999999e-06 2023-11-05T08:34:11.209 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.209 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.209 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:11.209 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.741090+0000", 2023-11-05T08:34:11.209 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001088419 2023-11-05T08:34:11.209 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.210 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.210 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:11.210 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.741127+0000", 2023-11-05T08:34:11.210 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.7298000000000001e-05 2023-11-05T08:34:11.210 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.210 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.211 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:11.211 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.741144+0000", 2023-11-05T08:34:11.211 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.6943e-05 2023-11-05T08:34:11.211 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.211 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.212 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:11.212 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.741287+0000", 2023-11-05T08:34:11.212 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00014318499999999999 2023-11-05T08:34:11.212 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.212 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.213 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op must wait for map", 2023-11-05T08:34:11.213 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.741300+0000", 2023-11-05T08:34:11.213 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.3033e-05 2023-11-05T08:34:11.213 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.213 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.213 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:11.214 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.746064+0000", 2023-11-05T08:34:11.214 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0047641630000000001 2023-11-05T08:34:11.214 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.214 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.214 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:34:11.215 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.746327+0000", 2023-11-05T08:34:11.215 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000262057 2023-11-05T08:34:11.215 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.215 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.215 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:11.215 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.748980+0000", 2023-11-05T08:34:11.216 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0026535930000000001 2023-11-05T08:34:11.216 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.216 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.216 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:11.216 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.757920+0000", 2023-11-05T08:34:11.217 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0089402059999999992 2023-11-05T08:34:11.217 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.217 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.217 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:34:11.217 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.758238+0000", 2023-11-05T08:34:11.217 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000318115 2023-11-05T08:34:11.218 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.218 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.218 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:11.218 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.758540+0000", 2023-11-05T08:34:11.218 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00030195200000000003 2023-11-05T08:34:11.219 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.219 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:11.219 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.219 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.219 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.219 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:663 3.9s0 3:96f693b2:::smithi09039252-13:head [append 0~655360] snapc 38=[7] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2023-11-05T08:34:11.220 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:45.750575+0000", 2023-11-05T08:34:11.220 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 25.325936767000002, 2023-11-05T08:34:11.220 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0094943219999999995, 2023-11-05T08:34:11.220 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:11.220 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:11.220 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:34:11.221 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:34:11.221 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:11.221 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 663 2023-11-05T08:34:11.221 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.221 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:11.221 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.222 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:11.222 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.750575+0000", 2023-11-05T08:34:11.222 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.222 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.222 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.223 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:11.223 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.750575+0000", 2023-11-05T08:34:11.223 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.223 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.223 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.224 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:11.224 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.750578+0000", 2023-11-05T08:34:11.224 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.14e-06 2023-11-05T08:34:11.224 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.224 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.224 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:11.225 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.751833+0000", 2023-11-05T08:34:11.225 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0012549060000000001 2023-11-05T08:34:11.225 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.225 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.225 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:11.226 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.751867+0000", 2023-11-05T08:34:11.226 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.4591999999999997e-05 2023-11-05T08:34:11.226 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.226 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.226 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:11.227 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.751892+0000", 2023-11-05T08:34:11.227 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4341999999999998e-05 2023-11-05T08:34:11.227 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.227 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.227 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:11.227 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.752729+0000", 2023-11-05T08:34:11.228 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00083733500000000003 2023-11-05T08:34:11.228 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.228 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.228 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:34:11.228 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.753047+0000", 2023-11-05T08:34:11.229 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00031830500000000001 2023-11-05T08:34:11.229 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.229 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.229 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:11.229 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.755226+0000", 2023-11-05T08:34:11.230 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0021783839999999998 2023-11-05T08:34:11.230 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.230 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.230 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:11.230 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.759221+0000", 2023-11-05T08:34:11.231 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0039956569999999997 2023-11-05T08:34:11.231 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.231 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.231 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:34:11.231 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.759799+0000", 2023-11-05T08:34:11.232 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00057719800000000001 2023-11-05T08:34:11.232 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.232 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.232 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:11.232 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.760069+0000", 2023-11-05T08:34:11.232 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00027046299999999998 2023-11-05T08:34:11.233 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.233 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:11.233 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.233 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.233 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.234 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:665 3.9s0 3:96f693b2:::smithi09039252-13:head [setxattr _header (57) in=7b] snapc 38=[7] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2023-11-05T08:34:11.234 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:45.759991+0000", 2023-11-05T08:34:11.234 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 25.316520916000002, 2023-11-05T08:34:11.234 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.010817032000000001, 2023-11-05T08:34:11.234 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:11.235 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:11.235 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:34:11.235 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:34:11.235 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:11.235 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 665 2023-11-05T08:34:11.236 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.236 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:11.236 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.236 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:11.236 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.759991+0000", 2023-11-05T08:34:11.236 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.237 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.237 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.237 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:11.237 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.759991+0000", 2023-11-05T08:34:11.237 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.238 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.238 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.238 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:11.238 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.759992+0000", 2023-11-05T08:34:11.238 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0499999999999999e-06 2023-11-05T08:34:11.239 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.239 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.239 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:11.239 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.759997+0000", 2023-11-05T08:34:11.239 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 5.5629999999999999e-06 2023-11-05T08:34:11.239 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.240 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.240 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:11.240 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.760009+0000", 2023-11-05T08:34:11.240 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.1559999999999999e-05 2023-11-05T08:34:11.240 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.241 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.241 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:11.241 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.760017+0000", 2023-11-05T08:34:11.241 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.3259999999999996e-06 2023-11-05T08:34:11.241 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.241 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.242 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:11.242 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.764481+0000", 2023-11-05T08:34:11.242 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0044635789999999996 2023-11-05T08:34:11.242 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.243 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.243 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:34:11.243 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.764655+0000", 2023-11-05T08:34:11.243 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000173773 2023-11-05T08:34:11.243 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.243 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.244 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:11.244 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.765442+0000", 2023-11-05T08:34:11.244 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00078758400000000003 2023-11-05T08:34:11.244 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.244 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.245 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:11.245 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.770202+0000", 2023-11-05T08:34:11.245 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0047597129999999996 2023-11-05T08:34:11.245 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.245 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.246 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:34:11.246 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.770508+0000", 2023-11-05T08:34:11.246 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00030603899999999997 2023-11-05T08:34:11.246 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.246 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.247 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:11.247 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.770808+0000", 2023-11-05T08:34:11.247 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00029984500000000003 2023-11-05T08:34:11.247 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.247 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:11.248 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.248 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.248 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.248 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:666 3.9s0 3:96f693b2:::smithi09039252-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e92)", 2023-11-05T08:34:11.248 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:45.760029+0000", 2023-11-05T08:34:11.249 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 25.316482329999999, 2023-11-05T08:34:11.249 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.014754087000000001, 2023-11-05T08:34:11.249 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:11.249 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "started", 2023-11-05T08:34:11.249 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:34:11.249 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:34:11.250 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:11.250 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 666 2023-11-05T08:34:11.250 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.250 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:11.250 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.251 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:11.251 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.760029+0000", 2023-11-05T08:34:11.251 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.251 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.251 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.251 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:11.252 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.760029+0000", 2023-11-05T08:34:11.252 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.252 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.252 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.252 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:11.253 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.760030+0000", 2023-11-05T08:34:11.253 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.4300000000000002e-07 2023-11-05T08:34:11.253 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.253 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.253 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:11.253 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.760034+0000", 2023-11-05T08:34:11.254 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.1169999999999997e-06 2023-11-05T08:34:11.254 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.254 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.254 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:11.254 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.760045+0000", 2023-11-05T08:34:11.255 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.0779e-05 2023-11-05T08:34:11.255 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.255 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.255 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:11.255 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.760053+0000", 2023-11-05T08:34:11.255 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 7.8900000000000007e-06 2023-11-05T08:34:11.256 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.256 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.256 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:11.256 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.766830+0000", 2023-11-05T08:34:11.256 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0067772910000000004 2023-11-05T08:34:11.257 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.257 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.257 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "waiting for rw locks", 2023-11-05T08:34:11.257 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.767112+0000", 2023-11-05T08:34:11.257 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00028158600000000001 2023-11-05T08:34:11.257 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.258 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.258 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:11.258 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.771458+0000", 2023-11-05T08:34:11.258 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0043464460000000003 2023-11-05T08:34:11.258 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.259 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.259 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:34:11.259 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.771631+0000", 2023-11-05T08:34:11.259 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00017279599999999999 2023-11-05T08:34:11.259 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.259 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.260 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:11.260 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:45.774783+0000", 2023-11-05T08:34:11.260 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0031524389999999999 2023-11-05T08:34:11.260 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.260 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:11.260 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.261 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.261 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.261 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.fs2 101/37 ECSubWrite(tid=173, reqid=client.4691.0:776, at_version=101'60, trim_to=0'0, roll_forward_to=90'57))", 2023-11-05T08:34:11.261 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:54.766764+0000", 2023-11-05T08:34:11.261 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 16.309748032000002, 2023-11-05T08:34:11.262 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.011572742, 2023-11-05T08:34:11.262 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:11.262 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:11.262 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:11.262 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.262 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:11.263 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.766764+0000", 2023-11-05T08:34:11.263 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.263 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.263 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.263 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:11.264 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.766764+0000", 2023-11-05T08:34:11.264 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.264 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.264 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.264 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:11.264 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.766765+0000", 2023-11-05T08:34:11.265 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.7400000000000001e-06 2023-11-05T08:34:11.265 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.265 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.265 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:11.265 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.767065+0000", 2023-11-05T08:34:11.265 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00029994 2023-11-05T08:34:11.266 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.266 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.266 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:11.266 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.767097+0000", 2023-11-05T08:34:11.266 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.1572999999999997e-05 2023-11-05T08:34:11.267 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.267 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.267 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:11.267 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.767120+0000", 2023-11-05T08:34:11.267 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2765999999999999e-05 2023-11-05T08:34:11.268 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.268 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.268 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:11.268 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.767275+0000", 2023-11-05T08:34:11.268 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00015550199999999999 2023-11-05T08:34:11.268 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.269 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.269 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:11.269 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.767303+0000", 2023-11-05T08:34:11.269 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7576000000000002e-05 2023-11-05T08:34:11.269 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.270 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.270 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:11.270 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.778250+0000", 2023-11-05T08:34:11.270 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.010947077 2023-11-05T08:34:11.270 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.271 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.271 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:11.271 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.778336+0000", 2023-11-05T08:34:11.271 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 8.6568000000000002e-05 2023-11-05T08:34:11.271 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.272 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:11.272 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.272 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.272 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.272 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.fs2 101/37 ECSubWrite(tid=174, reqid=client.4691.0:777, at_version=101'61, trim_to=0'0, roll_forward_to=101'59))", 2023-11-05T08:34:11.273 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:54.773320+0000", 2023-11-05T08:34:11.273 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 16.303191109, 2023-11-05T08:34:11.273 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.013436813000000001, 2023-11-05T08:34:11.273 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:11.273 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:11.273 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:11.274 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.274 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:11.274 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.773320+0000", 2023-11-05T08:34:11.274 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.274 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.275 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.275 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:11.275 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.773320+0000", 2023-11-05T08:34:11.275 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.275 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.276 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.276 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:11.276 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.773323+0000", 2023-11-05T08:34:11.276 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1399999999999998e-06 2023-11-05T08:34:11.276 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.276 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.277 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:11.277 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.773742+0000", 2023-11-05T08:34:11.277 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000419737 2023-11-05T08:34:11.277 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.277 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.277 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:11.278 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.773776+0000", 2023-11-05T08:34:11.278 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.3188999999999997e-05 2023-11-05T08:34:11.278 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.278 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.278 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:11.279 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.773804+0000", 2023-11-05T08:34:11.279 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8679000000000001e-05 2023-11-05T08:34:11.279 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.279 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.279 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:11.279 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.773959+0000", 2023-11-05T08:34:11.280 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00015507800000000001 2023-11-05T08:34:11.280 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.280 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.280 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:11.280 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.773986+0000", 2023-11-05T08:34:11.280 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6988999999999999e-05 2023-11-05T08:34:11.281 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.281 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.281 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:11.281 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.786027+0000", 2023-11-05T08:34:11.281 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.012040252 2023-11-05T08:34:11.281 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.282 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.282 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:11.282 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.786757+0000", 2023-11-05T08:34:11.282 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00073074900000000005 2023-11-05T08:34:11.282 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.283 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:11.283 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.283 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.283 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.283 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.fs2 101/37 ECSubWrite(tid=175, reqid=client.4691.0:778, at_version=101'62, trim_to=0'0, roll_forward_to=101'59))", 2023-11-05T08:34:11.283 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:54.775481+0000", 2023-11-05T08:34:11.284 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 16.301030191999999, 2023-11-05T08:34:11.284 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.011318572000000001, 2023-11-05T08:34:11.284 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:11.284 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:11.284 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:11.285 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.285 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:11.285 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.775481+0000", 2023-11-05T08:34:11.285 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.285 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.286 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.286 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:11.286 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.775481+0000", 2023-11-05T08:34:11.286 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.286 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.287 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.287 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:11.287 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.775483+0000", 2023-11-05T08:34:11.287 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.086e-06 2023-11-05T08:34:11.287 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.288 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.288 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:11.288 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.775697+0000", 2023-11-05T08:34:11.288 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00021394400000000001 2023-11-05T08:34:11.288 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.289 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.289 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:11.289 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.775732+0000", 2023-11-05T08:34:11.289 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.4152000000000001e-05 2023-11-05T08:34:11.289 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.290 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.290 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:11.290 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.775755+0000", 2023-11-05T08:34:11.290 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.3019000000000001e-05 2023-11-05T08:34:11.290 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.291 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.291 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:11.291 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.776287+0000", 2023-11-05T08:34:11.291 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000532081 2023-11-05T08:34:11.291 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.292 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.292 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:11.292 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.776313+0000", 2023-11-05T08:34:11.292 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.6038999999999999e-05 2023-11-05T08:34:11.292 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.292 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.293 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:11.293 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.786764+0000", 2023-11-05T08:34:11.293 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.010451132 2023-11-05T08:34:11.293 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.293 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.293 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:11.294 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.786800+0000", 2023-11-05T08:34:11.294 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.6118999999999998e-05 2023-11-05T08:34:11.294 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.294 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:11.294 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.295 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.295 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.295 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "MOSDECSubOpWrite(3.fs2 101/37 ECSubWrite(tid=176, reqid=client.4691.0:779, at_version=101'63, trim_to=0'0, roll_forward_to=101'59))", 2023-11-05T08:34:11.295 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:54.776535+0000", 2023-11-05T08:34:11.295 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 16.299977068, 2023-11-05T08:34:11.296 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.010318318, 2023-11-05T08:34:11.296 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:11.296 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "reached pg", 2023-11-05T08:34:11.296 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:11.296 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.297 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:11.297 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.776535+0000", 2023-11-05T08:34:11.297 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.297 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.297 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.298 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:11.298 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.776535+0000", 2023-11-05T08:34:11.298 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.298 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.298 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.299 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:11.299 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.776537+0000", 2023-11-05T08:34:11.299 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.2960000000000001e-06 2023-11-05T08:34:11.299 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.299 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.300 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:11.300 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.776585+0000", 2023-11-05T08:34:11.300 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 4.7988999999999997e-05 2023-11-05T08:34:11.300 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.300 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.301 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:11.301 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.776616+0000", 2023-11-05T08:34:11.301 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.1272999999999997e-05 2023-11-05T08:34:11.301 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.301 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.302 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:11.302 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.776638+0000", 2023-11-05T08:34:11.302 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1449000000000001e-05 2023-11-05T08:34:11.302 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.302 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.303 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:11.303 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.777854+0000", 2023-11-05T08:34:11.303 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001216469 2023-11-05T08:34:11.303 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.303 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.304 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:11.304 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.777879+0000", 2023-11-05T08:34:11.304 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.4598999999999999e-05 2023-11-05T08:34:11.304 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.304 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.304 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:11.305 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.786814+0000", 2023-11-05T08:34:11.305 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0089351610000000005 2023-11-05T08:34:11.305 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.305 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.305 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:11.306 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:54.786853+0000", 2023-11-05T08:34:11.306 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.9082000000000003e-05 2023-11-05T08:34:11.306 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.306 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:11.306 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.306 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.307 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.307 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:813 3.9s0 3:936041dc:::smithi09039252-3:head [append 0~589824] snapc 45=[45,44,42,41] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2023-11-05T08:34:11.307 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:56.766020+0000", 2023-11-05T08:34:11.307 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 14.310491783, 2023-11-05T08:34:11.307 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.023070057000000001, 2023-11-05T08:34:11.308 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:11.308 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:11.308 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:34:11.308 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:34:11.308 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:11.309 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 813 2023-11-05T08:34:11.309 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.309 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:11.309 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.309 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:11.309 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:56.766020+0000", 2023-11-05T08:34:11.310 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.310 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.310 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.310 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:11.310 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:56.766020+0000", 2023-11-05T08:34:11.311 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.311 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.311 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.311 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:11.311 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:56.766023+0000", 2023-11-05T08:34:11.312 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.0599999999999999e-06 2023-11-05T08:34:11.312 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.312 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.312 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:11.312 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:56.767065+0000", 2023-11-05T08:34:11.312 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0010420799999999999 2023-11-05T08:34:11.313 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.313 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.313 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:11.313 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:56.767100+0000", 2023-11-05T08:34:11.313 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 3.5385000000000001e-05 2023-11-05T08:34:11.314 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.314 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.314 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:11.314 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:56.767119+0000", 2023-11-05T08:34:11.314 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.8777e-05 2023-11-05T08:34:11.315 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.315 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.315 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:11.315 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:56.767253+0000", 2023-11-05T08:34:11.315 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00013374900000000001 2023-11-05T08:34:11.315 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.316 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.316 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "op must wait for map", 2023-11-05T08:34:11.316 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:56.767270+0000", 2023-11-05T08:34:11.316 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 1.696e-05 2023-11-05T08:34:11.316 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.316 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.317 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:11.317 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:56.771942+0000", 2023-11-05T08:34:11.317 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0046720590000000001 2023-11-05T08:34:11.317 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.317 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.318 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:34:11.318 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:56.772146+0000", 2023-11-05T08:34:11.318 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00020407800000000001 2023-11-05T08:34:11.318 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.318 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.319 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:11.319 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:56.774628+0000", 2023-11-05T08:34:11.319 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0024816339999999999 2023-11-05T08:34:11.319 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.319 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.319 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:11.320 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:56.787890+0000", 2023-11-05T08:34:11.320 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.013262473 2023-11-05T08:34:11.320 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.320 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.320 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:34:11.321 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:56.788722+0000", 2023-11-05T08:34:11.321 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00083189600000000005 2023-11-05T08:34:11.321 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.321 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.321 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:11.321 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:56.789090+0000", 2023-11-05T08:34:11.322 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00036790599999999998 2023-11-05T08:34:11.322 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.322 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:11.322 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.322 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.322 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.323 INFO:teuthology.orchestra.run.smithi069.stdout: "description": "osd_op(client.4691.0:814 3.9s0 3:936041dc:::smithi09039252-3:head [append 0~753664] snapc 45=[45,44,42,41] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2023-11-05T08:34:11.323 INFO:teuthology.orchestra.run.smithi069.stdout: "initiated_at": "2023-11-05T08:33:56.773391+0000", 2023-11-05T08:34:11.323 INFO:teuthology.orchestra.run.smithi069.stdout: "age": 14.303121021000001, 2023-11-05T08:34:11.323 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.016452161999999999, 2023-11-05T08:34:11.323 INFO:teuthology.orchestra.run.smithi069.stdout: "type_data": { 2023-11-05T08:34:11.324 INFO:teuthology.orchestra.run.smithi069.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:11.324 INFO:teuthology.orchestra.run.smithi069.stdout: "client_info": { 2023-11-05T08:34:11.324 INFO:teuthology.orchestra.run.smithi069.stdout: "client": "client.4691", 2023-11-05T08:34:11.324 INFO:teuthology.orchestra.run.smithi069.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:11.324 INFO:teuthology.orchestra.run.smithi069.stdout: "tid": 814 2023-11-05T08:34:11.324 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.325 INFO:teuthology.orchestra.run.smithi069.stdout: "events": [ 2023-11-05T08:34:11.325 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.325 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "initiated", 2023-11-05T08:34:11.325 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:56.773391+0000", 2023-11-05T08:34:11.325 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.325 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.326 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.326 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "header_read", 2023-11-05T08:34:11.326 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:56.773391+0000", 2023-11-05T08:34:11.326 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0 2023-11-05T08:34:11.326 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.327 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.327 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "throttled", 2023-11-05T08:34:11.327 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:56.773393+0000", 2023-11-05T08:34:11.327 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.7460000000000001e-06 2023-11-05T08:34:11.327 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.328 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.328 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "all_read", 2023-11-05T08:34:11.328 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:56.774812+0000", 2023-11-05T08:34:11.328 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.001418283 2023-11-05T08:34:11.328 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.328 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.329 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "dispatched", 2023-11-05T08:34:11.329 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:56.774840+0000", 2023-11-05T08:34:11.329 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.8402e-05 2023-11-05T08:34:11.329 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.329 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.330 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "queued_for_pg", 2023-11-05T08:34:11.330 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:56.774862+0000", 2023-11-05T08:34:11.330 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 2.1749000000000001e-05 2023-11-05T08:34:11.330 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.330 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.331 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "reached_pg", 2023-11-05T08:34:11.331 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:56.778037+0000", 2023-11-05T08:34:11.331 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0031756620000000001 2023-11-05T08:34:11.331 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.331 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.332 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "started", 2023-11-05T08:34:11.332 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:56.778326+0000", 2023-11-05T08:34:11.332 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.00028893899999999999 2023-11-05T08:34:11.332 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.332 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.333 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_started", 2023-11-05T08:34:11.333 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:56.780945+0000", 2023-11-05T08:34:11.333 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0026181619999999998 2023-11-05T08:34:11.333 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.333 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.333 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "sub_op_committed", 2023-11-05T08:34:11.334 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:56.788247+0000", 2023-11-05T08:34:11.334 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0073025399999999997 2023-11-05T08:34:11.334 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.334 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.334 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "commit_sent", 2023-11-05T08:34:11.335 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:56.789664+0000", 2023-11-05T08:34:11.335 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.0014172430000000001 2023-11-05T08:34:11.335 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2023-11-05T08:34:11.335 INFO:teuthology.orchestra.run.smithi069.stdout: { 2023-11-05T08:34:11.335 INFO:teuthology.orchestra.run.smithi069.stdout: "event": "done", 2023-11-05T08:34:11.336 INFO:teuthology.orchestra.run.smithi069.stdout: "time": "2023-11-05T08:33:56.789843+0000", 2023-11-05T08:34:11.336 INFO:teuthology.orchestra.run.smithi069.stdout: "duration": 0.000178436 2023-11-05T08:34:11.336 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.336 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:11.336 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.337 INFO:teuthology.orchestra.run.smithi069.stdout: } 2023-11-05T08:34:11.337 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2023-11-05T08:34:11.337 INFO:teuthology.orchestra.run.smithi069.stdout:} 2023-11-05T08:34:11.337 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 7, 11, 15, 3] 2023-11-05T08:34:11.337 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-11-05T08:34:11.338 INFO:tasks.thrashosds.thrasher:primary_affinity 2023-11-05T08:34:11.338 INFO:tasks.thrashosds.thrasher:Setting osd 12 primary_affinity to 0.260655 2023-11-05T08:34:11.338 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 12 0.2606547026001782 2023-11-05T08:34:11.341 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:34:11.341 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":117,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:34:10.654856+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.94039440155029297,"average_primary_affinity":0.93999999761581421,"average_primary_affinity_weighted":0.93999999761581421}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.94039440155029297,"average_primary_affinity":0.93999999761581421,"average_primary_affinity_weighted":0.93999999761581421}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2023-11-05T08:32:42.050417+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":7,"min_size":5,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":26,"pg_placement_num":26,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":26,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"40","auid":0,"snap_mode":"selfmanaged","snap_seq":83,"snap_epoch":117,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"shecprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":16384,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":0.0463104248046875,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":43,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:56.093990+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:48.502629+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:46.337041+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:37.463501+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:54.776271+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:46.876488+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:52.787222+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:37.669537+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:45.201083+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:51.913597+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:26.992305+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:29.690235+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:33.658634+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:40.245020+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:43.650883+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"3.9","osds":[15,11,8,1,6,3,13]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"3.10","osds":[13,6,7,4,8,6,11]},{"pgid":"3.13","osds":[11,7,9,13,2,4,10]},{"pgid":"3.18","osds":[13,6,2,4,5,7,12]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":77,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:34:11.342 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-11-05T08:34:11.343 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-05T08:34:11.344 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:34:11.339+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:34:11.441 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:34:11.436+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:34:11.474 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:34:11.474 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-11-05T08:34:11.474 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-11-05T08:34:11.474 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:34:11.489 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-11-05T08:34:11.542 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:34:11.535+0000 7f1c6dde4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:34:11.605 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:34:11.605 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-11-05T08:34:11.605 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-11-05T08:34:11.606 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-11-05T08:34:11.606 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:34:11.619 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-11-05T08:34:11.642 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:34:11.636+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:34:11.661 INFO:teuthology.orchestra.run.smithi002.stderr:noscrub is unset 2023-11-05T08:34:11.710 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:34:11.712 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-11-05T08:34:11.728 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2023-11-05T08:34:11.663233+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":113500704,"num_objects":100,"num_object_clones":52,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":14,"num_objects_degraded":59,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":4,"num_read":472,"num_read_kb":132356,"num_write":864,"num_write_kb":121596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":23068672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1218,"ondisk_log_size":1218,"up":199,"acting":199,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":192,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":676764,"kb_used_data":205764,"kb_used_omap":29,"kb_used_meta":470690,"kb_avail":1509272676,"statfs":{"total":1546188226560,"available":1545495220224,"internally_reserved":0,"allocated":210702336,"data_stored":308683880,"data_compressed":2666174,"data_compressed_allocated":5771264,"data_compressed_original":12435456,"omap_allocated":30361,"internal_metadata":481986919},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":-1,"num_object_clones":-1,"num_object_copies":-7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.787864"},"pg_stats":[{"pgid":"3.19","version":"113'72","reported_seq":393,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-11-05T08:34:06.915793+0000","last_change":"2023-11-05T08:34:06.919092+0000","last_active":"2023-11-05T08:34:06.915793+0000","last_peered":"2023-11-05T08:34:06.915793+0000","last_clean":"2023-11-05T08:34:06.915793+0000","last_became_active":"2023-11-05T08:32:59.211522+0000","last_became_peered":"2023-11-05T08:32:59.211522+0000","last_unstale":"2023-11-05T08:34:06.915793+0000","last_undegraded":"2023-11-05T08:34:06.915793+0000","last_fullsized":"2023-11-05T08:34:06.915793+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":5,"last_scrub":"39'22","last_scrub_stamp":"2023-11-05T08:33:01.870959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:01.870959+0000","objects_scrubbed":3,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:26.120318+0000","scrub_duration":0.0087276880000000008,"objects_trimmed":3,"snaptrim_duration":0.051510817,"stat_sum":{"num_bytes":4636672,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":3027,"num_write":11,"num_write_kb":4144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,11,9,13,7,1],"acting":[8,4,11,9,13,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"2"},{"start":"44","length":"3"},{"start":"49","length":"1"}]},{"pgid":"3.18","version":"113'65","reported_seq":378,"reported_epoch":114,"state":"active+clean+remapped","last_fresh":"2023-11-05T08:34:07.899721+0000","last_change":"2023-11-05T08:34:06.922017+0000","last_active":"2023-11-05T08:34:07.899721+0000","last_peered":"2023-11-05T08:34:07.899721+0000","last_clean":"2023-11-05T08:34:07.899721+0000","last_became_active":"2023-11-05T08:32:59.217241+0000","last_became_peered":"2023-11-05T08:32:59.217241+0000","last_unstale":"2023-11-05T08:34:07.899721+0000","last_undegraded":"2023-11-05T08:34:07.899721+0000","last_fullsized":"2023-11-05T08:34:07.899721+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":5,"last_scrub":"52'28","last_scrub_stamp":"2023-11-05T08:33:13.971775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:13.971775+0000","objects_scrubbed":5,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:37.325135+0000","scrub_duration":0.019915236999999999,"objects_trimmed":1,"snaptrim_duration":0.044767529,"stat_sum":{"num_bytes":6160384,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3739,"num_write":31,"num_write_kb":4336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,2147483647,4,5,7,12],"acting":[13,6,2,4,5,7,12],"avail_no_missing":["13(0)","2(2)","4(3)","5(4)","6(1)","7(5)","12(6)"],"object_location_counts":[{"shards":"2(2),4(3),5(4),6(1),7(5),12(6),13(0)","objects":3}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"2"},{"start":"44","length":"3"},{"start":"49","length":"1"}]},{"pgid":"3.17","version":"105'36","reported_seq":405,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-11-05T08:34:06.876329+0000","last_change":"2023-11-05T08:34:06.879553+0000","last_active":"2023-11-05T08:34:06.876329+0000","last_peered":"2023-11-05T08:34:06.876329+0000","last_clean":"2023-11-05T08:34:06.876329+0000","last_became_active":"2023-11-05T08:33:50.677365+0000","last_became_peered":"2023-11-05T08:33:50.677365+0000","last_unstale":"2023-11-05T08:34:06.876329+0000","last_undegraded":"2023-11-05T08:34:06.876329+0000","last_fullsized":"2023-11-05T08:34:06.876329+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"49'17","last_scrub_stamp":"2023-11-05T08:33:13.975356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:13.975356+0000","objects_scrubbed":3,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:29.585773+0000","scrub_duration":0.014066680999999999,"objects_trimmed":0,"snaptrim_duration":0.0038009839999999999,"stat_sum":{"num_bytes":2588672,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":45,"num_write_kb":2176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":999424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,10,9,7,1,13],"acting":[5,3,10,9,7,1,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"2"},{"start":"44","length":"3"},{"start":"49","length":"1"}]},{"pgid":"3.16","version":"97'48","reported_seq":348,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-11-05T08:34:06.879254+0000","last_change":"2023-11-05T08:34:06.888477+0000","last_active":"2023-11-05T08:34:06.879254+0000","last_peered":"2023-11-05T08:34:06.879254+0000","last_clean":"2023-11-05T08:34:06.879254+0000","last_became_active":"2023-11-05T08:32:59.210858+0000","last_became_peered":"2023-11-05T08:32:59.210858+0000","last_unstale":"2023-11-05T08:34:06.879254+0000","last_undegraded":"2023-11-05T08:34:06.879254+0000","last_fullsized":"2023-11-05T08:34:06.879254+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":5,"last_scrub":"46'13","last_scrub_stamp":"2023-11-05T08:33:00.659926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:00.659926+0000","objects_scrubbed":1,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:22.744133+0000","scrub_duration":0.0077151709999999998,"objects_trimmed":0,"snaptrim_duration":0.0097672939999999993,"stat_sum":{"num_bytes":4669440,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":15,"num_write_kb":4560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15,4,8,9,5,10],"acting":[11,15,4,8,9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"2"},{"start":"44","length":"3"},{"start":"49","length":"1"}]},{"pgid":"3.15","version":"36'15","reported_seq":290,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-11-05T08:34:06.886129+0000","last_change":"2023-11-05T08:34:06.899394+0000","last_active":"2023-11-05T08:34:06.886129+0000","last_peered":"2023-11-05T08:34:06.886129+0000","last_clean":"2023-11-05T08:34:06.886129+0000","last_became_active":"2023-11-05T08:33:50.675923+0000","last_became_peered":"2023-11-05T08:33:50.675923+0000","last_unstale":"2023-11-05T08:34:06.886129+0000","last_undegraded":"2023-11-05T08:34:06.886129+0000","last_fullsized":"2023-11-05T08:34:06.886129+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"36'15","last_scrub_stamp":"2023-11-05T08:33:52.537969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:52.537969+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:35:17.273297+0000","scrub_duration":0.0054037720000000003,"objects_trimmed":0,"snaptrim_duration":0.014283227000000001,"stat_sum":{"num_bytes":32768,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,3,13,14,2,15],"acting":[9,6,3,13,14,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"2"},{"start":"44","length":"3"},{"start":"49","length":"1"}]},{"pgid":"3.14","version":"101'27","reported_seq":303,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-11-05T08:34:06.899764+0000","last_change":"2023-11-05T08:34:06.916998+0000","last_active":"2023-11-05T08:34:06.899764+0000","last_peered":"2023-11-05T08:34:06.899764+0000","last_clean":"2023-11-05T08:34:06.899764+0000","last_became_active":"2023-11-05T08:33:50.674132+0000","last_became_peered":"2023-11-05T08:33:50.674132+0000","last_unstale":"2023-11-05T08:34:06.899764+0000","last_undegraded":"2023-11-05T08:34:06.899764+0000","last_fullsized":"2023-11-05T08:34:06.899764+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"49'18","last_scrub_stamp":"2023-11-05T08:33:04.812402+0000","last_deep_scrub":"49'18","last_deep_scrub_stamp":"2023-11-05T08:33:04.812402+0000","last_clean_scrub_stamp":"2023-11-05T08:33:04.812402+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0086643829999999995,"objects_trimmed":0,"snaptrim_duration":0.031284815000000001,"stat_sum":{"num_bytes":294912,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1966080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,15,1,13,8,4],"acting":[3,9,15,1,13,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"2"},{"start":"44","length":"3"},{"start":"49","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":103,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-11-05T08:34:05.863169+0000","last_change":"2023-11-05T08:33:50.142139+0000","last_active":"2023-11-05T08:34:05.863169+0000","last_peered":"2023-11-05T08:34:05.863169+0000","last_clean":"2023-11-05T08:34:05.863169+0000","last_became_active":"2023-11-05T08:32:31.941858+0000","last_became_peered":"2023-11-05T08:32:31.941858+0000","last_unstale":"2023-11-05T08:34:05.863169+0000","last_undegraded":"2023-11-05T08:34:05.863169+0000","last_fullsized":"2023-11-05T08:34:05.863169+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:33:50.141700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:33:50.141700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:51.467865+0000","scrub_duration":0.0025514660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"35'9","reported_seq":196,"reported_epoch":114,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:34:07.900712+0000","last_change":"2023-11-05T08:32:54.153555+0000","last_active":"2023-11-05T08:34:07.900712+0000","last_peered":"2023-11-05T08:34:07.900712+0000","last_clean":"2023-11-05T08:32:48.102597+0000","last_became_active":"2023-11-05T08:32:54.153555+0000","last_became_peered":"2023-11-05T08:32:54.153555+0000","last_unstale":"2023-11-05T08:34:07.900712+0000","last_undegraded":"2023-11-05T08:32:54.121924+0000","last_fullsized":"2023-11-05T08:32:54.120000+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":38,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1089536,"num_objects":1,"num_object_clones":1,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":1064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,13,1,15,12,7,9],"acting":[2147483647,13,1,15,12,7,9],"avail_no_missing":["13(1)","1(2)","7(5)","9(6)","12(4)","15(3)"],"object_location_counts":[{"shards":"1(2),7(5),9(6),12(4),13(1),15(3)","objects":1}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":104,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-11-05T08:34:05.856626+0000","last_change":"2023-11-05T08:33:52.971633+0000","last_active":"2023-11-05T08:34:05.856626+0000","last_peered":"2023-11-05T08:34:05.856626+0000","last_clean":"2023-11-05T08:34:05.856626+0000","last_became_active":"2023-11-05T08:32:31.941207+0000","last_became_peered":"2023-11-05T08:32:31.941207+0000","last_unstale":"2023-11-05T08:34:05.856626+0000","last_undegraded":"2023-11-05T08:34:05.856626+0000","last_fullsized":"2023-11-05T08:34:05.856626+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:33:52.971456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:33:52.971456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:35:07.735098+0000","scrub_duration":0.002374414,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"39'12","reported_seq":207,"reported_epoch":113,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:34:06.876062+0000","last_change":"2023-11-05T08:32:54.151639+0000","last_active":"2023-11-05T08:34:06.876062+0000","last_peered":"2023-11-05T08:34:06.876062+0000","last_clean":"2023-11-05T08:32:49.074431+0000","last_became_active":"2023-11-05T08:32:54.151639+0000","last_became_peered":"2023-11-05T08:32:54.151639+0000","last_unstale":"2023-11-05T08:34:06.876062+0000","last_undegraded":"2023-11-05T08:32:54.120694+0000","last_fullsized":"2023-11-05T08:32:54.119652+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":37,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030081180000000002,"stat_sum":{"num_bytes":319488,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,2147483647,8,5,11],"acting":[4,6,9,2147483647,8,5,11],"avail_no_missing":["4(0)","5(5)","6(1)","8(4)","9(2)","11(6)"],"object_location_counts":[{"shards":"4(0),5(5),6(1),8(4),9(2),11(6)","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":104,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-11-05T08:34:04.846968+0000","last_change":"2023-11-05T08:33:50.446617+0000","last_active":"2023-11-05T08:34:04.846968+0000","last_peered":"2023-11-05T08:34:04.846968+0000","last_clean":"2023-11-05T08:34:04.846968+0000","last_became_active":"2023-11-05T08:32:31.940312+0000","last_became_peered":"2023-11-05T08:32:31.940312+0000","last_unstale":"2023-11-05T08:34:04.846968+0000","last_undegraded":"2023-11-05T08:34:04.846968+0000","last_fullsized":"2023-11-05T08:34:04.846968+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:33:50.446312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:33:50.446312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:57.987976+0000","scrub_duration":0.0037389620000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"39'12","reported_seq":206,"reported_epoch":113,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:34:06.868331+0000","last_change":"2023-11-05T08:32:54.141915+0000","last_active":"2023-11-05T08:34:06.868331+0000","last_peered":"2023-11-05T08:34:06.868331+0000","last_clean":"2023-11-05T08:32:49.088117+0000","last_became_active":"2023-11-05T08:32:54.141915+0000","last_became_peered":"2023-11-05T08:32:54.141915+0000","last_unstale":"2023-11-05T08:34:06.868331+0000","last_undegraded":"2023-11-05T08:32:54.115420+0000","last_fullsized":"2023-11-05T08:32:54.114438+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":37,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.011061272,"stat_sum":{"num_bytes":1376256,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,4,1,2147483647,14,13],"acting":[8,2,4,1,2147483647,14,13],"avail_no_missing":["8(0)","1(3)","2(1)","4(2)","13(6)","14(5)"],"object_location_counts":[{"shards":"1(3),2(1),4(2),8(0),13(6),14(5)","objects":1}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":101,"reported_epoch":112,"state":"active+undersized","last_fresh":"2023-11-05T08:34:06.483521+0000","last_change":"2023-11-05T08:32:51.087583+0000","last_active":"2023-11-05T08:34:06.483521+0000","last_peered":"2023-11-05T08:34:06.483521+0000","last_clean":"2023-11-05T08:32:49.071396+0000","last_became_active":"2023-11-05T08:32:51.087583+0000","last_became_peered":"2023-11-05T08:32:51.087583+0000","last_unstale":"2023-11-05T08:34:06.483521+0000","last_undegraded":"2023-11-05T08:34:06.483521+0000","last_fullsized":"2023-11-05T08:32:51.085658+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:32:30.915125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"103'31","reported_seq":232,"reported_epoch":113,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:34:06.867158+0000","last_change":"2023-11-05T08:32:54.139002+0000","last_active":"2023-11-05T08:34:06.867158+0000","last_peered":"2023-11-05T08:34:06.867158+0000","last_clean":"2023-11-05T08:32:49.088591+0000","last_became_active":"2023-11-05T08:32:54.139002+0000","last_became_peered":"2023-11-05T08:32:54.139002+0000","last_unstale":"2023-11-05T08:34:06.867158+0000","last_undegraded":"2023-11-05T08:32:54.118261+0000","last_fullsized":"2023-11-05T08:32:54.117408+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":37,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.020413489999999999,"stat_sum":{"num_bytes":5988352,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":12,"num_read_kb":2382,"num_write":18,"num_write_kb":5848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,4,13,7,2147483647,6],"acting":[15,8,4,13,7,2147483647,6],"avail_no_missing":["15(0)","4(2)","6(6)","7(4)","8(1)","13(3)"],"object_location_counts":[{"shards":"4(2),6(6),7(4),8(1),13(3),15(0)","objects":5}],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.3","version":"97'17","reported_seq":401,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-11-05T08:34:07.900377+0000","last_change":"2023-11-05T08:34:06.889717+0000","last_active":"2023-11-05T08:34:07.900377+0000","last_peered":"2023-11-05T08:34:07.900377+0000","last_clean":"2023-11-05T08:34:07.900377+0000","last_became_active":"2023-11-05T08:33:50.683698+0000","last_became_peered":"2023-11-05T08:33:50.683698+0000","last_unstale":"2023-11-05T08:34:07.900377+0000","last_undegraded":"2023-11-05T08:34:07.900377+0000","last_fullsized":"2023-11-05T08:34:07.900377+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'17","last_scrub_stamp":"2023-11-05T08:33:53.073917+0000","last_deep_scrub":"97'17","last_deep_scrub_stamp":"2023-11-05T08:33:53.073917+0000","last_clean_scrub_stamp":"2023-11-05T08:33:53.073917+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:35:16.149535+0000","scrub_duration":0.0068216960000000004,"objects_trimmed":0,"snaptrim_duration":0.013339142999999999,"stat_sum":{"num_bytes":524288,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":1537,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,4,15,2,5,3],"acting":[13,7,4,15,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"2"},{"start":"44","length":"3"},{"start":"49","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":109,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-11-05T08:34:06.558628+0000","last_change":"2023-11-05T08:33:53.537740+0000","last_active":"2023-11-05T08:34:06.558628+0000","last_peered":"2023-11-05T08:34:06.558628+0000","last_clean":"2023-11-05T08:34:06.558628+0000","last_became_active":"2023-11-05T08:33:47.679907+0000","last_became_peered":"2023-11-05T08:33:47.679907+0000","last_unstale":"2023-11-05T08:34:06.558628+0000","last_undegraded":"2023-11-05T08:34:06.558628+0000","last_fullsized":"2023-11-05T08:34:06.558628+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:33:53.537442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:33:53.537442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:35:15.667752+0000","scrub_duration":0.003485906,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3],"acting":[12,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.0","version":"33'9","reported_seq":364,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-11-05T08:34:06.878720+0000","last_change":"2023-11-05T08:34:06.881241+0000","last_active":"2023-11-05T08:34:06.878720+0000","last_peered":"2023-11-05T08:34:06.878720+0000","last_clean":"2023-11-05T08:34:06.878720+0000","last_became_active":"2023-11-05T08:32:54.144790+0000","last_became_peered":"2023-11-05T08:32:54.144790+0000","last_unstale":"2023-11-05T08:34:06.878720+0000","last_undegraded":"2023-11-05T08:34:06.878720+0000","last_fullsized":"2023-11-05T08:34:06.878720+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'9","last_scrub_stamp":"2023-11-05T08:33:03.810524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:03.810524+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:15.106203+0000","scrub_duration":0.0066041449999999996,"objects_trimmed":0,"snaptrim_duration":0.0030128989999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,15,1,5,6,4],"acting":[14,2,15,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"2"},{"start":"44","length":"3"},{"start":"49","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":104,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-11-05T08:34:06.879231+0000","last_change":"2023-11-05T08:34:02.933179+0000","last_active":"2023-11-05T08:34:06.879231+0000","last_peered":"2023-11-05T08:34:06.879231+0000","last_clean":"2023-11-05T08:34:06.879231+0000","last_became_active":"2023-11-05T08:32:31.940564+0000","last_became_peered":"2023-11-05T08:32:31.940564+0000","last_unstale":"2023-11-05T08:34:06.879231+0000","last_undegraded":"2023-11-05T08:34:06.879231+0000","last_fullsized":"2023-11-05T08:34:06.879231+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:34:02.932904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:34:02.932904+0000","last_clean_scrub_stamp":"2023-11-05T08:34:02.932904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:35:11.818210+0000","scrub_duration":0.0033566400000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.1","version":"113'46","reported_seq":412,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-11-05T08:34:06.904795+0000","last_change":"2023-11-05T08:34:06.912461+0000","last_active":"2023-11-05T08:34:06.904795+0000","last_peered":"2023-11-05T08:34:06.904795+0000","last_clean":"2023-11-05T08:34:06.904795+0000","last_became_active":"2023-11-05T08:32:54.144895+0000","last_became_peered":"2023-11-05T08:32:54.144895+0000","last_unstale":"2023-11-05T08:34:06.904795+0000","last_undegraded":"2023-11-05T08:34:06.904795+0000","last_fullsized":"2023-11-05T08:34:06.904795+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'11","last_scrub_stamp":"2023-11-05T08:33:04.003557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:04.003557+0000","objects_scrubbed":3,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:21.790445+0000","scrub_duration":0.010761797,"objects_trimmed":2,"snaptrim_duration":0.036461375999999997,"stat_sum":{"num_bytes":1163264,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":772,"num_write":7,"num_write_kb":2480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,5,10,2,13],"acting":[4,11,9,5,10,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"2"},{"start":"44","length":"3"},{"start":"49","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-11-05T08:34:04.846610+0000","last_change":"2023-11-05T08:34:02.462635+0000","last_active":"2023-11-05T08:34:04.846610+0000","last_peered":"2023-11-05T08:34:04.846610+0000","last_clean":"2023-11-05T08:34:04.846610+0000","last_became_active":"2023-11-05T08:33:47.699438+0000","last_became_peered":"2023-11-05T08:33:47.699438+0000","last_unstale":"2023-11-05T08:34:04.846610+0000","last_undegraded":"2023-11-05T08:34:04.846610+0000","last_fullsized":"2023-11-05T08:34:04.846610+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:34:02.462335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:34:02.462335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:35:14.687268+0000","scrub_duration":0.003490197,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"111'33","reported_seq":418,"reported_epoch":114,"state":"active+clean","last_fresh":"2023-11-05T08:34:07.900030+0000","last_change":"2023-11-05T08:34:06.899065+0000","last_active":"2023-11-05T08:34:07.900030+0000","last_peered":"2023-11-05T08:34:07.900030+0000","last_clean":"2023-11-05T08:34:07.900030+0000","last_became_active":"2023-11-05T08:33:50.678084+0000","last_became_peered":"2023-11-05T08:33:50.678084+0000","last_unstale":"2023-11-05T08:34:07.900030+0000","last_undegraded":"2023-11-05T08:34:07.900030+0000","last_fullsized":"2023-11-05T08:34:07.900030+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'25","last_scrub_stamp":"2023-11-05T08:33:52.100890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:52.100890+0000","objects_scrubbed":1,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:35:00.127267+0000","scrub_duration":0.007958952,"objects_trimmed":0,"snaptrim_duration":0.024161518,"stat_sum":{"num_bytes":3670016,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":2080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,3,6,15,1,11],"acting":[13,2,3,6,15,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"2"},{"start":"44","length":"3"},{"start":"49","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":104,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-11-05T08:34:06.350972+0000","last_change":"2023-11-05T08:34:00.133458+0000","last_active":"2023-11-05T08:34:06.350972+0000","last_peered":"2023-11-05T08:34:06.350972+0000","last_clean":"2023-11-05T08:34:06.350972+0000","last_became_active":"2023-11-05T08:32:31.939705+0000","last_became_peered":"2023-11-05T08:32:31.939705+0000","last_unstale":"2023-11-05T08:34:06.350972+0000","last_undegraded":"2023-11-05T08:34:06.350972+0000","last_fullsized":"2023-11-05T08:34:06.350972+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:34:00.133184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:30.915125+0000","last_clean_scrub_stamp":"2023-11-05T08:34:00.133184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:35:05.010050+0000","scrub_duration":0.0035944599999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"25'157","reported_seq":303,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-11-05T08:34:06.878218+0000","last_change":"2023-11-05T08:33:41.087862+0000","last_active":"2023-11-05T08:34:06.878218+0000","last_peered":"2023-11-05T08:34:06.878218+0000","last_clean":"2023-11-05T08:34:06.878218+0000","last_became_active":"2023-11-05T08:32:28.758834+0000","last_became_peered":"2023-11-05T08:32:28.758834+0000","last_unstale":"2023-11-05T08:34:06.878218+0000","last_undegraded":"2023-11-05T08:34:06.878218+0000","last_fullsized":"2023-11-05T08:34:06.878218+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'157","last_scrub_stamp":"2023-11-05T08:33:41.087696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:27.693099+0000","last_clean_scrub_stamp":"2023-11-05T08:33:41.087696+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:35:09.900066+0000","scrub_duration":0.0045831459999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.8","version":"108'34","reported_seq":232,"reported_epoch":113,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:34:06.874730+0000","last_change":"2023-11-05T08:32:54.151946+0000","last_active":"2023-11-05T08:34:06.874730+0000","last_peered":"2023-11-05T08:34:06.874730+0000","last_clean":"2023-11-05T08:32:49.090022+0000","last_became_active":"2023-11-05T08:32:54.151946+0000","last_became_peered":"2023-11-05T08:32:54.151946+0000","last_unstale":"2023-11-05T08:34:06.874730+0000","last_undegraded":"2023-11-05T08:32:54.119646+0000","last_fullsized":"2023-11-05T08:32:54.118811+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":37,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.013731910999999999,"stat_sum":{"num_bytes":4816896,"num_objects":6,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":10,"num_read_kb":1684,"num_write":27,"num_write_kb":4144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2147483647,2,1,14,15,6],"acting":[5,2147483647,2,1,14,15,6],"avail_no_missing":["5(0)","1(3)","2(2)","6(6)","14(4)","15(5)"],"object_location_counts":[{"shards":"1(3),2(2),5(0),6(6),14(4),15(5)","objects":6}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.9","version":"103'64","reported_seq":2,"reported_epoch":113,"state":"peering","last_fresh":"2023-11-05T08:33:56.802940+0000","last_change":"2023-11-05T08:34:07.721770+0000","last_active":"2023-11-05T08:34:07.721770+0000","last_peered":"2023-11-05T08:33:56.802940+0000","last_clean":"2023-11-05T08:32:49.072716+0000","last_became_active":"2023-11-05T08:33:50.691854+0000","last_became_peered":"2023-11-05T08:33:50.691854+0000","last_unstale":"2023-11-05T08:33:56.802940+0000","last_undegraded":"2023-11-05T08:33:50.658262+0000","last_fullsized":"2023-11-05T08:33:50.657391+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015323719999999999,"stat_sum":{"num_bytes":14761984,"num_objects":13,"num_object_clones":9,"num_object_copies":91,"num_objects_missing_on_primary":0,"num_objects_missing":14,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":35,"num_read_kb":16477,"num_write":100,"num_write_kb":14416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":10584064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,8,1,6,3,13],"acting":[2147483647,11,8,1,6,3,2147483647],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.a","version":"113'57","reported_seq":437,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-11-05T08:34:06.910829+0000","last_change":"2023-11-05T08:34:06.915471+0000","last_active":"2023-11-05T08:34:06.910829+0000","last_peered":"2023-11-05T08:34:06.910829+0000","last_clean":"2023-11-05T08:34:06.910829+0000","last_became_active":"2023-11-05T08:33:50.684400+0000","last_became_peered":"2023-11-05T08:33:50.684400+0000","last_unstale":"2023-11-05T08:34:06.910829+0000","last_undegraded":"2023-11-05T08:34:06.910829+0000","last_fullsized":"2023-11-05T08:34:06.910829+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'44","last_scrub_stamp":"2023-11-05T08:34:01.079975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:34:01.079975+0000","objects_scrubbed":3,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:35:01.355870+0000","scrub_duration":0.0091389039999999998,"objects_trimmed":4,"snaptrim_duration":0.038502712000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":25,"num_read_kb":17912,"num_write":36,"num_write_kb":3696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":6406144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,9,14,6,11],"acting":[3,7,1,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"2"},{"start":"44","length":"3"},{"start":"49","length":"1"}]},{"pgid":"3.b","version":"111'60","reported_seq":264,"reported_epoch":113,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:34:06.876325+0000","last_change":"2023-11-05T08:32:51.095575+0000","last_active":"2023-11-05T08:34:06.876325+0000","last_peered":"2023-11-05T08:34:06.876325+0000","last_clean":"2023-11-05T08:32:49.072970+0000","last_became_active":"2023-11-05T08:32:51.095575+0000","last_became_peered":"2023-11-05T08:32:51.095575+0000","last_unstale":"2023-11-05T08:34:06.876325+0000","last_undegraded":"2023-11-05T08:32:51.085902+0000","last_fullsized":"2023-11-05T08:32:51.084936+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":37,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021279290000000002,"stat_sum":{"num_bytes":12435456,"num_objects":14,"num_object_clones":10,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":1,"num_read":38,"num_read_kb":17439,"num_write":65,"num_write_kb":12768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2147483647,1,12,6,7,9],"acting":[4,2147483647,1,12,6,7,9],"avail_no_missing":["4(0)","1(2)","6(4)","7(5)","9(6)","12(3)"],"object_location_counts":[{"shards":"1(2),4(0),6(4),7(5),9(6),12(3)","objects":14}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.c","version":"79'6","reported_seq":196,"reported_epoch":113,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:34:06.878686+0000","last_change":"2023-11-05T08:32:51.099852+0000","last_active":"2023-11-05T08:34:06.878686+0000","last_peered":"2023-11-05T08:34:06.878686+0000","last_clean":"2023-11-05T08:32:49.087645+0000","last_became_active":"2023-11-05T08:32:51.099852+0000","last_became_peered":"2023-11-05T08:32:51.099852+0000","last_unstale":"2023-11-05T08:34:06.878686+0000","last_undegraded":"2023-11-05T08:32:51.089699+0000","last_fullsized":"2023-11-05T08:32:51.089041+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":37,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015630430000000001,"stat_sum":{"num_bytes":999424,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":7,"num_read_kb":914,"num_write":4,"num_write_kb":976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,2147483647,10,6,12,5],"acting":[11,8,2147483647,10,6,12,5],"avail_no_missing":["11(0)","5(6)","6(4)","8(1)","10(3)","12(5)"],"object_location_counts":[{"shards":"5(6),6(4),8(1),10(3),11(0),12(5)","objects":3}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.d","version":"103'36","reported_seq":267,"reported_epoch":113,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:34:06.877030+0000","last_change":"2023-11-05T08:33:50.690639+0000","last_active":"2023-11-05T08:34:06.877030+0000","last_peered":"2023-11-05T08:34:06.877030+0000","last_clean":"2023-11-05T08:32:49.088145+0000","last_became_active":"2023-11-05T08:33:50.690639+0000","last_became_peered":"2023-11-05T08:33:50.690639+0000","last_unstale":"2023-11-05T08:34:06.877030+0000","last_undegraded":"2023-11-05T08:33:50.664683+0000","last_fullsized":"2023-11-05T08:33:50.663817+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":37,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.024583713,"stat_sum":{"num_bytes":5013504,"num_objects":7,"num_object_clones":5,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":6009,"num_write":23,"num_write_kb":3536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3112960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,2147483647,8,4],"acting":[7,3,14,10,2147483647,8,4],"avail_no_missing":["7(0)","3(1)","4(6)","8(5)","10(3)","14(2)"],"object_location_counts":[{"shards":"3(1),4(6),7(0),8(5),10(3),14(2)","objects":7}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.e","version":"113'138","reported_seq":481,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-11-05T08:34:06.911302+0000","last_change":"2023-11-05T08:34:06.915304+0000","last_active":"2023-11-05T08:34:06.911302+0000","last_peered":"2023-11-05T08:34:06.911302+0000","last_clean":"2023-11-05T08:34:06.911302+0000","last_became_active":"2023-11-05T08:32:44.949185+0000","last_became_peered":"2023-11-05T08:32:44.949185+0000","last_unstale":"2023-11-05T08:34:06.911302+0000","last_undegraded":"2023-11-05T08:34:06.911302+0000","last_fullsized":"2023-11-05T08:34:06.911302+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'96","last_scrub_stamp":"2023-11-05T08:33:48.102281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:48.102281+0000","objects_scrubbed":12,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:53.327483+0000","scrub_duration":0.014167007000000001,"objects_trimmed":2,"snaptrim_duration":0.031759298999999998,"stat_sum":{"num_bytes":7028736,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":45,"num_read_kb":12450,"num_write":75,"num_write_kb":13632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1,11,8,6,12],"acting":[2,9,1,11,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"2"},{"start":"44","length":"3"},{"start":"49","length":"1"}]},{"pgid":"3.f","version":"108'85","reported_seq":294,"reported_epoch":113,"state":"active+undersized+degraded","last_fresh":"2023-11-05T08:34:06.882111+0000","last_change":"2023-11-05T08:32:51.104425+0000","last_active":"2023-11-05T08:34:06.882111+0000","last_peered":"2023-11-05T08:34:06.882111+0000","last_clean":"2023-11-05T08:32:49.098585+0000","last_became_active":"2023-11-05T08:32:51.104425+0000","last_became_peered":"2023-11-05T08:32:51.104425+0000","last_unstale":"2023-11-05T08:34:06.882111+0000","last_undegraded":"2023-11-05T08:32:51.091474+0000","last_fullsized":"2023-11-05T08:32:51.089784+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:32:43.000991+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":37,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012093372,"stat_sum":{"num_bytes":22069248,"num_objects":20,"num_object_clones":16,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":50,"num_read_kb":25938,"num_write":122,"num_write_kb":23184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,14,13,8,10,2147483647],"acting":[11,2,14,13,8,10,2147483647],"avail_no_missing":["11(0)","2(1)","8(4)","10(5)","13(3)","14(2)"],"object_location_counts":[{"shards":"2(1),8(4),10(5),11(0),13(3),14(2)","objects":20}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.10","version":"113'76","reported_seq":390,"reported_epoch":114,"state":"active+clean+remapped","last_fresh":"2023-11-05T08:34:07.899220+0000","last_change":"2023-11-05T08:34:06.911853+0000","last_active":"2023-11-05T08:34:07.899220+0000","last_peered":"2023-11-05T08:34:07.899220+0000","last_clean":"2023-11-05T08:34:07.899220+0000","last_became_active":"2023-11-05T08:32:59.210399+0000","last_became_peered":"2023-11-05T08:32:59.210399+0000","last_unstale":"2023-11-05T08:34:07.899220+0000","last_undegraded":"2023-11-05T08:34:07.899220+0000","last_fullsized":"2023-11-05T08:34:07.899220+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":5,"last_scrub":"109'74","last_scrub_stamp":"2023-11-05T08:34:03.951729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:34:03.951729+0000","objects_scrubbed":4,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:35:22.249843+0000","scrub_duration":0.010194440000000001,"objects_trimmed":1,"snaptrim_duration":0.036947638999999997,"stat_sum":{"num_bytes":6897664,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":21,"num_read_kb":5385,"num_write":48,"num_write_kb":8880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,7,4,8,2147483647,11],"acting":[13,6,7,4,8,6,11],"avail_no_missing":["13(0)","4(3)","6(1)","6(5)","7(2)","8(4)","11(6)"],"object_location_counts":[{"shards":"4(3),6(1),6(5),7(2),8(4),11(6),13(0)","objects":3}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"2"},{"start":"44","length":"3"},{"start":"49","length":"1"}]},{"pgid":"3.11","version":"113'41","reported_seq":414,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-11-05T08:34:06.888735+0000","last_change":"2023-11-05T08:34:06.892434+0000","last_active":"2023-11-05T08:34:06.888735+0000","last_peered":"2023-11-05T08:34:06.888735+0000","last_clean":"2023-11-05T08:34:06.888735+0000","last_became_active":"2023-11-05T08:32:59.214193+0000","last_became_peered":"2023-11-05T08:32:59.214193+0000","last_unstale":"2023-11-05T08:34:06.888735+0000","last_undegraded":"2023-11-05T08:34:06.888735+0000","last_fullsized":"2023-11-05T08:34:06.888735+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":5,"last_scrub":"46'17","last_scrub_stamp":"2023-11-05T08:33:00.070510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:00.070510+0000","objects_scrubbed":3,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:29.035324+0000","scrub_duration":0.0081678800000000006,"objects_trimmed":1,"snaptrim_duration":0.025545774,"stat_sum":{"num_bytes":6373376,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16566,"num_write":10,"num_write_kb":4272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,15,9,5,6,1],"acting":[10,11,15,9,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"2"},{"start":"44","length":"3"},{"start":"49","length":"1"}]},{"pgid":"3.12","version":"37'7","reported_seq":319,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-11-05T08:34:06.886024+0000","last_change":"2023-11-05T08:34:06.916430+0000","last_active":"2023-11-05T08:34:06.886024+0000","last_peered":"2023-11-05T08:34:06.886024+0000","last_clean":"2023-11-05T08:34:06.886024+0000","last_became_active":"2023-11-05T08:33:50.685746+0000","last_became_peered":"2023-11-05T08:33:50.685746+0000","last_unstale":"2023-11-05T08:34:06.886024+0000","last_undegraded":"2023-11-05T08:34:06.886024+0000","last_fullsized":"2023-11-05T08:34:06.886024+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":5,"last_scrub":"37'7","last_scrub_stamp":"2023-11-05T08:33:02.650387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:02.650387+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:31.578340+0000","scrub_duration":0.0067664470000000001,"objects_trimmed":0,"snaptrim_duration":0.032113293000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8,6,4,3,15],"acting":[9,7,8,6,4,3,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"2"},{"start":"44","length":"3"},{"start":"49","length":"1"}]},{"pgid":"3.13","version":"97'25","reported_seq":379,"reported_epoch":113,"state":"active+clean+remapped","last_fresh":"2023-11-05T08:34:06.883470+0000","last_change":"2023-11-05T08:34:06.898883+0000","last_active":"2023-11-05T08:34:06.883470+0000","last_peered":"2023-11-05T08:34:06.883470+0000","last_clean":"2023-11-05T08:34:06.883470+0000","last_became_active":"2023-11-05T08:32:59.212969+0000","last_became_peered":"2023-11-05T08:32:59.212969+0000","last_unstale":"2023-11-05T08:34:06.883470+0000","last_undegraded":"2023-11-05T08:34:06.883470+0000","last_fullsized":"2023-11-05T08:34:06.883470+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'14","last_scrub_stamp":"2023-11-05T08:33:02.644028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-05T08:32:43.000991+0000","last_clean_scrub_stamp":"2023-11-05T08:33:02.644028+0000","objects_scrubbed":1,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T08:34:20.169583+0000","scrub_duration":0.007386386,"objects_trimmed":0,"snaptrim_duration":0.020096545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,9,13,2147483647,4,10],"acting":[11,7,9,13,2,4,10],"avail_no_missing":["11(0)","2(4)","4(5)","7(1)","9(2)","10(6)","13(3)"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"2"},{"start":"5","length":"4"},{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"10","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"3"},{"start":"2b","length":"2"},{"start":"2e","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3b","length":"1"},{"start":"3d","length":"2"},{"start":"41","length":"2"},{"start":"44","length":"3"},{"start":"49","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":112910336,"num_objects":98,"num_object_clones":52,"num_object_copies":686,"num_objects_missing_on_primary":0,"num_objects_missing":14,"num_objects_degraded":59,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":4,"num_read":336,"num_read_kb":132238,"num_write":661,"num_write_kb":118136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":23068672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":187592704,"data_stored":284393472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1061,"ondisk_log_size":1061,"up":182,"acting":182,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":15,"acting":15,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":13898,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":22,"seq":94489280535,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46108,"kb_used_data":14920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325732,"statfs":{"total":96636764160,"available":96589549568,"internally_reserved":0,"allocated":15278080,"data_stored":24813728,"data_compressed":182223,"data_compressed_allocated":409600,"data_compressed_original":1265664,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":1,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.256,"15min":1.256},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":3.4590000000000001,"5min":3.4590000000000001,"15min":3.4590000000000001},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.341,"5min":1.341,"15min":1.341},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":2.7280000000000002,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":1.3640000000000001}]},{"osd":2,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":2.2080000000000002,"5min":2.2080000000000002,"15min":2.2080000000000002},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.218,"5min":1.218,"15min":1.218},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":3.0139999999999998,"5min":3.0139999999999998,"15min":3.0139999999999998},"last":1.2170000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2410000000000001}]},{"osd":4,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.222,"15min":1.222},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":3.5979999999999999,"5min":3.5979999999999999,"15min":3.5979999999999999},"last":0.997},{"interface":"front","average":{"1min":1.321,"5min":1.321,"15min":1.321},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":3.383,"5min":3.383,"15min":3.383},"last":1.7470000000000001}]},{"osd":5,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.466,"15min":1.466},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":3.5059999999999998,"5min":3.5059999999999998,"15min":3.5059999999999998},"last":1.4219999999999999},{"interface":"front","average":{"1min":1.5029999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":3.1579999999999999,"5min":3.1579999999999999,"15min":3.1579999999999999},"last":1.054}]},{"osd":6,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.625,"5min":1.625,"15min":1.625},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":3.4820000000000002,"5min":3.4820000000000002,"15min":3.4820000000000002},"last":1.621},{"interface":"front","average":{"1min":1.579,"5min":1.579,"15min":1.579},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":3.0059999999999998,"5min":3.0059999999999998,"15min":3.0059999999999998},"last":1.294}]},{"osd":7,"last update":"Sun Nov 5 08:33:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"min":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"max":{"1min":2.165,"5min":2.165,"15min":2.165},"last":2.0070000000000001},{"interface":"front","average":{"1min":1.54,"5min":1.54,"15min":1.54},"min":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"max":{"1min":2.2759999999999998,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":1.889}]},{"osd":8,"last update":"Sun Nov 5 08:33:52 2023","interfaces":[{"interface":"back","average":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.6850000000000001},"min":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":3.5739999999999998,"5min":3.5739999999999998,"15min":3.5739999999999998},"last":1.8340000000000001},{"interface":"front","average":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"min":{"1min":1.044,"5min":1.044,"15min":1.044},"max":{"1min":2.9569999999999999,"5min":2.9569999999999999,"15min":2.9569999999999999},"last":1.7090000000000001}]},{"osd":9,"last update":"Sun Nov 5 08:33:52 2023","interfaces":[{"interface":"back","average":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":1.7849999999999999},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":4.6520000000000001,"5min":4.6520000000000001,"15min":4.6520000000000001},"last":1.8009999999999999},{"interface":"front","average":{"1min":2.0720000000000001,"5min":2.0720000000000001,"15min":2.0720000000000001},"min":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"max":{"1min":4.7450000000000001,"5min":4.7450000000000001,"15min":4.7450000000000001},"last":1.9299999999999999}]},{"osd":11,"last update":"Sun Nov 5 08:33:48 2023","interfaces":[{"interface":"back","average":{"1min":1.6619999999999999,"5min":1.6619999999999999,"15min":1.6619999999999999},"min":{"1min":1.2,"5min":1.2,"15min":1.2},"max":{"1min":2.5129999999999999,"5min":2.5129999999999999,"15min":2.5129999999999999},"last":2.1259999999999999},{"interface":"front","average":{"1min":1.7909999999999999,"5min":1.7909999999999999,"15min":1.7909999999999999},"min":{"1min":1.056,"5min":1.056,"15min":1.056},"max":{"1min":3.6619999999999999,"5min":3.6619999999999999,"15min":3.6619999999999999},"last":2.1640000000000001}]},{"osd":12,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.962,"5min":1.962,"15min":1.962},"min":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"max":{"1min":5.7270000000000003,"5min":5.7270000000000003,"15min":5.7270000000000003},"last":2.0950000000000002},{"interface":"front","average":{"1min":1.9239999999999999,"5min":1.9239999999999999,"15min":1.9239999999999999},"min":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"max":{"1min":6.093,"5min":6.093,"15min":6.093},"last":2.2410000000000001}]},{"osd":14,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.9470000000000001,"5min":1.9470000000000001,"15min":1.9470000000000001},"min":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"max":{"1min":3.7050000000000001,"5min":3.7050000000000001,"15min":3.7050000000000001},"last":2.0529999999999999},{"interface":"front","average":{"1min":1.982,"5min":1.982,"15min":1.982},"min":{"1min":1.099,"5min":1.099,"15min":1.099},"max":{"1min":3.2349999999999999,"5min":3.2349999999999999,"15min":3.2349999999999999},"last":2.2109999999999999}]},{"osd":15,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.9370000000000001,"5min":1.9370000000000001,"15min":1.9370000000000001},"min":{"1min":1.155,"5min":1.155,"15min":1.155},"max":{"1min":3.048,"5min":3.048,"15min":3.048},"last":2.3050000000000002},{"interface":"front","average":{"1min":2.129,"5min":2.129,"15min":2.129},"min":{"1min":1.135,"5min":1.135,"15min":1.135},"max":{"1min":5.3920000000000003,"5min":5.3920000000000003,"15min":5.3920000000000003},"last":1.964}]}]},{"osd":15,"up_from":21,"seq":90194313239,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38628,"kb_used_data":11600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94333212,"statfs":{"total":96636764160,"available":96597209088,"internally_reserved":0,"allocated":11878400,"data_stored":17419318,"data_compressed":175274,"data_compressed_allocated":376832,"data_compressed_original":753664,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,3,4,6,7,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":1,"last update":"Sun Nov 5 08:33:55 2023","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.139,"15min":1.139},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":3.0329999999999999,"5min":3.0329999999999999,"15min":3.0329999999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":8.6170000000000009,"5min":8.6170000000000009,"15min":8.6170000000000009},"last":0.88400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.131}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1739999999999999}]},{"osd":4,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.125,"15min":1.125},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":2.4649999999999999,"5min":2.4649999999999999,"15min":2.4649999999999999},"last":0.85299999999999998},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":2.331,"5min":2.331,"15min":2.331},"last":1.2549999999999999}]},{"osd":5,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.113,"15min":1.113},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":2.9460000000000002,"5min":2.9460000000000002,"15min":2.9460000000000002},"last":0.48399999999999999},{"interface":"front","average":{"1min":1.452,"5min":1.452,"15min":1.452},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":3.8849999999999998,"5min":3.8849999999999998,"15min":3.8849999999999998},"last":1.042}]},{"osd":6,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.119,"15min":1.119},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":2.4769999999999999,"5min":2.4769999999999999,"15min":2.4769999999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":3.0880000000000001,"5min":3.0880000000000001,"15min":3.0880000000000001},"last":0.97899999999999998}]},{"osd":7,"last update":"Sun Nov 5 08:33:44 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.169,"15min":1.169},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.996,"5min":2.996,"15min":2.996},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.389,"5min":1.389,"15min":1.389},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":5.444,"5min":5.444,"15min":5.444},"last":1.056}]},{"osd":8,"last update":"Sun Nov 5 08:33:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":3.7389999999999999,"5min":3.7389999999999999,"15min":3.7389999999999999},"last":1.0960000000000001},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":4.2489999999999997,"5min":4.2489999999999997,"15min":4.2489999999999997},"last":1.3160000000000001}]},{"osd":9,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.5900000000000001,"5min":1.5900000000000001,"15min":1.5900000000000001},"min":{"1min":1.044,"5min":1.044,"15min":1.044},"max":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.883,"5min":1.883,"15min":1.883},"min":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"max":{"1min":13.029999999999999,"5min":13.029999999999999,"15min":13.029999999999999},"last":1.375}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.286}]},{"osd":11,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"min":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":2.9590000000000001,"5min":2.9590000000000001,"15min":2.9590000000000001},"last":1.2},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"min":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"max":{"1min":2.6469999999999998,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":1.4830000000000001}]},{"osd":13,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.591,"5min":1.591,"15min":1.591},"min":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"max":{"1min":3.3580000000000001,"5min":3.3580000000000001,"15min":3.3580000000000001},"last":1.351},{"interface":"front","average":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":1.7849999999999999},"min":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"max":{"1min":5.1379999999999999,"5min":5.1379999999999999,"15min":5.1379999999999999},"last":1.5269999999999999}]},{"osd":14,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.718,"5min":1.718,"15min":1.718},"min":{"1min":0.999,"5min":0.999,"15min":0.999},"max":{"1min":5.7720000000000002,"5min":5.7720000000000002,"15min":5.7720000000000002},"last":1.575},{"interface":"front","average":{"1min":1.696,"5min":1.696,"15min":1.696},"min":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"max":{"1min":7.5039999999999996,"5min":7.5039999999999996,"15min":7.5039999999999996},"last":1.1419999999999999}]}]},{"osd":14,"up_from":22,"seq":94489280534,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40108,"kb_used_data":13084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331732,"statfs":{"total":96636764160,"available":96595693568,"internally_reserved":0,"allocated":13398016,"data_stored":25621457,"data_compressed":173314,"data_compressed_allocated":372736,"data_compressed_original":745472,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":1,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.77200000000000002},{"interface":"front","average":{"1min":1.131,"5min":1.131,"15min":1.131},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.732,"5min":1.732,"15min":1.732},"last":0.996}]},{"osd":2,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.839,"5min":1.839,"15min":1.839},"last":1.0609999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4650000000000001}]},{"osd":4,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":1.137},{"interface":"front","average":{"1min":0.997,"5min":0.997,"15min":0.997},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.391,"5min":1.391,"15min":1.391},"last":1.429}]},{"osd":5,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":1.5229999999999999},{"interface":"front","average":{"1min":1.101,"5min":1.101,"15min":1.101},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.605,"5min":1.605,"15min":1.605},"last":0.88900000000000001}]},{"osd":6,"last update":"Sun Nov 5 08:33:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.575,"5min":1.575,"15min":1.575},"last":1.181},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.9610000000000001,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":1.3919999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.571}]},{"osd":8,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.7050000000000001,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":1.6719999999999999},{"interface":"front","average":{"1min":1.22,"5min":1.22,"15min":1.22},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":1.647}]},{"osd":10,"last update":"Sun Nov 5 08:33:44 2023","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.29,"15min":1.29},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.802,"5min":1.802,"15min":1.802},"last":1.7410000000000001},{"interface":"front","average":{"1min":1.325,"5min":1.325,"15min":1.325},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.8049999999999999,"5min":1.8049999999999999,"15min":1.8049999999999999},"last":1.5469999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.429}]},{"osd":12,"last update":"Sun Nov 5 08:33:53 2023","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":1.4430000000000001},"min":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"max":{"1min":2.2389999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.6180000000000001},{"interface":"front","average":{"1min":1.544,"5min":1.544,"15min":1.544},"min":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"max":{"1min":2.1320000000000001,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":1.72}]},{"osd":13,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.5900000000000001,"5min":1.5900000000000001,"15min":1.5900000000000001},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":2.4590000000000001,"5min":2.4590000000000001,"15min":2.4590000000000001},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"min":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":2.028,"5min":2.028,"15min":2.028},"last":1.4970000000000001}]},{"osd":15,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":2.2429999999999999,"5min":2.2429999999999999,"15min":2.2429999999999999},"last":1.794},{"interface":"front","average":{"1min":1.6399999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":2.3559999999999999,"5min":2.3559999999999999,"15min":2.3559999999999999},"last":1.6970000000000001}]}]},{"osd":0,"up_from":21,"seq":90194313223,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":4392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":4497408,"data_stored":4339168,"data_compressed":21230,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,9,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]}]},{"osd":3,"up_from":93,"seq":399431958534,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44004,"kb_used_data":15832,"kb_used_omap":6,"kb_used_meta":28153,"kb_avail":94327836,"statfs":{"total":96636764160,"available":96591704064,"internally_reserved":0,"allocated":16211968,"data_stored":18591182,"data_compressed":181130,"data_compressed_allocated":389120,"data_compressed_original":778240,"omap_allocated":6448,"internal_metadata":28829392},"hb_peers":[2,4,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0800000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88900000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96799999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91800000000000004}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.198}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.131}]}]},{"osd":6,"up_from":21,"seq":90194313238,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48368,"kb_used_data":17184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323472,"statfs":{"total":96636764160,"available":96587235328,"internally_reserved":0,"allocated":17596416,"data_stored":23953799,"data_compressed":173314,"data_compressed_allocated":372736,"data_compressed_original":745472,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,3,4,5,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.02,"15min":1.02},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.153,"5min":1.153,"15min":1.153},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.899,"5min":1.899,"15min":1.899},"last":0.72499999999999998}]},{"osd":2,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.77100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4370000000000001}]},{"osd":4,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.038,"15min":1.038},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":1.268},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.617,"5min":1.617,"15min":1.617},"last":1.5369999999999999}]},{"osd":5,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.532,"5min":1.532,"15min":1.532},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.212,"5min":1.212,"15min":1.212},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.577,"5min":1.577,"15min":1.577},"last":1.1220000000000001}]},{"osd":7,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.72,"5min":1.72,"15min":1.72},"last":0.95099999999999996},{"interface":"front","average":{"1min":1.069,"5min":1.069,"15min":1.069},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.6859999999999999,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":1.3}]},{"osd":8,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.958,"5min":1.958,"15min":1.958},"last":1.2230000000000001},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.772,"5min":1.772,"15min":1.772},"last":1.641}]},{"osd":9,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.831},{"interface":"front","average":{"1min":1.306,"5min":1.306,"15min":1.306},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.7749999999999999,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":1.468}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3999999999999999}]},{"osd":11,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.379,"15min":1.379},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":2.0579999999999998,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":1.3879999999999999},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":2.1930000000000001,"5min":2.1930000000000001,"15min":2.1930000000000001},"last":1.718}]},{"osd":12,"last update":"Sun Nov 5 08:33:31 2023","interfaces":[{"interface":"back","average":{"1min":1.548,"5min":1.548,"15min":1.548},"min":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":2.1139999999999999,"5min":2.1139999999999999,"15min":2.1139999999999999},"last":1.796},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.841,"5min":1.841,"15min":1.841},"last":1.6020000000000001}]},{"osd":13,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":1.6930000000000001,"5min":1.6930000000000001,"15min":1.6930000000000001},"min":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"max":{"1min":2.2759999999999998,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":1.8660000000000001},{"interface":"front","average":{"1min":1.532,"5min":1.532,"15min":1.532},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.9710000000000001,"5min":1.9710000000000001,"15min":1.9710000000000001},"last":1.7490000000000001}]},{"osd":14,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":2.319,"5min":2.319,"15min":2.319},"last":1.5780000000000001},{"interface":"front","average":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":2.4870000000000001,"5min":2.4870000000000001,"15min":2.4870000000000001},"last":1.964}]},{"osd":15,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.679,"5min":1.679,"15min":1.679},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.4009999999999998,"5min":2.4009999999999998,"15min":2.4009999999999998},"last":1.9179999999999999},{"interface":"front","average":{"1min":1.482,"5min":1.482,"15min":1.482},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"last":1.679}]}]},{"osd":1,"up_from":22,"seq":94489280534,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46180,"kb_used_data":14992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325660,"statfs":{"total":96636764160,"available":96589475840,"internally_reserved":0,"allocated":15351808,"data_stored":21028480,"data_compressed":175274,"data_compressed_allocated":376832,"data_compressed_original":753664,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,5,6,7,8,9,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":2,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.272,"5min":1.272,"15min":1.272},"last":0.58199999999999996},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.1240000000000001,"5min":2.1240000000000001,"15min":2.1240000000000001},"last":1.1160000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.873}]},{"osd":4,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.6100000000000001,"5min":1.6100000000000001,"15min":1.6100000000000001},"last":1.117}]},{"osd":5,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.071,"15min":1.071},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":2.6110000000000002,"5min":2.6110000000000002,"15min":2.6110000000000002},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.7310000000000001,"5min":1.7310000000000001,"15min":1.7310000000000001},"last":0.68300000000000005}]},{"osd":6,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.135,"15min":1.135},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.73299999999999998},{"interface":"front","average":{"1min":1.246,"5min":1.246,"15min":1.246},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.0029999999999999}]},{"osd":7,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.282,"15min":1.282},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.708,"5min":1.708,"15min":1.708},"last":0.629},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":2.3090000000000002,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":0.90900000000000003}]},{"osd":8,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.288,"15min":1.288},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":2.5630000000000002,"5min":2.5630000000000002,"15min":2.5630000000000002},"last":1.0649999999999999},{"interface":"front","average":{"1min":1.238,"5min":1.238,"15min":1.238},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":0.76300000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3879999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.268}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.252}]},{"osd":14,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"min":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"max":{"1min":2.2320000000000002,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.498,"5min":1.498,"15min":1.498},"min":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"max":{"1min":2.7610000000000001,"5min":2.7610000000000001,"15min":2.7610000000000001},"last":1.4319999999999999}]},{"osd":15,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.5640000000000001,"5min":1.5640000000000001,"15min":1.5640000000000001},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":2.7250000000000001,"5min":2.7250000000000001,"15min":2.7250000000000001},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"min":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":1.504}]}]},{"osd":2,"up_from":22,"seq":94489280534,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43512,"kb_used_data":12324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328328,"statfs":{"total":96636764160,"available":96592207872,"internally_reserved":0,"allocated":12619776,"data_stored":22220960,"data_compressed":182223,"data_compressed_allocated":409600,"data_compressed_original":1265664,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,3,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.5529999999999999,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.482,"5min":1.482,"15min":1.482},"last":1.119}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94199999999999995}]},{"osd":5,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.5029999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":1.02}]},{"osd":6,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.84399999999999997},{"interface":"front","average":{"1min":1.077,"5min":1.077,"15min":1.077},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":1.3100000000000001}]},{"osd":7,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.167,"15min":1.167},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":2.0880000000000001,"5min":2.0880000000000001,"15min":2.0880000000000001},"last":1.0720000000000001},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.744,"5min":1.744,"15min":1.744},"last":0.97199999999999998}]},{"osd":8,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.831,"5min":1.831,"15min":1.831},"last":1.264},{"interface":"front","average":{"1min":1.129,"5min":1.129,"15min":1.129},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":2.1859999999999999,"5min":2.1859999999999999,"15min":2.1859999999999999},"last":1.1499999999999999}]},{"osd":9,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.7969999999999999,"5min":1.7969999999999999,"15min":1.7969999999999999},"last":1.4890000000000001},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":2.27,"5min":2.27,"15min":2.27},"last":1.5569999999999999}]},{"osd":10,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":2.976,"5min":2.976,"15min":2.976},"last":1.4530000000000001},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":2.5739999999999998,"5min":2.5739999999999998,"15min":2.5739999999999998},"last":1.6000000000000001}]},{"osd":11,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.9009999999999998,"5min":2.9009999999999998,"15min":2.9009999999999998},"last":1.417},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":2.6859999999999999,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":1.5209999999999999}]},{"osd":12,"last update":"Sun Nov 5 08:33:45 2023","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.2,"15min":1.2},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.6850000000000001}]}]},{"osd":4,"up_from":22,"seq":94489280534,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44188,"kb_used_data":13000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327652,"statfs":{"total":96636764160,"available":96591515648,"internally_reserved":0,"allocated":13312000,"data_stored":18082870,"data_compressed":175274,"data_compressed_allocated":376832,"data_compressed_original":753664,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,5,6,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":1,"last update":"Sun Nov 5 08:33:43 2023","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.6659999999999999,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":0.97799999999999998}]},{"osd":2,"last update":"Sun Nov 5 08:33:43 2023","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":1.1499999999999999},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.9850000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":1.2989999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.875}]},{"osd":5,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.0299999999999998,"5min":2.0299999999999998,"15min":2.0299999999999998},"last":1.496},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.7,"5min":1.7,"15min":1.7},"last":1.0309999999999999}]},{"osd":6,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":0.995,"15min":0.995},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.7290000000000001,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":1.0649999999999999},{"interface":"front","average":{"1min":1.238,"5min":1.238,"15min":1.238},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":2.2789999999999999,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":1.6719999999999999}]},{"osd":7,"last update":"Sun Nov 5 08:33:43 2023","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.889,"5min":1.889,"15min":1.889},"last":1.181},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":1.1299999999999999}]},{"osd":8,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.171,"15min":1.171},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":2.2490000000000001,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.7789999999999999,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":1.4390000000000001}]},{"osd":9,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":2.5249999999999999,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":1.7949999999999999},{"interface":"front","average":{"1min":1.468,"5min":1.468,"15min":1.468},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.3290000000000002,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":1.3280000000000001}]},{"osd":10,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":2.0459999999999998,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.593},{"interface":"front","average":{"1min":1.508,"5min":1.508,"15min":1.508},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.3919999999999999,"5min":2.3919999999999999,"15min":2.3919999999999999},"last":1.619}]},{"osd":11,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.431,"15min":1.431},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.1280000000000001,"5min":2.1280000000000001,"15min":2.1280000000000001},"last":1.353},{"interface":"front","average":{"1min":1.3,"5min":1.3,"15min":1.3},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.9379999999999999,"5min":1.9379999999999999,"15min":1.9379999999999999},"last":1.238}]},{"osd":12,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":1.542},{"interface":"front","average":{"1min":1.3,"5min":1.3,"15min":1.3},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":2.161,"5min":2.161,"15min":2.161},"last":1.696}]},{"osd":13,"last update":"Sun Nov 5 08:33:29 2023","interfaces":[{"interface":"back","average":{"1min":1.5640000000000001,"5min":1.5640000000000001,"15min":1.5640000000000001},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":2.3420000000000001,"5min":2.3420000000000001,"15min":2.3420000000000001},"last":1.726},{"interface":"front","average":{"1min":1.538,"5min":1.538,"15min":1.538},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":2.3140000000000001,"5min":2.3140000000000001,"15min":2.3140000000000001},"last":1.645}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5109999999999999}]},{"osd":15,"last update":"Sun Nov 5 08:33:57 2023","interfaces":[{"interface":"back","average":{"1min":1.6990000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":2.5750000000000002,"5min":2.5750000000000002,"15min":2.5750000000000002},"last":1.754},{"interface":"front","average":{"1min":1.6719999999999999,"5min":1.6719999999999999,"15min":1.6719999999999999},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":2.3940000000000001,"5min":2.3940000000000001,"15min":2.3940000000000001},"last":1.8500000000000001}]}]},{"osd":5,"up_from":22,"seq":94489280534,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34672,"kb_used_data":7644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337168,"statfs":{"total":96636764160,"available":96601260032,"internally_reserved":0,"allocated":7827456,"data_stored":8547968,"data_compressed":175274,"data_compressed_allocated":376832,"data_compressed_original":753664,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,6,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":1,"last update":"Sun Nov 5 08:33:45 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":1.2989999999999999},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.46800000000000003}]},{"osd":2,"last update":"Sun Nov 5 08:33:45 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.575,"5min":1.575,"15min":1.575},"last":1.0720000000000001},{"interface":"front","average":{"1min":1.04,"5min":1.04,"15min":1.04},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.546,"5min":1.546,"15min":1.546},"last":1.2350000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":4,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.645,"5min":1.645,"15min":1.645},"last":1.167},{"interface":"front","average":{"1min":1.036,"5min":1.036,"15min":1.036},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.77,"5min":1.77,"15min":1.77},"last":1.351}]},{"osd":6,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":1.0149999999999999}]},{"osd":7,"last update":"Sun Nov 5 08:33:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.696,"5min":1.696,"15min":1.696},"last":1.109},{"interface":"front","average":{"1min":1.177,"5min":1.177,"15min":1.177},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":1.2629999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5760000000000001}]},{"osd":9,"last update":"Sun Nov 5 08:33:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.3,"15min":1.3},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.6990000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"min":{"1min":0.88,"5min":0.88,"15min":0.88},"max":{"1min":1.8180000000000001,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":0.71799999999999997}]},{"osd":10,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.48},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":1.052}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4990000000000001}]},{"osd":12,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":1.635,"15min":1.635},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":3.488,"5min":3.488,"15min":3.488},"last":1.212},{"interface":"front","average":{"1min":1.46,"5min":1.46,"15min":1.46},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.1970000000000001,"5min":2.1970000000000001,"15min":2.1970000000000001},"last":1.1890000000000001}]},{"osd":13,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.6799999999999999,"5min":1.6799999999999999,"15min":1.6799999999999999},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":3.714,"5min":3.714,"15min":3.714},"last":1.4490000000000001},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.5649999999999999,"15min":1.5649999999999999},"min":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"max":{"1min":2.121,"5min":2.121,"15min":2.121},"last":0.75700000000000001}]},{"osd":14,"last update":"Sun Nov 5 08:33:45 2023","interfaces":[{"interface":"back","average":{"1min":1.718,"5min":1.718,"15min":1.718},"min":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"max":{"1min":2.1920000000000002,"5min":2.1920000000000002,"15min":2.1920000000000002},"last":1.3200000000000001},{"interface":"front","average":{"1min":1.647,"5min":1.647,"15min":1.647},"min":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"max":{"1min":2.2930000000000001,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":1.4199999999999999}]},{"osd":15,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.974,"5min":1.974,"15min":1.974},"last":1.377},{"interface":"front","average":{"1min":1.8140000000000001,"5min":1.8140000000000001,"15min":1.8140000000000001},"min":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"max":{"1min":3.5870000000000002,"5min":3.5870000000000002,"15min":3.5870000000000002},"last":1.3939999999999999}]}]},{"osd":7,"up_from":22,"seq":94489280535,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43328,"kb_used_data":12140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328512,"statfs":{"total":96636764160,"available":96592396288,"internally_reserved":0,"allocated":12431360,"data_stored":17198720,"data_compressed":175274,"data_compressed_allocated":376832,"data_compressed_original":753664,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[1,2,3,4,5,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1930000000000001}]},{"osd":1,"last update":"Sun Nov 5 08:33:47 2023","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":2.1360000000000001,"5min":2.1360000000000001,"15min":2.1360000000000001},"last":1.0309999999999999},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":1.2370000000000001}]},{"osd":2,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.401,"15min":1.401},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":1.889,"5min":1.889,"15min":1.889},"last":1.4219999999999999},{"interface":"front","average":{"1min":1.238,"5min":1.238,"15min":1.238},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.663,"5min":1.663,"15min":1.663},"last":1.5700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97199999999999998}]},{"osd":4,"last update":"Sun Nov 5 08:33:30 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.095,"15min":1.095},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.7949999999999999,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":1.1870000000000001},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":2.294,"5min":2.294,"15min":2.294},"last":1.5209999999999999}]},{"osd":5,"last update":"Sun Nov 5 08:33:30 2023","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.591,"5min":1.591,"15min":1.591},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.726,"5min":1.726,"15min":1.726},"last":1.71}]},{"osd":6,"last update":"Sun Nov 5 08:33:30 2023","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.232,"15min":1.232},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.6040000000000001,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":1.6339999999999999},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":2.0510000000000002,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.744}]},{"osd":8,"last update":"Sun Nov 5 08:33:30 2023","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.373},{"interface":"front","average":{"1min":1.381,"5min":1.381,"15min":1.381},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.48,"5min":2.48,"15min":2.48},"last":1.786}]},{"osd":9,"last update":"Sun Nov 5 08:33:47 2023","interfaces":[{"interface":"back","average":{"1min":1.6799999999999999,"5min":1.6799999999999999,"15min":1.6799999999999999},"min":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"max":{"1min":2.823,"5min":2.823,"15min":2.823},"last":2.0819999999999999},{"interface":"front","average":{"1min":1.48,"5min":1.48,"15min":1.48},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":2.3450000000000002,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":2.0299999999999998}]},{"osd":10,"last update":"Sun Nov 5 08:33:30 2023","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":1.478,"15min":1.478},"min":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"max":{"1min":1.931,"5min":1.931,"15min":1.931},"last":1.8140000000000001},{"interface":"front","average":{"1min":1.671,"5min":1.671,"15min":1.671},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.5550000000000002,"5min":2.5550000000000002,"15min":2.5550000000000002},"last":1.913}]},{"osd":11,"last update":"Sun Nov 5 08:33:47 2023","interfaces":[{"interface":"back","average":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":2.093,"5min":2.093,"15min":2.093},"last":1.889},{"interface":"front","average":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":2.6600000000000001,"5min":2.6600000000000001,"15min":2.6600000000000001},"last":1.9610000000000001}]},{"osd":12,"last update":"Sun Nov 5 08:33:52 2023","interfaces":[{"interface":"back","average":{"1min":1.841,"5min":1.841,"15min":1.841},"min":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"max":{"1min":2.7250000000000001,"5min":2.7250000000000001,"15min":2.7250000000000001},"last":1.8580000000000001},{"interface":"front","average":{"1min":1.639,"5min":1.639,"15min":1.639},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.3780000000000001,"5min":2.3780000000000001,"15min":2.3780000000000001},"last":1.9379999999999999}]},{"osd":13,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":1.667,"5min":1.667,"15min":1.667},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.589,"5min":2.589,"15min":2.589},"last":2.2080000000000002},{"interface":"front","average":{"1min":1.625,"5min":1.625,"15min":1.625},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":2.3050000000000002,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":2.0539999999999998}]},{"osd":14,"last update":"Sun Nov 5 08:33:30 2023","interfaces":[{"interface":"back","average":{"1min":1.75,"5min":1.75,"15min":1.75},"min":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":2.4910000000000001,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":2.1280000000000001},{"interface":"front","average":{"1min":1.7529999999999999,"5min":1.7529999999999999,"15min":1.7529999999999999},"min":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"max":{"1min":2.9750000000000001,"5min":2.9750000000000001,"15min":2.9750000000000001},"last":2.1070000000000002}]},{"osd":15,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":1.7010000000000001,"5min":1.7010000000000001,"15min":1.7010000000000001},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":3.1760000000000002,"5min":3.1760000000000002,"15min":3.1760000000000002},"last":1.984},{"interface":"front","average":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":2.73,"5min":2.73,"15min":2.73},"last":2.0070000000000001}]}]},{"osd":8,"up_from":22,"seq":94489280534,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50652,"kb_used_data":19464,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321188,"statfs":{"total":96636764160,"available":96584896512,"internally_reserved":0,"allocated":19931136,"data_stored":34610816,"data_compressed":175274,"data_compressed_allocated":376832,"data_compressed_original":753664,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":1,"last update":"Sun Nov 5 08:33:46 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.65900000000000003}]},{"osd":2,"last update":"Sun Nov 5 08:33:46 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":1.077},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":0.83899999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97399999999999998}]},{"osd":4,"last update":"Sun Nov 5 08:33:46 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.54600000000000004}]},{"osd":5,"last update":"Sun Nov 5 08:33:33 2023","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.786,"5min":1.786,"15min":1.786},"last":1.484},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":0.71299999999999997}]},{"osd":6,"last update":"Sun Nov 5 08:33:33 2023","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.6799999999999999,"5min":1.6799999999999999,"15min":1.6799999999999999},"last":1.173},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":1.125}]},{"osd":7,"last update":"Sun Nov 5 08:33:33 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.556,"5min":1.556,"15min":1.556},"last":0.81100000000000005},{"interface":"front","average":{"1min":1.214,"5min":1.214,"15min":1.214},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":2.157,"5min":2.157,"15min":2.157},"last":1.395}]},{"osd":9,"last update":"Sun Nov 5 08:33:33 2023","interfaces":[{"interface":"back","average":{"1min":2.1509999999999998,"5min":2.1509999999999998,"15min":2.1509999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":19.849,"5min":19.849,"15min":19.849},"last":1.226},{"interface":"front","average":{"1min":2.2170000000000001,"5min":2.2170000000000001,"15min":2.2170000000000001},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":19.867999999999999,"5min":19.867999999999999,"15min":19.867999999999999},"last":1.0329999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]},{"osd":11,"last update":"Sun Nov 5 08:33:33 2023","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.298,"15min":1.298},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":2.0710000000000002,"5min":2.0710000000000002,"15min":2.0710000000000002},"last":1.302},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":1.446}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":13,"last update":"Sun Nov 5 08:33:33 2023","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":0.89900000000000002},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.8560000000000001,"5min":1.8560000000000001,"15min":1.8560000000000001},"last":1.3540000000000001}]},{"osd":14,"last update":"Sun Nov 5 08:33:33 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":2.226,"5min":2.226,"15min":2.226},"last":0.94599999999999995},{"interface":"front","average":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":1.4430000000000001},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.891,"5min":1.891,"15min":1.891},"last":1.6399999999999999}]},{"osd":15,"last update":"Sun Nov 5 08:33:55 2023","interfaces":[{"interface":"back","average":{"1min":1.552,"5min":1.552,"15min":1.552},"min":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"max":{"1min":2.1629999999999998,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":1.5780000000000001},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.4490000000000001,"15min":1.4490000000000001},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.1970000000000001,"5min":2.1970000000000001,"15min":2.1970000000000001},"last":1.532}]}]},{"osd":9,"up_from":21,"seq":90194313238,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42044,"kb_used_data":10864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329796,"statfs":{"total":96636764160,"available":96593711104,"internally_reserved":0,"allocated":11124736,"data_stored":12873344,"data_compressed":175274,"data_compressed_allocated":376832,"data_compressed_original":753664,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,6,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":1,"last update":"Sun Nov 5 08:34:00 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.506,"5min":1.506,"15min":1.506},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.734,"5min":1.734,"15min":1.734},"last":0.88900000000000001}]},{"osd":2,"last update":"Sun Nov 5 08:34:00 2023","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.5620000000000001,"5min":1.5620000000000001,"15min":1.5620000000000001},"last":1.337}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":4,"last update":"Sun Nov 5 08:34:00 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.095,"15min":1.095},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.478,"5min":1.478,"15min":1.478},"last":1.6479999999999999},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.595,"5min":1.595,"15min":1.595},"last":1.0580000000000001}]},{"osd":6,"last update":"Sun Nov 5 08:33:32 2023","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.101,"15min":1.101},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":1.641,"5min":1.641,"15min":1.641},"last":1.1990000000000001},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.486,"5min":1.486,"15min":1.486},"last":1.425}]},{"osd":7,"last update":"Sun Nov 5 08:34:00 2023","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"min":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"max":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.369,"5min":1.369,"15min":1.369},"min":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":1.925,"5min":1.925,"15min":1.925},"last":1.2749999999999999}]},{"osd":8,"last update":"Sun Nov 5 08:33:32 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.202,"15min":1.202},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.944,"5min":1.944,"15min":1.944},"last":1.099}]},{"osd":10,"last update":"Sun Nov 5 08:33:32 2023","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.752,"5min":1.752,"15min":1.752},"last":1.7210000000000001},{"interface":"front","average":{"1min":1.319,"5min":1.319,"15min":1.319},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":2.1240000000000001,"5min":2.1240000000000001,"15min":2.1240000000000001},"last":1.5640000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2669999999999999}]},{"osd":12,"last update":"Sun Nov 5 08:33:32 2023","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.381,"15min":1.381},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":2.0710000000000002,"5min":2.0710000000000002,"15min":2.0710000000000002},"last":1.4750000000000001},{"interface":"front","average":{"1min":1.462,"5min":1.462,"15min":1.462},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":2.0350000000000001,"5min":2.0350000000000001,"15min":2.0350000000000001},"last":1.7709999999999999}]},{"osd":13,"last update":"Sun Nov 5 08:33:32 2023","interfaces":[{"interface":"back","average":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"min":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"max":{"1min":2.8010000000000002,"5min":2.8010000000000002,"15min":2.8010000000000002},"last":1.794},{"interface":"front","average":{"1min":1.478,"5min":1.478,"15min":1.478},"min":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"max":{"1min":2.177,"5min":2.177,"15min":2.177},"last":1.6000000000000001}]},{"osd":14,"last update":"Sun Nov 5 08:33:32 2023","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.506,"15min":1.506},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.849,"5min":1.849,"15min":1.849},"last":1.8320000000000001},{"interface":"front","average":{"1min":1.6699999999999999,"5min":1.6699999999999999,"15min":1.6699999999999999},"min":{"1min":1.294,"5min":1.294,"15min":1.294},"max":{"1min":2.9710000000000001,"5min":2.9710000000000001,"15min":2.9710000000000001},"last":1.6890000000000001}]},{"osd":15,"last update":"Sun Nov 5 08:33:32 2023","interfaces":[{"interface":"back","average":{"1min":1.603,"5min":1.603,"15min":1.603},"min":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"max":{"1min":1.9670000000000001,"5min":1.9670000000000001,"15min":1.9670000000000001},"last":1.883},{"interface":"front","average":{"1min":1.716,"5min":1.716,"15min":1.716},"min":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"max":{"1min":3.0950000000000002,"5min":3.0950000000000002,"15min":3.0950000000000002},"last":1.7390000000000001}]}]},{"osd":10,"up_from":22,"seq":94489280534,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42240,"kb_used_data":15212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329600,"statfs":{"total":96636764160,"available":96593510400,"internally_reserved":0,"allocated":15577088,"data_stored":26795648,"data_compressed":175274,"data_compressed_allocated":376832,"data_compressed_original":753664,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,4,5,6,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":1,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.749},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.86699999999999999}]},{"osd":2,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.768,"5min":1.768,"15min":1.768},"last":1.258},{"interface":"front","average":{"1min":1.01,"5min":1.01,"15min":1.01},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":2.9729999999999999,"5min":2.9729999999999999,"15min":2.9729999999999999},"last":0.81299999999999994}]},{"osd":4,"last update":"Sun Nov 5 08:33:57 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":1.0569999999999999},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.482,"5min":1.482,"15min":1.482},"last":1.153}]},{"osd":5,"last update":"Sun Nov 5 08:33:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":3.4790000000000001,"5min":3.4790000000000001,"15min":3.4790000000000001},"last":0.97899999999999998},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":3.5339999999999998,"5min":3.5339999999999998,"15min":3.5339999999999998},"last":1.016}]},{"osd":6,"last update":"Sun Nov 5 08:33:57 2023","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.55,"5min":1.55,"15min":1.55},"last":1.1970000000000001},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.5469999999999999,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":1.0900000000000001}]},{"osd":8,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.105,"15min":1.105},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.661,"5min":1.661,"15min":1.661},"last":1.331},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.673,"5min":1.673,"15min":1.673},"last":1.3600000000000001}]},{"osd":9,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.264,"15min":1.264},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":2.8199999999999998,"5min":2.8199999999999998,"15min":2.8199999999999998},"last":1.4770000000000001},{"interface":"front","average":{"1min":1.319,"5min":1.319,"15min":1.319},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":1.827,"5min":1.827,"15min":1.827},"last":1.3879999999999999}]},{"osd":11,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":1.877,"5min":1.877,"15min":1.877},"last":1.5089999999999999},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":2.8889999999999998,"5min":2.8889999999999998,"15min":2.8889999999999998},"last":1.4059999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":13,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.536,"5min":1.536,"15min":1.536},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":2.778,"5min":2.778,"15min":2.778},"last":1.4490000000000001},{"interface":"front","average":{"1min":1.6200000000000001,"5min":1.6200000000000001,"15min":1.6200000000000001},"min":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"max":{"1min":3.681,"5min":3.681,"15min":3.681},"last":1.2949999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]},{"osd":15,"last update":"Sun Nov 5 08:33:27 2023","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":2.0550000000000002,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":1.429},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.5289999999999999,"15min":1.5289999999999999},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.4620000000000002,"5min":2.4620000000000002,"15min":2.4620000000000002},"last":1.5569999999999999}]}]},{"osd":11,"up_from":21,"seq":90194313239,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46508,"kb_used_data":15328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325332,"statfs":{"total":96636764160,"available":96589139968,"internally_reserved":0,"allocated":15695872,"data_stored":22863488,"data_compressed":175274,"data_compressed_allocated":376832,"data_compressed_original":753664,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,4,5,6,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":1,"last update":"Sun Nov 5 08:33:53 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.044,"15min":1.044},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.80400000000000005},{"interface":"front","average":{"1min":1.069,"5min":1.069,"15min":1.069},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":1.4610000000000001}]},{"osd":2,"last update":"Sun Nov 5 08:33:48 2023","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.216,"15min":1.216},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":2.298,"5min":2.298,"15min":2.298},"last":0.85799999999999998},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":2.9020000000000001,"5min":2.9020000000000001,"15min":2.9020000000000001},"last":0.90000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":4,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.167,"15min":1.167},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.7390000000000001,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":1.1220000000000001},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.94799999999999995}]},{"osd":5,"last update":"Sun Nov 5 08:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.244,"15min":1.244},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":3.0459999999999998,"5min":3.0459999999999998,"15min":3.0459999999999998},"last":1.296},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":3.3799999999999999,"5min":3.3799999999999999,"15min":3.3799999999999999},"last":1.2669999999999999}]},{"osd":6,"last update":"Sun Nov 5 08:33:30 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.244,"15min":1.244},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":2.0630000000000002,"5min":2.0630000000000002,"15min":2.0630000000000002},"last":1.018},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":2.7770000000000001,"5min":2.7770000000000001,"15min":2.7770000000000001},"last":1.5449999999999999}]},{"osd":7,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.76,"5min":1.76,"15min":1.76},"last":1.163},{"interface":"front","average":{"1min":1.395,"5min":1.395,"15min":1.395},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.909,"5min":1.909,"15min":1.909},"last":1.3959999999999999}]},{"osd":8,"last update":"Sun Nov 5 08:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.9079999999999999,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":1.2130000000000001},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.948,"5min":1.948,"15min":1.948},"last":1.5780000000000001}]},{"osd":9,"last update":"Sun Nov 5 08:33:58 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":2.089,"5min":2.089,"15min":2.089},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.5660000000000001,"5min":1.5660000000000001,"15min":1.5660000000000001},"min":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"max":{"1min":2.1720000000000002,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":1.679}]},{"osd":10,"last update":"Sun Nov 5 08:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.599,"5min":1.599,"15min":1.599},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.6890000000000001,"5min":2.6890000000000001,"15min":2.6890000000000001},"last":1.657},{"interface":"front","average":{"1min":1.371,"5min":1.371,"15min":1.371},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":2.3679999999999999,"5min":2.3679999999999999,"15min":2.3679999999999999},"last":1.51}]},{"osd":12,"last update":"Sun Nov 5 08:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"max":{"1min":2.423,"5min":2.423,"15min":2.423},"last":1.7749999999999999},{"interface":"front","average":{"1min":1.702,"5min":1.702,"15min":1.702},"min":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"max":{"1min":2.6160000000000001,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":1.7030000000000001}]},{"osd":13,"last update":"Sun Nov 5 08:33:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.4450000000000001,"15min":1.4450000000000001},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":2.48,"5min":2.48,"15min":2.48},"last":1.633},{"interface":"front","average":{"1min":1.714,"5min":1.714,"15min":1.714},"min":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"max":{"1min":2.6219999999999999,"5min":2.6219999999999999,"15min":2.6219999999999999},"last":1.835}]},{"osd":14,"last update":"Sun Nov 5 08:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.8580000000000001,"5min":1.8580000000000001,"15min":1.8580000000000001},"min":{"1min":1.137,"5min":1.137,"15min":1.137},"max":{"1min":5.375,"5min":5.375,"15min":5.375},"last":1.726},{"interface":"front","average":{"1min":1.754,"5min":1.754,"15min":1.754},"min":{"1min":1.038,"5min":1.038,"15min":1.038},"max":{"1min":2.5369999999999999,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.8819999999999999}]},{"osd":15,"last update":"Sun Nov 5 08:33:30 2023","interfaces":[{"interface":"back","average":{"1min":2.125,"5min":2.125,"15min":2.125},"min":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":1.609},{"interface":"front","average":{"1min":1.762,"5min":1.762,"15min":1.762},"min":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"max":{"1min":5.2549999999999999,"5min":5.2549999999999999,"15min":5.2549999999999999},"last":1.8080000000000001}]}]},{"osd":12,"up_from":22,"seq":94489280534,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34812,"kb_used_data":7784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337028,"statfs":{"total":96636764160,"available":96601116672,"internally_reserved":0,"allocated":7970816,"data_stored":9722934,"data_compressed":175274,"data_compressed_allocated":376832,"data_compressed_original":753664,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[2,3,4,5,6,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":1,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.603,"5min":1.603,"15min":1.603},"last":1.2909999999999999}]},{"osd":2,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":1.0309999999999999},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.64400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0980000000000001}]},{"osd":4,"last update":"Sun Nov 5 08:34:02 2023","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.429,"5min":1.429,"15min":1.429},"last":1.0680000000000001}]},{"osd":5,"last update":"Sun Nov 5 08:34:02 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":1.123},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":1.2809999999999999}]},{"osd":6,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.73299999999999998},{"interface":"front","average":{"1min":1.073,"5min":1.073,"15min":1.073},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":1.343}]},{"osd":9,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.208,"15min":1.208},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":2.2109999999999999,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":0.99099999999999999},{"interface":"front","average":{"1min":1.206,"5min":1.206,"15min":1.206},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.964,"5min":1.964,"15min":1.964},"last":0.96399999999999997}]},{"osd":11,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":2.1659999999999999,"5min":2.1659999999999999,"15min":2.1659999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.8680000000000001,"5min":1.8680000000000001,"15min":1.8680000000000001},"last":1.149}]},{"osd":13,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.25,"5min":2.25,"15min":2.25},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":2.399,"5min":2.399,"15min":2.399},"last":1.391}]},{"osd":14,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":2.1070000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.218},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":2.3420000000000001,"5min":2.3420000000000001,"15min":2.3420000000000001},"last":1.2450000000000001}]},{"osd":15,"last update":"Sun Nov 5 08:33:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.2959999999999998,"5min":2.2959999999999998,"15min":2.2959999999999998},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.4099999999999999,"15min":1.4099999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.873,"5min":1.873,"15min":1.873},"last":1.3169999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6949,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6949,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4165632,"data_stored":4165632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14032896,"data_stored":19992576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11051008,"data_stored":19505152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":9543680,"data_stored":10022912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11997184,"data_stored":17047552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":6508544,"data_stored":7512064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16293888,"data_stored":22929408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":11112448,"data_stored":16162816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":18337792,"data_stored":31731712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":9805824,"data_stored":11837440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":14258176,"data_stored":25759744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":14376960,"data_stored":21827584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":6656000,"data_stored":8687616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":17682432,"data_stored":27115520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":11206656,"data_stored":23711744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":10563584,"data_stored":16384000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-05T08:34:11.734 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2023-11-05T08:34:11.735 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2023-11-05T08:34:11.744 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:34:11.740+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:34:11.745 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:34:11.745 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-11-05T08:34:11.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-11-05T08:34:11.745 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-11-05T08:34:11.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.746 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:233 3.ds0 3:bf7e600d:::smithi09039252-23:head [append 0~0] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2023-11-05T08:34:11.746 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.059123+0000", 2023-11-05T08:34:11.746 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 82.683406372999997, 2023-11-05T08:34:11.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019416857999999999, 2023-11-05T08:34:11.746 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:34:11.747 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:11.747 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:34:11.747 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:34:11.747 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:11.747 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 233 2023-11-05T08:34:11.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.748 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:34:11.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:34:11.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059123+0000", 2023-11-05T08:34:11.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:11.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:34:11.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059123+0000", 2023-11-05T08:34:11.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:11.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:34:11.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059126+0000", 2023-11-05T08:34:11.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8370000000000001e-06 2023-11-05T08:34:11.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:34:11.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059140+0000", 2023-11-05T08:34:11.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3669000000000001e-05 2023-11-05T08:34:11.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:34:11.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059176+0000", 2023-11-05T08:34:11.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6548000000000003e-05 2023-11-05T08:34:11.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:34:11.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059198+0000", 2023-11-05T08:34:11.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1489e-05 2023-11-05T08:34:11.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:34:11.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059401+0000", 2023-11-05T08:34:11.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020313300000000001 2023-11-05T08:34:11.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-11-05T08:34:11.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059422+0000", 2023-11-05T08:34:11.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1212e-05 2023-11-05T08:34:11.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:34:11.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.063981+0000", 2023-11-05T08:34:11.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004558604 2023-11-05T08:34:11.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:34:11.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.064128+0000", 2023-11-05T08:34:11.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014734899999999999 2023-11-05T08:34:11.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:34:11.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.066042+0000", 2023-11-05T08:34:11.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019138079999999999 2023-11-05T08:34:11.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.758 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:34:11.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.077186+0000", 2023-11-05T08:34:11.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011144536 2023-11-05T08:34:11.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:34:11.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.078357+0000", 2023-11-05T08:34:11.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011709089999999999 2023-11-05T08:34:11.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:34:11.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.078540+0000", 2023-11-05T08:34:11.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000182764 2023-11-05T08:34:11.760 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:11.761 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:34:11.761 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:11.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.761 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:234 3.ds0 3:bf7e600d:::smithi09039252-23:head [rollback 3] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2023-11-05T08:34:11.762 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.059247+0000", 2023-11-05T08:34:11.762 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 82.683282775999999, 2023-11-05T08:34:11.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019739059, 2023-11-05T08:34:11.762 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:34:11.762 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:11.763 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:34:11.763 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:34:11.763 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:11.763 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 234 2023-11-05T08:34:11.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.764 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:34:11.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:34:11.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059247+0000", 2023-11-05T08:34:11.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:11.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:34:11.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059247+0000", 2023-11-05T08:34:11.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:11.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:34:11.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059248+0000", 2023-11-05T08:34:11.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.077e-06 2023-11-05T08:34:11.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:34:11.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059256+0000", 2023-11-05T08:34:11.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1359999999999997e-06 2023-11-05T08:34:11.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:34:11.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059283+0000", 2023-11-05T08:34:11.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6855e-05 2023-11-05T08:34:11.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:34:11.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059298+0000", 2023-11-05T08:34:11.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5356e-05 2023-11-05T08:34:11.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:34:11.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059582+0000", 2023-11-05T08:34:11.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028341299999999999 2023-11-05T08:34:11.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:34:11.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059592+0000", 2023-11-05T08:34:11.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0103e-05 2023-11-05T08:34:11.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:34:11.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.067711+0000", 2023-11-05T08:34:11.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0081197660000000005 2023-11-05T08:34:11.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:34:11.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.067965+0000", 2023-11-05T08:34:11.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025397000000000001 2023-11-05T08:34:11.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:34:11.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.069296+0000", 2023-11-05T08:34:11.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013308910000000001 2023-11-05T08:34:11.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:34:11.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.077189+0000", 2023-11-05T08:34:11.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0078931079999999994 2023-11-05T08:34:11.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:34:11.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.078829+0000", 2023-11-05T08:34:11.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016394459999999999 2023-11-05T08:34:11.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:34:11.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.078986+0000", 2023-11-05T08:34:11.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015693799999999999 2023-11-05T08:34:11.780 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:11.780 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:34:11.780 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:11.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.781 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:235 3.ds0 3:bf7e600d:::smithi09039252-23:head [append 0~0] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2023-11-05T08:34:11.781 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:32:49.059370+0000", 2023-11-05T08:34:11.781 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 82.683159562, 2023-11-05T08:34:11.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025190444999999999, 2023-11-05T08:34:11.781 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:34:11.782 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:11.782 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:34:11.782 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:34:11.782 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:11.782 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 235 2023-11-05T08:34:11.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.783 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:34:11.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:34:11.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059370+0000", 2023-11-05T08:34:11.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:11.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:34:11.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059370+0000", 2023-11-05T08:34:11.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:11.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:34:11.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059371+0000", 2023-11-05T08:34:11.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2899999999999999e-06 2023-11-05T08:34:11.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:34:11.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059378+0000", 2023-11-05T08:34:11.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0729999999999999e-06 2023-11-05T08:34:11.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:34:11.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059408+0000", 2023-11-05T08:34:11.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9904999999999999e-05 2023-11-05T08:34:11.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:34:11.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059418+0000", 2023-11-05T08:34:11.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7659999999999993e-06 2023-11-05T08:34:11.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:34:11.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059648+0000", 2023-11-05T08:34:11.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022963200000000001 2023-11-05T08:34:11.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:34:11.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.059657+0000", 2023-11-05T08:34:11.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9700000000000005e-06 2023-11-05T08:34:11.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:34:11.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.070350+0000", 2023-11-05T08:34:11.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010693342 2023-11-05T08:34:11.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:34:11.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.070601+0000", 2023-11-05T08:34:11.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025082000000000001 2023-11-05T08:34:11.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:34:11.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.071898+0000", 2023-11-05T08:34:11.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012976190000000001 2023-11-05T08:34:11.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:34:11.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.077192+0000", 2023-11-05T08:34:11.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052931840000000003 2023-11-05T08:34:11.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:34:11.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.084398+0000", 2023-11-05T08:34:11.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0072062619999999997 2023-11-05T08:34:11.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:34:11.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:32:49.084560+0000", 2023-11-05T08:34:11.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016258200000000001 2023-11-05T08:34:11.796 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:11.796 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:34:11.796 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:11.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.797 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:349 3.as1 3:56f12b5c:::smithi09039252-15:head [copy-from ver 2] snapc 10=[7] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:34:11.797 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:05.254950+0000", 2023-11-05T08:34:11.797 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 66.487579564000001, 2023-11-05T08:34:11.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018320570000000001, 2023-11-05T08:34:11.797 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:34:11.798 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:11.798 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:34:11.798 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:34:11.798 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:11.798 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 349 2023-11-05T08:34:11.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.799 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:34:11.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:34:11.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.254950+0000", 2023-11-05T08:34:11.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:11.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:34:11.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.254950+0000", 2023-11-05T08:34:11.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:11.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:34:11.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.254953+0000", 2023-11-05T08:34:11.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6699999999999998e-06 2023-11-05T08:34:11.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:34:11.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.254966+0000", 2023-11-05T08:34:11.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.344e-05 2023-11-05T08:34:11.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:34:11.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255101+0000", 2023-11-05T08:34:11.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013478200000000001 2023-11-05T08:34:11.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:34:11.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255123+0000", 2023-11-05T08:34:11.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2541999999999999e-05 2023-11-05T08:34:11.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:34:11.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255409+0000", 2023-11-05T08:34:11.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028556799999999999 2023-11-05T08:34:11.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:34:11.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.255826+0000", 2023-11-05T08:34:11.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041679299999999999 2023-11-05T08:34:11.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:34:11.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.262849+0000", 2023-11-05T08:34:11.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0070237030000000001 2023-11-05T08:34:11.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:34:11.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.272188+0000", 2023-11-05T08:34:11.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0093385970000000006 2023-11-05T08:34:11.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:34:11.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.273116+0000", 2023-11-05T08:34:11.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00092777199999999999 2023-11-05T08:34:11.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:34:11.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.273271+0000", 2023-11-05T08:34:11.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015470300000000001 2023-11-05T08:34:11.810 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:11.810 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:34:11.811 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:11.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.811 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:357 3.as1 3:5396e455:::smithi09039252-43:head [read 0~1851392,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2023-11-05T08:34:11.811 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:05.279538+0000", 2023-11-05T08:34:11.812 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 66.462991591000005, 2023-11-05T08:34:11.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015991607000000001, 2023-11-05T08:34:11.812 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:34:11.812 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:34:11.812 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:34:11.812 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:34:11.813 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:11.813 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 357 2023-11-05T08:34:11.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.813 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:34:11.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:34:11.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.279538+0000", 2023-11-05T08:34:11.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:11.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:34:11.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.279538+0000", 2023-11-05T08:34:11.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:11.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:34:11.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.279538+0000", 2023-11-05T08:34:11.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0100000000000005e-07 2023-11-05T08:34:11.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:34:11.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.279542+0000", 2023-11-05T08:34:11.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8519999999999997e-06 2023-11-05T08:34:11.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:34:11.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.279551+0000", 2023-11-05T08:34:11.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2409999999999995e-06 2023-11-05T08:34:11.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:34:11.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.279557+0000", 2023-11-05T08:34:11.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4219999999999997e-06 2023-11-05T08:34:11.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:34:11.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.281448+0000", 2023-11-05T08:34:11.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001891104 2023-11-05T08:34:11.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:34:11.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.281739+0000", 2023-11-05T08:34:11.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029091399999999997 2023-11-05T08:34:11.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:34:11.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:05.295530+0000", 2023-11-05T08:34:11.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013790573 2023-11-05T08:34:11.821 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:11.821 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:34:11.821 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:11.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.822 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:405 3.as1 3:5396e455:::smithi09039252-43:head [append 0~622592] snapc 1b=[1b,1a,17,7] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2023-11-05T08:34:11.822 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:14.355514+0000", 2023-11-05T08:34:11.822 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 57.387015112, 2023-11-05T08:34:11.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022581047, 2023-11-05T08:34:11.823 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:34:11.823 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:11.823 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:34:11.823 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:34:11.824 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:11.824 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 405 2023-11-05T08:34:11.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.824 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:34:11.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:34:11.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.355514+0000", 2023-11-05T08:34:11.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:11.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:34:11.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.355514+0000", 2023-11-05T08:34:11.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:11.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:34:11.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.355517+0000", 2023-11-05T08:34:11.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7930000000000002e-06 2023-11-05T08:34:11.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:34:11.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.356220+0000", 2023-11-05T08:34:11.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00070250200000000003 2023-11-05T08:34:11.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:34:11.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.356253+0000", 2023-11-05T08:34:11.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3719e-05 2023-11-05T08:34:11.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:34:11.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.356268+0000", 2023-11-05T08:34:11.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5033e-05 2023-11-05T08:34:11.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:34:11.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.356444+0000", 2023-11-05T08:34:11.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000175549 2023-11-05T08:34:11.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-11-05T08:34:11.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.356471+0000", 2023-11-05T08:34:11.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6755999999999999e-05 2023-11-05T08:34:11.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:34:11.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.361462+0000", 2023-11-05T08:34:11.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049916350000000003 2023-11-05T08:34:11.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:34:11.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.361764+0000", 2023-11-05T08:34:11.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030161199999999998 2023-11-05T08:34:11.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:34:11.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.365071+0000", 2023-11-05T08:34:11.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033064940000000001 2023-11-05T08:34:11.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:34:11.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.377076+0000", 2023-11-05T08:34:11.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012005969999999999 2023-11-05T08:34:11.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:34:11.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.377756+0000", 2023-11-05T08:34:11.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067910900000000001 2023-11-05T08:34:11.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:34:11.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.378095+0000", 2023-11-05T08:34:11.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033987500000000001 2023-11-05T08:34:11.837 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:11.837 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:34:11.837 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:11.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.838 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:406 3.as1 3:5396e455:::smithi09039252-43:head [append 0~475136] snapc 1b=[1b,1a,17,7] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2023-11-05T08:34:11.838 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:14.363708+0000", 2023-11-05T08:34:11.838 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 57.378821492999997, 2023-11-05T08:34:11.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018912735999999999, 2023-11-05T08:34:11.839 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:34:11.839 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:11.839 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:34:11.839 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:34:11.839 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:11.839 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 406 2023-11-05T08:34:11.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.840 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:34:11.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:34:11.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.363708+0000", 2023-11-05T08:34:11.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:11.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:34:11.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.363708+0000", 2023-11-05T08:34:11.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:11.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:34:11.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.363710+0000", 2023-11-05T08:34:11.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3700000000000002e-06 2023-11-05T08:34:11.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:34:11.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.364282+0000", 2023-11-05T08:34:11.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057111800000000002 2023-11-05T08:34:11.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:34:11.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.364315+0000", 2023-11-05T08:34:11.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3868999999999997e-05 2023-11-05T08:34:11.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:34:11.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.364331+0000", 2023-11-05T08:34:11.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5130000000000001e-05 2023-11-05T08:34:11.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:34:11.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.369348+0000", 2023-11-05T08:34:11.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050178250000000001 2023-11-05T08:34:11.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:34:11.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.369681+0000", 2023-11-05T08:34:11.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033315800000000001 2023-11-05T08:34:11.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:34:11.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.371729+0000", 2023-11-05T08:34:11.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020475340000000002 2023-11-05T08:34:11.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:34:11.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.377079+0000", 2023-11-05T08:34:11.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053501529999999999 2023-11-05T08:34:11.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:34:11.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.382367+0000", 2023-11-05T08:34:11.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052874100000000002 2023-11-05T08:34:11.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:34:11.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.382621+0000", 2023-11-05T08:34:11.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000254169 2023-11-05T08:34:11.851 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:11.851 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:34:11.851 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:11.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.852 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:408 3.as1 3:5396e455:::smithi09039252-43:head [setxattr _header (57) in=7b] snapc 1b=[1b,1a,17,7] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2023-11-05T08:34:11.852 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:14.370011+0000", 2023-11-05T08:34:11.852 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 57.372518114999998, 2023-11-05T08:34:11.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016422719999999998, 2023-11-05T08:34:11.852 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:34:11.853 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:11.853 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:34:11.853 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:34:11.853 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:11.853 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 408 2023-11-05T08:34:11.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.854 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:34:11.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:34:11.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.370011+0000", 2023-11-05T08:34:11.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:11.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:34:11.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.370011+0000", 2023-11-05T08:34:11.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:11.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:34:11.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.370013+0000", 2023-11-05T08:34:11.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7370000000000001e-06 2023-11-05T08:34:11.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:34:11.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.370024+0000", 2023-11-05T08:34:11.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0976e-05 2023-11-05T08:34:11.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:34:11.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.370046+0000", 2023-11-05T08:34:11.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2065999999999998e-05 2023-11-05T08:34:11.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:34:11.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.370062+0000", 2023-11-05T08:34:11.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5407e-05 2023-11-05T08:34:11.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:34:11.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.378961+0000", 2023-11-05T08:34:11.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0088990579999999996 2023-11-05T08:34:11.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:34:11.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.379295+0000", 2023-11-05T08:34:11.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033471100000000001 2023-11-05T08:34:11.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:34:11.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.380741+0000", 2023-11-05T08:34:11.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014457129999999999 2023-11-05T08:34:11.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:34:11.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.383921+0000", 2023-11-05T08:34:11.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031802660000000002 2023-11-05T08:34:11.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:34:11.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.386122+0000", 2023-11-05T08:34:11.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022002219999999999 2023-11-05T08:34:11.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:34:11.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.386434+0000", 2023-11-05T08:34:11.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031256399999999998 2023-11-05T08:34:11.864 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:11.865 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:34:11.865 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:11.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.865 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:409 3.as1 3:5396e455:::smithi09039252-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e61)", 2023-11-05T08:34:11.866 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:14.370085+0000", 2023-11-05T08:34:11.866 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 57.372444162999997, 2023-11-05T08:34:11.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019674614999999999, 2023-11-05T08:34:11.866 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:34:11.866 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:34:11.867 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:34:11.867 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:34:11.867 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:11.867 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 409 2023-11-05T08:34:11.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.868 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:34:11.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:34:11.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.370085+0000", 2023-11-05T08:34:11.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:11.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:34:11.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.370085+0000", 2023-11-05T08:34:11.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:11.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:34:11.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.370087+0000", 2023-11-05T08:34:11.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2729999999999999e-06 2023-11-05T08:34:11.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:34:11.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.370093+0000", 2023-11-05T08:34:11.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0700000000000003e-06 2023-11-05T08:34:11.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:34:11.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.370113+0000", 2023-11-05T08:34:11.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0276e-05 2023-11-05T08:34:11.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:34:11.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.370132+0000", 2023-11-05T08:34:11.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8626000000000001e-05 2023-11-05T08:34:11.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:34:11.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.383576+0000", 2023-11-05T08:34:11.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013444717 2023-11-05T08:34:11.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-11-05T08:34:11.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.383807+0000", 2023-11-05T08:34:11.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023070100000000001 2023-11-05T08:34:11.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:34:11.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.387071+0000", 2023-11-05T08:34:11.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032641710000000002 2023-11-05T08:34:11.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:34:11.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.387425+0000", 2023-11-05T08:34:11.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035352400000000003 2023-11-05T08:34:11.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:34:11.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:14.389760+0000", 2023-11-05T08:34:11.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023352569999999999 2023-11-05T08:34:11.878 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:11.878 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:34:11.878 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:11.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.879 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:571 3.ds0 3:bf7e600d:::smithi09039252-23:head [rollback 7] snapc 27=[7] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2023-11-05T08:34:11.879 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:29.487342+0000", 2023-11-05T08:34:11.879 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 42.255187866999997, 2023-11-05T08:34:11.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017607034000000001, 2023-11-05T08:34:11.880 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:34:11.880 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:11.880 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:34:11.880 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:34:11.880 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:11.881 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 571 2023-11-05T08:34:11.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.881 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:34:11.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:34:11.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487342+0000", 2023-11-05T08:34:11.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:11.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:34:11.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487342+0000", 2023-11-05T08:34:11.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:11.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:34:11.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487343+0000", 2023-11-05T08:34:11.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1699999999999997e-07 2023-11-05T08:34:11.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:34:11.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487347+0000", 2023-11-05T08:34:11.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7099999999999998e-06 2023-11-05T08:34:11.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:34:11.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487362+0000", 2023-11-05T08:34:11.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4639e-05 2023-11-05T08:34:11.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:34:11.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487371+0000", 2023-11-05T08:34:11.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7700000000000007e-06 2023-11-05T08:34:11.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:34:11.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487637+0000", 2023-11-05T08:34:11.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026612599999999999 2023-11-05T08:34:11.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:34:11.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487655+0000", 2023-11-05T08:34:11.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7691999999999999e-05 2023-11-05T08:34:11.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:34:11.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.495233+0000", 2023-11-05T08:34:11.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0075784720000000002 2023-11-05T08:34:11.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:34:11.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.495448+0000", 2023-11-05T08:34:11.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000215274 2023-11-05T08:34:11.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:34:11.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.496724+0000", 2023-11-05T08:34:11.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012761669999999999 2023-11-05T08:34:11.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:34:11.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.499780+0000", 2023-11-05T08:34:11.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030555449999999998 2023-11-05T08:34:11.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:34:11.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.504307+0000", 2023-11-05T08:34:11.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0045266990000000003 2023-11-05T08:34:11.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:34:11.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.504949+0000", 2023-11-05T08:34:11.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064202300000000003 2023-11-05T08:34:11.893 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:11.894 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:34:11.894 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:11.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.894 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:572 3.ds0 3:bf7e600d:::smithi09039252-23:head [append 0~0] snapc 27=[7] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2023-11-05T08:34:11.895 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:29.487466+0000", 2023-11-05T08:34:11.895 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 42.255063827999997, 2023-11-05T08:34:11.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018798550000000001, 2023-11-05T08:34:11.895 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:34:11.895 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-05T08:34:11.896 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:34:11.896 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:34:11.896 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:11.896 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 572 2023-11-05T08:34:11.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.896 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:34:11.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:34:11.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487466+0000", 2023-11-05T08:34:11.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:11.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:34:11.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487466+0000", 2023-11-05T08:34:11.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:11.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:34:11.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487466+0000", 2023-11-05T08:34:11.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9599999999999998e-07 2023-11-05T08:34:11.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:34:11.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487471+0000", 2023-11-05T08:34:11.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2139999999999998e-06 2023-11-05T08:34:11.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:34:11.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487483+0000", 2023-11-05T08:34:11.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2048999999999999e-05 2023-11-05T08:34:11.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:34:11.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487496+0000", 2023-11-05T08:34:11.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2819999999999999e-05 2023-11-05T08:34:11.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:34:11.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487722+0000", 2023-11-05T08:34:11.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022617000000000001 2023-11-05T08:34:11.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:34:11.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.487738+0000", 2023-11-05T08:34:11.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6303e-05 2023-11-05T08:34:11.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:34:11.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.497597+0000", 2023-11-05T08:34:11.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098588319999999997 2023-11-05T08:34:11.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:34:11.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.497825+0000", 2023-11-05T08:34:11.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022776099999999999 2023-11-05T08:34:11.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:34:11.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.498970+0000", 2023-11-05T08:34:11.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001145622 2023-11-05T08:34:11.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:34:11.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.501416+0000", 2023-11-05T08:34:11.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002445268 2023-11-05T08:34:11.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-11-05T08:34:11.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.505581+0000", 2023-11-05T08:34:11.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0041650699999999999 2023-11-05T08:34:11.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:34:11.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:29.506264+0000", 2023-11-05T08:34:11.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00068364499999999998 2023-11-05T08:34:11.910 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:11.910 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:34:11.910 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:11.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.910 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:615 3.ds0 3:bf7e600d:::smithi09039252-23:head [read 0~1392640,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e81)", 2023-11-05T08:34:11.911 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:34.541498+0000", 2023-11-05T08:34:11.911 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 37.201031921999999, 2023-11-05T08:34:11.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01790396, 2023-11-05T08:34:11.911 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:34:11.911 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:34:11.911 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:34:11.912 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:34:11.912 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:11.912 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 615 2023-11-05T08:34:11.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.912 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:34:11.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:34:11.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541498+0000", 2023-11-05T08:34:11.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:11.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:34:11.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541498+0000", 2023-11-05T08:34:11.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:11.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:34:11.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541499+0000", 2023-11-05T08:34:11.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.623e-06 2023-11-05T08:34:11.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:34:11.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541510+0000", 2023-11-05T08:34:11.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0366999999999999e-05 2023-11-05T08:34:11.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:34:11.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541532+0000", 2023-11-05T08:34:11.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2762e-05 2023-11-05T08:34:11.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:34:11.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541549+0000", 2023-11-05T08:34:11.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6946e-05 2023-11-05T08:34:11.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:34:11.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541716+0000", 2023-11-05T08:34:11.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016715100000000001 2023-11-05T08:34:11.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:34:11.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541732+0000", 2023-11-05T08:34:11.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5506e-05 2023-11-05T08:34:11.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:34:11.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.549010+0000", 2023-11-05T08:34:11.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0072776270000000001 2023-11-05T08:34:11.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:34:11.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.549197+0000", 2023-11-05T08:34:11.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000187646 2023-11-05T08:34:11.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:34:11.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.559402+0000", 2023-11-05T08:34:11.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010204332 2023-11-05T08:34:11.922 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:11.923 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:34:11.923 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:11.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.923 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4691.0:616 3.ds0 3:bf7e600d:::smithi09039252-23:head [read 0~1392640,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e81)", 2023-11-05T08:34:11.924 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:34.541580+0000", 2023-11-05T08:34:11.924 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 37.200949588, 2023-11-05T08:34:11.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020950334000000001, 2023-11-05T08:34:11.924 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:34:11.924 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-11-05T08:34:11.924 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-11-05T08:34:11.925 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4691", 2023-11-05T08:34:11.925 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/1764107968", 2023-11-05T08:34:11.925 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 616 2023-11-05T08:34:11.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.925 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:34:11.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:34:11.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541580+0000", 2023-11-05T08:34:11.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:11.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.927 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:34:11.927 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541580+0000", 2023-11-05T08:34:11.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:11.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:34:11.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541581+0000", 2023-11-05T08:34:11.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.226e-06 2023-11-05T08:34:11.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:34:11.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541609+0000", 2023-11-05T08:34:11.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7645999999999999e-05 2023-11-05T08:34:11.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:34:11.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541629+0000", 2023-11-05T08:34:11.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0186e-05 2023-11-05T08:34:11.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:34:11.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541643+0000", 2023-11-05T08:34:11.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.428e-05 2023-11-05T08:34:11.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:34:11.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541783+0000", 2023-11-05T08:34:11.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013925799999999999 2023-11-05T08:34:11.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-11-05T08:34:11.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.541796+0000", 2023-11-05T08:34:11.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3426e-05 2023-11-05T08:34:11.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:34:11.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.556302+0000", 2023-11-05T08:34:11.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014506022 2023-11-05T08:34:11.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-11-05T08:34:11.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.556475+0000", 2023-11-05T08:34:11.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017302999999999999 2023-11-05T08:34:11.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:34:11.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:34.562530+0000", 2023-11-05T08:34:11.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0060552599999999998 2023-11-05T08:34:11.935 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:11.935 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:34:11.935 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:11.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.936 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.19s5 97/45 ECSubWrite(tid=61, reqid=osd.8.0:61, at_version=97'41, trim_to=0'0, roll_forward_to=89'37))", 2023-11-05T08:34:11.936 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:50.661913+0000", 2023-11-05T08:34:11.936 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 21.080616729999999, 2023-11-05T08:34:11.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020246515999999999, 2023-11-05T08:34:11.937 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:34:11.937 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:34:11.937 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:34:11.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:34:11.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.661913+0000", 2023-11-05T08:34:11.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:11.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:34:11.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.661913+0000", 2023-11-05T08:34:11.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:11.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:34:11.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.661914+0000", 2023-11-05T08:34:11.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2300000000000001e-06 2023-11-05T08:34:11.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:34:11.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.661979+0000", 2023-11-05T08:34:11.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4523999999999994e-05 2023-11-05T08:34:11.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:34:11.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.662002+0000", 2023-11-05T08:34:11.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3813000000000001e-05 2023-11-05T08:34:11.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:34:11.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.662023+0000", 2023-11-05T08:34:11.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.09e-05 2023-11-05T08:34:11.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:34:11.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.662168+0000", 2023-11-05T08:34:11.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014468700000000001 2023-11-05T08:34:11.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:34:11.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.662190+0000", 2023-11-05T08:34:11.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1818999999999999e-05 2023-11-05T08:34:11.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:34:11.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.682067+0000", 2023-11-05T08:34:11.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019877145999999998 2023-11-05T08:34:11.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:34:11.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.682159+0000", 2023-11-05T08:34:11.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2397000000000003e-05 2023-11-05T08:34:11.946 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:11.946 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:34:11.947 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:11.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.947 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.18s5 97/45 ECSubWrite(tid=319, reqid=osd.13.0:319, at_version=97'44, trim_to=0'0, roll_forward_to=68'40))", 2023-11-05T08:34:11.947 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:50.668304+0000", 2023-11-05T08:34:11.947 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 21.074225723000001, 2023-11-05T08:34:11.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0221633, 2023-11-05T08:34:11.948 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:34:11.948 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:34:11.948 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:34:11.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:34:11.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.668304+0000", 2023-11-05T08:34:11.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:11.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:34:11.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.668304+0000", 2023-11-05T08:34:11.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:11.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:34:11.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.668304+0000", 2023-11-05T08:34:11.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1600000000000001e-07 2023-11-05T08:34:11.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:34:11.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.668356+0000", 2023-11-05T08:34:11.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1258000000000003e-05 2023-11-05T08:34:11.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:34:11.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.668372+0000", 2023-11-05T08:34:11.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6521999999999999e-05 2023-11-05T08:34:11.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:34:11.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.668384+0000", 2023-11-05T08:34:11.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2157999999999999e-05 2023-11-05T08:34:11.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:34:11.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.674941+0000", 2023-11-05T08:34:11.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0065561839999999996 2023-11-05T08:34:11.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:34:11.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.674968+0000", 2023-11-05T08:34:11.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7946e-05 2023-11-05T08:34:11.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:34:11.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.690370+0000", 2023-11-05T08:34:11.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015401540999999999 2023-11-05T08:34:11.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:34:11.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.690467+0000", 2023-11-05T08:34:11.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7075000000000001e-05 2023-11-05T08:34:11.957 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:11.957 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:34:11.957 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:11.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.958 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.10s2 97/45 ECSubWrite(tid=320, reqid=osd.13.0:320, at_version=97'60, trim_to=0'0, roll_forward_to=93'56))", 2023-11-05T08:34:11.958 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:50.668452+0000", 2023-11-05T08:34:11.958 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 21.074077688999999, 2023-11-05T08:34:11.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021907058, 2023-11-05T08:34:11.959 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:34:11.959 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:34:11.959 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:34:11.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:34:11.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.668452+0000", 2023-11-05T08:34:11.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:11.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:34:11.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.668452+0000", 2023-11-05T08:34:11.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:11.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:34:11.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.668453+0000", 2023-11-05T08:34:11.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2399999999999997e-07 2023-11-05T08:34:11.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:34:11.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.668485+0000", 2023-11-05T08:34:11.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2336999999999999e-05 2023-11-05T08:34:11.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:34:11.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.668497+0000", 2023-11-05T08:34:11.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2303999999999999e-05 2023-11-05T08:34:11.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:34:11.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.668506+0000", 2023-11-05T08:34:11.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7839999999999992e-06 2023-11-05T08:34:11.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:34:11.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.674892+0000", 2023-11-05T08:34:11.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.006385913 2023-11-05T08:34:11.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:34:11.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.674918+0000", 2023-11-05T08:34:11.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5644999999999999e-05 2023-11-05T08:34:11.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:34:11.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.690272+0000", 2023-11-05T08:34:11.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01535446 2023-11-05T08:34:11.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:34:11.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.690359+0000", 2023-11-05T08:34:11.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6891000000000005e-05 2023-11-05T08:34:11.968 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:11.968 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:34:11.968 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:11.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.969 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.17s4 97/96 ECSubWrite(tid=56, reqid=osd.5.0:56, at_version=97'23, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:34:11.969 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:50.680183+0000", 2023-11-05T08:34:11.969 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 21.062346126000001, 2023-11-05T08:34:11.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020671399, 2023-11-05T08:34:11.969 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:34:11.970 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:34:11.970 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:34:11.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:34:11.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.680183+0000", 2023-11-05T08:34:11.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:11.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:34:11.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.680183+0000", 2023-11-05T08:34:11.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:11.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:34:11.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.680185+0000", 2023-11-05T08:34:11.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6929999999999999e-06 2023-11-05T08:34:11.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:34:11.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.680239+0000", 2023-11-05T08:34:11.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3915000000000001e-05 2023-11-05T08:34:11.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:34:11.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.680271+0000", 2023-11-05T08:34:11.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.184e-05 2023-11-05T08:34:11.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:34:11.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.680322+0000", 2023-11-05T08:34:11.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1644999999999998e-05 2023-11-05T08:34:11.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:34:11.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.680561+0000", 2023-11-05T08:34:11.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023886099999999999 2023-11-05T08:34:11.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:34:11.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.680584+0000", 2023-11-05T08:34:11.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2846000000000001e-05 2023-11-05T08:34:11.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:34:11.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.700774+0000", 2023-11-05T08:34:11.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020189948999999999 2023-11-05T08:34:11.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:34:11.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.700855+0000", 2023-11-05T08:34:11.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0649999999999995e-05 2023-11-05T08:34:11.979 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:11.979 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:34:11.980 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:11.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.980 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.13s1 97/45 ECSubWrite(tid=168, reqid=osd.11.0:168, at_version=97'25, trim_to=0'0, roll_forward_to=92'23))", 2023-11-05T08:34:11.980 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:50.683199+0000", 2023-11-05T08:34:11.980 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 21.059330547999998, 2023-11-05T08:34:11.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017730752999999998, 2023-11-05T08:34:11.981 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:34:11.981 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:34:11.981 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:34:11.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:34:11.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.683199+0000", 2023-11-05T08:34:11.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:11.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:34:11.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.683199+0000", 2023-11-05T08:34:11.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:11.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:34:11.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.683200+0000", 2023-11-05T08:34:11.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5000000000000002e-07 2023-11-05T08:34:11.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:34:11.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.683265+0000", 2023-11-05T08:34:11.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5061999999999995e-05 2023-11-05T08:34:11.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:34:11.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.683288+0000", 2023-11-05T08:34:11.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2982000000000001e-05 2023-11-05T08:34:11.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:34:11.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.683308+0000", 2023-11-05T08:34:11.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0699999999999998e-05 2023-11-05T08:34:11.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:34:11.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.683481+0000", 2023-11-05T08:34:11.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000172709 2023-11-05T08:34:11.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:34:11.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.683508+0000", 2023-11-05T08:34:11.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6546e-05 2023-11-05T08:34:11.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:34:11.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.700864+0000", 2023-11-05T08:34:11.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017355977000000002 2023-11-05T08:34:11.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:34:11.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.700930+0000", 2023-11-05T08:34:11.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6126999999999999e-05 2023-11-05T08:34:11.990 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:11.990 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:34:11.991 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:11.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.991 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.19s5 97/45 ECSubWrite(tid=62, reqid=osd.8.0:61, at_version=0'0, trim_to=0'0, roll_forward_to=97'41))", 2023-11-05T08:34:11.991 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:50.684275+0000", 2023-11-05T08:34:11.991 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 21.058254837, 2023-11-05T08:34:11.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017074500999999999, 2023-11-05T08:34:11.992 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:34:11.992 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:34:11.992 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:34:11.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:34:11.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.684275+0000", 2023-11-05T08:34:11.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:11.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:34:11.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.684275+0000", 2023-11-05T08:34:11.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:11.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:34:11.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.684277+0000", 2023-11-05T08:34:11.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0829999999999998e-06 2023-11-05T08:34:11.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:34:11.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.684306+0000", 2023-11-05T08:34:11.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9173e-05 2023-11-05T08:34:11.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:34:11.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.684383+0000", 2023-11-05T08:34:11.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6897999999999995e-05 2023-11-05T08:34:11.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:34:11.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.684408+0000", 2023-11-05T08:34:11.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5265999999999998e-05 2023-11-05T08:34:11.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:34:11.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.684576+0000", 2023-11-05T08:34:11.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016833000000000001 2023-11-05T08:34:11.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:11.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:11.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:34:11.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.684629+0000", 2023-11-05T08:34:11.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2914999999999997e-05 2023-11-05T08:34:12.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:12.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:12.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:34:12.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.701282+0000", 2023-11-05T08:34:12.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016653094 2023-11-05T08:34:12.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:12.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:12.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:34:12.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.701349+0000", 2023-11-05T08:34:12.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6742000000000006e-05 2023-11-05T08:34:12.001 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:12.002 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:34:12.002 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:12.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:12.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:12.002 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDECSubOpWrite(3.17s4 97/96 ECSubWrite(tid=57, reqid=osd.5.0:57, at_version=97'25, trim_to=0'0, roll_forward_to=0'0))", 2023-11-05T08:34:12.002 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-11-05T08:33:50.684443+0000", 2023-11-05T08:34:12.003 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 21.058086781, 2023-11-05T08:34:12.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021330211000000002, 2023-11-05T08:34:12.003 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-11-05T08:34:12.003 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-11-05T08:34:12.003 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-11-05T08:34:12.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:12.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-11-05T08:34:12.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.684443+0000", 2023-11-05T08:34:12.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:12.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:12.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:12.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-11-05T08:34:12.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.684443+0000", 2023-11-05T08:34:12.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-11-05T08:34:12.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:12.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:12.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-11-05T08:34:12.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.684444+0000", 2023-11-05T08:34:12.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2869999999999999e-06 2023-11-05T08:34:12.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:12.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:12.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-11-05T08:34:12.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.684497+0000", 2023-11-05T08:34:12.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2825000000000003e-05 2023-11-05T08:34:12.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:12.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:12.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-11-05T08:34:12.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.684538+0000", 2023-11-05T08:34:12.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1053000000000001e-05 2023-11-05T08:34:12.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:12.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:12.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-11-05T08:34:12.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.684560+0000", 2023-11-05T08:34:12.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2158999999999999e-05 2023-11-05T08:34:12.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:12.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:12.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-11-05T08:34:12.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.688274+0000", 2023-11-05T08:34:12.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037137720000000002 2023-11-05T08:34:12.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:12.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:12.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_started", 2023-11-05T08:34:12.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.688297+0000", 2023-11-05T08:34:12.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3623000000000001e-05 2023-11-05T08:34:12.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:12.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:12.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_committed", 2023-11-05T08:34:12.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.705699+0000", 2023-11-05T08:34:12.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017401137000000001 2023-11-05T08:34:12.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-11-05T08:34:12.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-11-05T08:34:12.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-11-05T08:34:12.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-11-05T08:33:50.705773+0000", 2023-11-05T08:34:12.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4355000000000002e-05 2023-11-05T08:34:12.013 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:12.013 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:34:12.013 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:12.013 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-11-05T08:34:12.013 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-11-05T08:34:12.014 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:34:12.015 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2023-11-05T08:34:12.016 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:34:12.011+0000 7f429c947640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:34:12.042 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-05T08:34:12.116 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-11-05T08:34:12.114+0000 7f72b2c5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 42525) UID: 0 2023-11-05T08:34:12.146 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:34:12.147 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-11-05T08:34:12.147 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-11-05T08:34:12.147 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:34:12.160 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2023-11-05T08:34:12.189 INFO:teuthology.orchestra.run.smithi002.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-11-05T08:34:12.210 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1: {} 2023-11-05T08:34:12.211 INFO:teuthology.orchestra.run.smithi002.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-11-05T08:34:12.217 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:34:12.212+0000 7f8cbc47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:34:12.231 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2: {} 2023-11-05T08:34:12.231 INFO:teuthology.orchestra.run.smithi002.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-11-05T08:34:12.249 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3: {} 2023-11-05T08:34:12.249 INFO:teuthology.orchestra.run.smithi002.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-11-05T08:34:12.265 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4: {} 2023-11-05T08:34:12.266 INFO:teuthology.orchestra.run.smithi002.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-11-05T08:34:12.286 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5: {} 2023-11-05T08:34:12.286 INFO:teuthology.orchestra.run.smithi002.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-11-05T08:34:12.306 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6: {} 2023-11-05T08:34:12.307 INFO:teuthology.orchestra.run.smithi002.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-11-05T08:34:12.318 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:34:12.315+0000 7fcd95813640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:34:12.324 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-11-05T08:34:12.325 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-11-05T08:34:12.325 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-11-05T08:34:12.325 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-11-05T08:34:12.325 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-11-05T08:34:12.339 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2023-11-05T08:34:12.342 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7: {} 2023-11-05T08:34:12.342 INFO:teuthology.orchestra.run.smithi002.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-11-05T08:34:12.359 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8: {} 2023-11-05T08:34:12.359 INFO:teuthology.orchestra.run.smithi002.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-11-05T08:34:12.379 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9: {} 2023-11-05T08:34:12.380 INFO:teuthology.orchestra.run.smithi002.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-11-05T08:34:12.399 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10: {} 2023-11-05T08:34:12.400 INFO:teuthology.orchestra.run.smithi002.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-11-05T08:34:12.417 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11: {} 2023-11-05T08:34:12.417 INFO:teuthology.orchestra.run.smithi002.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-11-05T08:34:12.418 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:34:12.414+0000 7f00940d2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:34:12.434 INFO:teuthology.orchestra.run.smithi002.stdout:osd.12: {} 2023-11-05T08:34:12.434 INFO:teuthology.orchestra.run.smithi002.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-11-05T08:34:12.454 INFO:teuthology.orchestra.run.smithi002.stdout:osd.13: {} 2023-11-05T08:34:12.455 INFO:teuthology.orchestra.run.smithi002.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-11-05T08:34:12.461 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:34:12.461 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:34:12.475 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2023-11-05T08:34:12.476 INFO:teuthology.orchestra.run.smithi002.stdout:osd.14: {} 2023-11-05T08:34:12.476 INFO:teuthology.orchestra.run.smithi002.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-11-05T08:34:12.493 INFO:teuthology.orchestra.run.smithi002.stdout:osd.15: {} 2023-11-05T08:34:12.494 INFO:teuthology.orchestra.run.smithi002.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-11-05T08:34:12.497 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-05T08:34:12.498 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-05T08:34:12.519 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:34:12.512+0000 7ff0d5267640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:34:12.620 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:34:12.616+0000 7f168a3b6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:34:12.636 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:34:12.637 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:34:12.651 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2023-11-05T08:34:12.673 INFO:teuthology.orchestra.run.smithi002.stderr:set osd.12 primary-affinity to 0.260655 (8170822) 2023-11-05T08:34:12.673 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'unique_pool_0' removed 2023-11-05T08:34:12.704 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2023-11-05T08:34:12.714 INFO:tasks.thrashosds:joining thrashosds 2023-11-05T08:34:12.785 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:34:12.786 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:34:12.800 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2023-11-05T08:34:12.923 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:34:12.923 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:34:12.937 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-11-05T08:34:13.054 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:34:13.054 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:34:13.068 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-11-05T08:34:13.189 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:34:13.189 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:34:13.203 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-11-05T08:34:13.325 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-11-05T08:34:13.325 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-05T08:34:13.690 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-11-05T08:34:14.706 INFO:teuthology.orchestra.run.smithi002.stderr:noscrub is unset 2023-11-05T08:34:14.731 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-11-05T08:34:15.716 INFO:teuthology.orchestra.run.smithi002.stderr:nodeep-scrub is unset 2023-11-05T08:34:15.934 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~80s 2023-11-05T08:34:17.715 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2023-11-05T08:34:17.863 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a: {} 2023-11-05T08:34:17.864 INFO:teuthology.orchestra.run.smithi002.stderr:mon.a: mon_osd_down_out_interval = '600' (not observed, change may require restart) 2023-11-05T08:34:17.876 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b: {} 2023-11-05T08:34:17.876 INFO:teuthology.orchestra.run.smithi002.stderr:mon.b: mon_osd_down_out_interval = '600' (not observed, change may require restart) 2023-11-05T08:34:17.890 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c: {} 2023-11-05T08:34:17.890 INFO:teuthology.orchestra.run.smithi002.stderr:mon.c: mon_osd_down_out_interval = '600' (not observed, change may require restart) 2023-11-05T08:34:17.907 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2023-11-05T08:34:18.739 INFO:teuthology.orchestra.run.smithi002.stderr:reweighted osd.0 to 1 (10000) 2023-11-05T08:34:18.766 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2023-11-05T08:34:19.742 INFO:teuthology.orchestra.run.smithi002.stderr:set osd.0 primary-affinity to 1 (8655362) 2023-11-05T08:34:19.767 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2023-11-05T08:34:20.754 INFO:teuthology.orchestra.run.smithi002.stderr:reweighted osd.4 to 1 (10000) 2023-11-05T08:34:20.782 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2023-11-05T08:34:21.766 INFO:teuthology.orchestra.run.smithi002.stderr:set osd.4 primary-affinity to 1 (8655362) 2023-11-05T08:34:21.791 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2023-11-05T08:34:22.778 INFO:teuthology.orchestra.run.smithi002.stderr:reweighted osd.8 to 1 (10000) 2023-11-05T08:34:22.806 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 8 1 2023-11-05T08:34:22.952 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~87s 2023-11-05T08:34:23.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-05T08:34:23.140+0000 7f5c61801640 0 --2- 172.21.15.2:0/1237585634 >> v2:172.21.15.2:3300/0 conn(0x7f5c5c0c6e20 0x7f5c5c157910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-11-05T08:34:23.790 INFO:teuthology.orchestra.run.smithi002.stderr:set osd.8 primary-affinity to 1 (8655362) 2023-11-05T08:34:23.818 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 12 1 2023-11-05T08:34:24.801 INFO:teuthology.orchestra.run.smithi002.stderr:reweighted osd.12 to 1 (10000) 2023-11-05T08:34:24.826 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 12 1 2023-11-05T08:34:25.814 INFO:teuthology.orchestra.run.smithi002.stderr:set osd.12 primary-affinity to 1 (8655362) 2023-11-05T08:34:25.841 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2023-11-05T08:34:26.833 INFO:teuthology.orchestra.run.smithi002.stderr:reweighted osd.1 to 1 (10000) 2023-11-05T08:34:26.859 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2023-11-05T08:34:27.836 INFO:teuthology.orchestra.run.smithi002.stderr:set osd.1 primary-affinity to 1 (8655362) 2023-11-05T08:34:27.868 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2023-11-05T08:34:28.849 INFO:teuthology.orchestra.run.smithi002.stderr:reweighted osd.5 to 1 (10000) 2023-11-05T08:34:28.876 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2023-11-05T08:34:29.861 INFO:teuthology.orchestra.run.smithi002.stderr:set osd.5 primary-affinity to 1 (8655362) 2023-11-05T08:34:29.890 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2023-11-05T08:34:29.968 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~94s 2023-11-05T08:34:30.873 INFO:teuthology.orchestra.run.smithi002.stderr:reweighted osd.9 to 1 (10000) 2023-11-05T08:34:30.898 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 9 1 2023-11-05T08:34:31.885 INFO:teuthology.orchestra.run.smithi002.stderr:set osd.9 primary-affinity to 1 (8655362) 2023-11-05T08:34:31.911 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 13 1 2023-11-05T08:34:32.904 INFO:teuthology.orchestra.run.smithi002.stderr:reweighted osd.13 to 1 (10000) 2023-11-05T08:34:32.928 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 13 1 2023-11-05T08:34:33.907 INFO:teuthology.orchestra.run.smithi002.stderr:set osd.13 primary-affinity to 1 (8655362) 2023-11-05T08:34:33.934 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2023-11-05T08:34:34.920 INFO:teuthology.orchestra.run.smithi002.stderr:reweighted osd.2 to 1 (10000) 2023-11-05T08:34:34.951 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2023-11-05T08:34:35.932 INFO:teuthology.orchestra.run.smithi002.stderr:set osd.2 primary-affinity to 1 (8655362) 2023-11-05T08:34:35.956 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2023-11-05T08:34:36.950 INFO:teuthology.orchestra.run.smithi002.stderr:reweighted osd.6 to 1 (10000) 2023-11-05T08:34:36.976 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2023-11-05T08:34:36.988 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~101s 2023-11-05T08:34:37.961 INFO:teuthology.orchestra.run.smithi002.stderr:set osd.6 primary-affinity to 1 (8655362) 2023-11-05T08:34:37.986 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2023-11-05T08:34:38.973 INFO:teuthology.orchestra.run.smithi002.stderr:reweighted osd.10 to 1 (10000) 2023-11-05T08:34:38.999 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 10 1 2023-11-05T08:34:39.985 INFO:teuthology.orchestra.run.smithi002.stderr:set osd.10 primary-affinity to 1 (8655362) 2023-11-05T08:34:40.014 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 1 2023-11-05T08:34:40.997 INFO:teuthology.orchestra.run.smithi002.stderr:reweighted osd.14 to 1 (10000) 2023-11-05T08:34:41.021 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 14 1 2023-11-05T08:34:42.009 INFO:teuthology.orchestra.run.smithi002.stderr:set osd.14 primary-affinity to 1 (8655362) 2023-11-05T08:34:42.037 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2023-11-05T08:34:43.021 INFO:teuthology.orchestra.run.smithi002.stderr:reweighted osd.7 to 1 (10000) 2023-11-05T08:34:43.045 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2023-11-05T08:34:44.005 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~108s 2023-11-05T08:34:44.033 INFO:teuthology.orchestra.run.smithi002.stderr:set osd.7 primary-affinity to 1 (8655362) 2023-11-05T08:34:44.057 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2023-11-05T08:34:45.053 INFO:teuthology.orchestra.run.smithi002.stderr:reweighted osd.11 to 1 (10000) 2023-11-05T08:34:45.077 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 11 1 2023-11-05T08:34:46.057 INFO:teuthology.orchestra.run.smithi002.stderr:set osd.11 primary-affinity to 1 (8655362) 2023-11-05T08:34:46.083 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2023-11-05T08:34:47.076 INFO:teuthology.orchestra.run.smithi002.stderr:reweighted osd.15 to 1 (10000) 2023-11-05T08:34:47.102 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 15 1 2023-11-05T08:34:48.086 INFO:teuthology.orchestra.run.smithi002.stderr:set osd.15 primary-affinity to 1 (8655362) 2023-11-05T08:34:48.110 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2023-11-05T08:34:49.090 INFO:teuthology.orchestra.run.smithi002.stderr:reweighted osd.3 to 1 (10000) 2023-11-05T08:34:49.117 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2023-11-05T08:34:50.101 INFO:teuthology.orchestra.run.smithi002.stderr:set osd.3 primary-affinity to 1 (8655362) 2023-11-05T08:34:50.127 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2023-11-05T08:34:50.128 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2023-11-05T08:34:50.128 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2023-11-05T08:34:50.128 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2023-11-05T08:34:50.128 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-11-05T08:34:50.129 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:34:50.430 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:34:50.430 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":154,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:34:50.098307+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":119,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:56.093990+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:48.502629+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:46.337041+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:37.463501+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:54.776271+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:46.876488+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:52.787222+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:44.790270+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:45.201083+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:51.913597+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:26.992305+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:29.690235+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:49.445290+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:40.245020+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:43.650883+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:34:51.022 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~115s 2023-11-05T08:34:53.450 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:34:53.762 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:34:53.762 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":155,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:34:51.110635+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:56.093990+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:48.502629+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:46.337041+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:37.463501+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:54.776271+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:46.876488+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:52.787222+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:44.790270+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:45.201083+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:51.913597+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:26.992305+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:29.690235+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:49.445290+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:40.245020+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:43.650883+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:34:56.781 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:34:57.088 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:34:57.089 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":157,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:34:55.144237+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:56.093990+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:48.502629+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:46.337041+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:37.463501+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:54.776271+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:46.876488+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:52.787222+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:44.790270+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:45.201083+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:51.913597+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:54.173556+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:53.738606+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:49.445290+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:40.245020+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:43.650883+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:34:58.040 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~122s 2023-11-05T08:35:00.107 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:35:00.426 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:35:00.427 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":158,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:34:58.611437+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:56.093990+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:48.502629+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:46.337041+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:37.463501+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:54.776271+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:46.876488+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:52.787222+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:44.790270+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:45.201083+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:51.913597+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:54.173556+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:53.738606+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:49.445290+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:58.559090+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:43.650883+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:35:03.448 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:35:03.762 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:35:03.763 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":161,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:35:03.691153+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:56.093990+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:48.502629+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:46.337041+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:01.210525+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:54.776271+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:46.876488+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:52.787222+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:44.790270+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:03.525312+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:00.106277+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:54.173556+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:53.738606+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:49.445290+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:58.559090+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:43.650883+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:35:05.057 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~129s 2023-11-05T08:35:06.781 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:35:07.093 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:35:07.094 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":163,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:35:06.722770+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:04.083659+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:06.619351+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:46.337041+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:01.210525+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:54.776271+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.922992+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:52.787222+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:44.790270+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:03.525312+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:00.106277+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:54.173556+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:53.738606+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:49.445290+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:58.559090+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.769648+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:35:10.114 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:35:10.425 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:35:10.425 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":163,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:35:06.722770+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:04.083659+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:06.619351+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:46.337041+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:01.210525+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:54.776271+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.922992+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:52.787222+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:44.790270+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:03.525312+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:00.106277+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:54.173556+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:53.738606+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:49.445290+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:58.559090+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.769648+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:35:12.073 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~136s 2023-11-05T08:35:13.445 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:35:13.754 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:35:13.755 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":163,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:35:06.722770+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:04.083659+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:06.619351+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:46.337041+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:01.210525+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:54.776271+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.922992+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:52.787222+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:44.790270+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:03.525312+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:00.106277+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:54.173556+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:53.738606+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:49.445290+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:58.559090+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.769648+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:35:16.774 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:35:17.087 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:35:17.087 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":164,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:35:13.757262+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:04.083659+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:06.619351+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:12.810102+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:01.210525+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:54.776271+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.922992+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:52.787222+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:44.790270+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:03.525312+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:00.106277+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:54.173556+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:53.738606+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:49.445290+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:58.559090+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.769648+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:35:19.091 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~143s 2023-11-05T08:35:20.107 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:35:20.426 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:35:20.426 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":164,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:35:13.757262+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:04.083659+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:06.619351+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:12.810102+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:01.210525+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:54.776271+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.922992+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:52.787222+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:44.790270+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:03.525312+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:00.106277+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:54.173556+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:53.738606+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:49.445290+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:58.559090+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.769648+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:35:23.442 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:35:23.749 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:35:23.749 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":165,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:35:21.793703+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:04.083659+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:06.619351+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:12.810102+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:01.210525+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:33:54.776271+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.922992+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:20.804823+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:44.790270+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:03.525312+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:00.106277+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:54.173556+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:53.738606+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:49.445290+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:58.559090+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.769648+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:35:26.107 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~150s 2023-11-05T08:35:26.769 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:35:27.083 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:35:27.084 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":166,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:35:23.813821+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:04.083659+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:06.619351+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:12.810102+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:01.210525+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:23.083007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.922992+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:20.804823+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:44.790270+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:03.525312+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:00.106277+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:54.173556+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:53.738606+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:49.445290+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:58.559090+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.769648+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:35:30.104 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:35:30.415 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:35:30.415 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":166,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:35:23.813821+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:04.083659+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:06.619351+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:12.810102+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:01.210525+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:23.083007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.922992+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:20.804823+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:44.790270+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:03.525312+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:00.106277+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:54.173556+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:53.738606+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:49.445290+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:58.559090+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.769648+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:35:33.125 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~157s 2023-11-05T08:35:33.435 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:35:33.746 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:35:33.747 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":166,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:35:23.813821+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:04.083659+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:06.619351+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:12.810102+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:01.210525+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:23.083007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.922992+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:20.804823+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:44.790270+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:03.525312+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:00.106277+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:54.173556+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:53.738606+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:49.445290+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:58.559090+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.769648+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:35:36.767 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:35:37.077 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:35:37.077 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":166,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:35:23.813821+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:04.083659+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:06.619351+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:12.810102+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:01.210525+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:23.083007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.922992+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:20.804823+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:44.790270+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:03.525312+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:00.106277+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:54.173556+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:53.738606+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:49.445290+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:58.559090+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.769648+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:35:40.095 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:35:40.141 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~164s 2023-11-05T08:35:40.407 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:35:40.407 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":166,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:35:23.813821+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:04.083659+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:06.619351+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:12.810102+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:01.210525+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:23.083007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.922992+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:20.804823+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:44.790270+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:03.525312+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:00.106277+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:54.173556+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:53.738606+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:49.445290+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:58.559090+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.769648+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:35:43.427 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:35:43.736 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:35:43.737 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":166,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:35:23.813821+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:04.083659+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:06.619351+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:12.810102+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:01.210525+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:23.083007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.922992+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:20.804823+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:44.790270+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:03.525312+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:00.106277+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:54.173556+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:53.738606+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:49.445290+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:58.559090+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.769648+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:35:46.757 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:35:47.069 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:35:47.070 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":166,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:35:23.813821+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:04.083659+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:06.619351+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:12.810102+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:01.210525+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:23.083007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.922992+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:20.804823+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:44.790270+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:03.525312+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:00.106277+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:54.173556+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:53.738606+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:49.445290+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:58.559090+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.769648+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:35:47.157 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~172s 2023-11-05T08:35:50.089 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:35:50.403 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:35:50.404 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":167,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:35:47.793969+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:04.083659+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:06.619351+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:12.810102+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:01.210525+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:23.083007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.922992+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:20.804823+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:47.646442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:03.525312+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:00.106277+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:54.173556+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:53.738606+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:49.445290+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:58.559090+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.769648+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:35:53.424 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:35:53.736 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:35:53.737 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":167,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:35:47.793969+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:04.083659+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:06.619351+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:12.810102+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:01.210525+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:23.083007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.922992+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:20.804823+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:47.646442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:03.525312+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:00.106277+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:54.173556+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:53.738606+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:49.445290+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:58.559090+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.769648+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:35:54.173 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~179s 2023-11-05T08:35:56.757 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:35:57.068 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:35:57.068 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":167,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:35:47.793969+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:04.083659+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:06.619351+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:12.810102+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:01.210525+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:23.083007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.922992+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:20.804823+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:47.646442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:03.525312+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:00.106277+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:54.173556+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:53.738606+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:49.445290+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:58.559090+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.769648+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:36:00.088 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:36:00.391 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:36:00.391 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":167,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:35:47.793969+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:04.083659+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:06.619351+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:12.810102+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:01.210525+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:23.083007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.922992+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:20.804823+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:47.646442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:03.525312+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:00.106277+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:54.173556+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:53.738606+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:49.445290+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:58.559090+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.769648+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:36:01.191 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~186s 2023-11-05T08:36:03.411 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:36:03.722 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:36:03.723 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":167,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:35:47.793969+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:04.083659+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:06.619351+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:12.810102+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:01.210525+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:23.083007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.922992+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:20.804823+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:47.646442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:03.525312+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:00.106277+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:54.173556+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:53.738606+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:49.445290+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:58.559090+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.769648+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:36:06.744 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:36:07.051 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:36:07.051 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":167,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:35:47.793969+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:04.083659+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:06.619351+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:12.810102+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:01.210525+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:23.083007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.922992+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:20.804823+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:47.646442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:03.525312+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:00.106277+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:54.173556+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:53.738606+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:49.445290+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:58.559090+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.769648+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:36:08.208 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~193s 2023-11-05T08:36:10.069 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:36:10.377 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:36:10.377 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":168,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:36:09.805507+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:04.083659+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:06.619351+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:12.810102+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:01.210525+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:23.083007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.922992+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:20.804823+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:47.646442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:03.525312+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:00.106277+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:54.173556+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:53.738606+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:09.324095+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:58.559090+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.769648+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:36:13.397 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:36:13.705 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:36:13.705 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":168,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:36:09.805507+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:04.083659+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:06.619351+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:12.810102+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:01.210525+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:23.083007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.922992+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:20.804823+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:47.646442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:03.525312+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:00.106277+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:54.173556+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:53.738606+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:09.324095+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:58.559090+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.769648+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:36:15.225 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~200s 2023-11-05T08:36:16.725 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:36:17.034 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:36:17.034 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":169,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:36:16.705252+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:04.083659+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:06.619351+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:12.810102+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:01.210525+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:23.083007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:16.458045+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:20.804823+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:47.646442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:03.525312+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:00.106277+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:54.173556+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:53.738606+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:09.324095+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:58.559090+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.769648+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:36:20.055 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:36:20.365 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:36:20.366 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":170,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:36:17.718062+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:04.083659+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:06.619351+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:12.810102+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:01.210525+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:23.083007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:16.458045+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:20.804823+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:47.646442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:03.525312+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:17.653294+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:54.173556+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:53.738606+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:09.324095+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:58.559090+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.769648+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:36:22.241 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~207s 2023-11-05T08:36:23.386 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:36:23.701 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:36:23.702 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":171,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:36:20.740860+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:04.083659+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:06.619351+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:12.810102+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:01.210525+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:23.083007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:16.458045+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:20.804823+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:47.646442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:03.525312+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:17.653294+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:20.448915+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:19.908684+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:09.324095+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:34:58.559090+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:05.769648+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:36:26.721 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:36:27.036 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:36:27.036 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":173,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:36:24.775471+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:04.083659+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:06.619351+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:12.810102+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:01.210525+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:23.083007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:16.458045+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:20.804823+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:47.646442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:03.525312+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:17.653294+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:20.448915+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:19.908684+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:09.324095+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:22.781495+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:24.403658+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:36:29.257 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~214s 2023-11-05T08:36:30.055 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:36:30.367 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:36:30.367 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":175,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:36:28.802631+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:04.083659+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:28.371822+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:12.810102+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.163932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:23.083007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:16.458045+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:20.804823+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:47.646442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.724147+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:17.653294+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:20.448915+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:19.908684+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:09.324095+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:22.781495+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:24.403658+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:36:33.387 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:36:33.694 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:36:33.694 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":176,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:36:31.824386+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:30.855831+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:28.371822+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:12.810102+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.163932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:23.083007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:16.458045+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:20.804823+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:47.646442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.724147+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:17.653294+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:20.448915+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:19.908684+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:09.324095+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:22.781495+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:24.403658+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:36:36.272 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~221s 2023-11-05T08:36:36.712 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:36:37.023 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:36:37.024 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":178,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:36:36.713957+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:30.855831+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:28.371822+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.495499+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.163932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:23.083007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:16.458045+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.734825+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:47.646442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.724147+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:17.653294+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:20.448915+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:19.908684+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:09.324095+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:22.781495+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:24.403658+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:36:40.043 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:36:40.353 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:36:40.353 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":178,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:36:36.713957+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:30.855831+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:28.371822+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.495499+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.163932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:23.083007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:16.458045+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.734825+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:47.646442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.724147+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:17.653294+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:20.448915+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:19.908684+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:09.324095+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:22.781495+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:24.403658+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:36:43.288 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~228s 2023-11-05T08:36:43.371 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:36:43.683 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:36:43.684 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":178,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:36:36.713957+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:30.855831+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:28.371822+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.495499+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.163932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:23.083007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:16.458045+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.734825+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:47.646442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.724147+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:17.653294+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:20.448915+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:19.908684+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:09.324095+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:22.781495+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:24.403658+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:36:46.703 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:36:47.016 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:36:47.017 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":178,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:36:36.713957+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:30.855831+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:28.371822+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.495499+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.163932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:23.083007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:16.458045+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.734825+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:47.646442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.724147+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:17.653294+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:20.448915+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:19.908684+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:09.324095+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:22.781495+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:24.403658+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:36:50.039 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:36:50.304 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~235s 2023-11-05T08:36:50.345 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:36:50.345 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":179,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:36:47.813671+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:30.855831+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:28.371822+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.495499+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.163932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:47.545638+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:16.458045+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.734825+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:47.646442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.724147+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:17.653294+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:20.448915+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:19.908684+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:09.324095+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:22.781495+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:24.403658+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:36:53.364 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:36:53.670 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:36:53.670 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":179,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:36:47.813671+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:30.855831+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:28.371822+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.495499+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.163932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:47.545638+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:16.458045+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.734825+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:47.646442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.724147+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:17.653294+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:20.448915+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:19.908684+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:09.324095+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:22.781495+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:24.403658+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:36:56.690 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:36:56.999 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:36:56.999 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":179,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:36:47.813671+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:30.855831+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:28.371822+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.495499+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.163932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:47.545638+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:16.458045+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.734825+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:47.646442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.724147+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:17.653294+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:20.448915+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:19.908684+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:09.324095+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:22.781495+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:24.403658+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:36:57.319 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~242s 2023-11-05T08:37:00.017 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:37:00.323 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:37:00.324 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":179,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:36:47.813671+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:30.855831+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:28.371822+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.495499+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.163932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:47.545638+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:16.458045+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.734825+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:47.646442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.724147+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:17.653294+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:20.448915+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:19.908684+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:09.324095+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:22.781495+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:24.403658+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:37:03.341 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:37:03.650 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:37:03.650 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":179,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:36:47.813671+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:30.855831+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:28.371822+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.495499+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.163932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:47.545638+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:16.458045+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.734825+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:47.646442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.724147+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:17.653294+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:20.448915+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:19.908684+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:09.324095+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:22.781495+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:24.403658+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:37:04.334 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~249s 2023-11-05T08:37:06.670 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:37:06.980 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:37:06.980 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":179,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:36:47.813671+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:30.855831+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:28.371822+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.495499+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.163932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:47.545638+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:16.458045+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.734825+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:35:47.646442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.724147+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:17.653294+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:20.448915+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:19.908684+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:09.324095+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:22.781495+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:24.403658+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:37:09.999 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:37:10.305 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:37:10.305 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":180,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:37:07.823206+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:30.855831+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:28.371822+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.495499+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.163932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:47.545638+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:16.458045+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.734825+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:07.524366+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.724147+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:17.653294+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:20.448915+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:19.908684+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:09.324095+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:22.781495+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:24.403658+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:37:11.352 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~256s 2023-11-05T08:37:13.324 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:37:13.631 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:37:13.632 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":180,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:37:07.823206+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:30.855831+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:28.371822+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.495499+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.163932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:47.545638+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:16.458045+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.734825+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:07.524366+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.724147+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:17.653294+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:20.448915+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:19.908684+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:09.324095+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:22.781495+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:24.403658+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:37:16.650 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:37:16.964 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:37:16.964 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":181,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:37:13.862043+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:30.855831+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:28.371822+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.495499+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.163932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:47.545638+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:16.458045+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.734825+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:07.524366+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.724147+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:17.653294+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:20.448915+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:19.908684+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:13.578948+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:22.781495+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:24.403658+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:37:18.372 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~263s 2023-11-05T08:37:19.982 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:37:20.292 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:37:20.293 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":181,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:37:13.862043+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:30.855831+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:28.371822+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.495499+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.163932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:47.545638+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:16.458045+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.734825+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:07.524366+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.724147+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:17.653294+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:20.448915+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:19.908684+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:13.578948+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:22.781495+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:24.403658+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:37:23.312 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:37:23.622 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:37:23.622 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":182,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:37:21.906284+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:30.855831+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:28.371822+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.495499+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.163932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:47.545638+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:21.322237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.734825+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:07.524366+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.724147+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:17.653294+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:20.448915+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:19.908684+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:13.578948+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:22.781495+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:24.403658+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:37:25.389 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~270s 2023-11-05T08:37:26.641 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:37:26.957 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:37:26.957 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":182,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:37:21.906284+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:30.855831+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:28.371822+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.495499+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.163932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:47.545638+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:21.322237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.734825+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:07.524366+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.724147+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:17.653294+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:20.448915+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:19.908684+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:13.578948+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:22.781495+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:24.403658+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:37:29.977 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:37:30.286 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:37:30.287 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":182,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:37:21.906284+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:30.855831+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:28.371822+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.495499+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.163932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:47.545638+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:21.322237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.734825+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:07.524366+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.724147+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:17.653294+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:20.448915+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:19.908684+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:13.578948+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:22.781495+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:24.403658+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:37:32.406 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~277s 2023-11-05T08:37:33.306 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:37:33.618 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:37:33.618 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":182,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:37:21.906284+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:30.855831+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:28.371822+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.495499+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.163932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:47.545638+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:21.322237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.734825+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:07.524366+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.724147+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:17.653294+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:20.448915+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:19.908684+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:13.578948+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:22.781495+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:24.403658+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:37:36.638 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:37:36.957 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:37:36.958 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":182,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:37:21.906284+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:30.855831+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:28.371822+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.495499+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.163932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:47.545638+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:21.322237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.734825+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:07.524366+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.724147+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:17.653294+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:20.448915+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:19.908684+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:13.578948+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:22.781495+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:24.403658+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:37:39.424 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~284s 2023-11-05T08:37:39.977 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:37:40.285 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:37:40.285 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":184,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:37:38.840116+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:30.855831+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:28.371822+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.495499+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.163932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:47.545638+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:21.322237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.734825+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:07.524366+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.724147+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:38.716590+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:20.448915+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:19.908684+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:13.578948+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:22.781495+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:36.972533+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:37:43.304 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:37:43.614 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:37:43.614 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":185,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:37:41.862725+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:30.855831+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:28.371822+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.495499+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.163932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:47.545638+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:21.322237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.734825+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:07.524366+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.724147+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:38.716590+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:41.020270+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:40.923342+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:13.578948+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:22.781495+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:36.972533+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:37:46.441 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~291s 2023-11-05T08:37:46.634 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:37:46.941 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:37:46.942 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":185,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:37:41.862725+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:30.855831+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:28.371822+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.495499+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.163932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:47.545638+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:21.322237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.734825+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:07.524366+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.724147+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:38.716590+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:41.020270+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:40.923342+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:13.578948+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:22.781495+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:36.972533+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:37:49.962 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:37:50.270 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:37:50.270 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":187,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:37:48.844041+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:30.855831+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:47.397970+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.495499+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:48.218324+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:47.545638+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:21.322237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.734825+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:07.524366+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.724147+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:38.716590+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:41.020270+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:40.923342+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:13.578948+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:22.781495+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:36.972533+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:37:53.290 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:37:53.459 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~298s 2023-11-05T08:37:53.601 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:37:53.602 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":190,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:37:52.877901+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:51.825640+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:47.397970+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.495499+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:48.218324+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:52.107453+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:21.322237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.734825+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:07.524366+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:27.724147+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:38.716590+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:41.020270+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:40.923342+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:13.578948+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:50.500167+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:36.972533+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:37:56.622 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:37:56.931 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:37:56.932 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":192,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:37:55.899907+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:51.825640+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:47.397970+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:53.227768+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:48.218324+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:52.107453+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:21.322237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.734825+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:07.524366+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:55.292885+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:38.716590+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:41.020270+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:40.923342+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:13.578948+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:50.500167+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:36.972533+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:37:59.951 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:38:00.270 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-05T08:38:00.271 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":192,"fsid":"8f1fa9b5-06de-4da7-985f-30d4ad103e2d","created":"2023-11-05T08:32:15.953286+0000","modified":"2023-11-05T08:37:55.899907+0000","last_up_change":"2023-11-05T08:33:46.664317+0000","last_in_change":"2023-11-05T08:32:20.284340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-05T08:32:26.748874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-05T08:32:30.772862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3626fa17-5bde-458d-96fb-f5e70ac6edd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":30,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6803","nonce":1948090427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6805","nonce":1948090427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6809","nonce":1948090427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1948090427},{"type":"v1","addr":"172.21.15.2:6807","nonce":1948090427}]},"public_addr":"172.21.15.2:6803/1948090427","cluster_addr":"172.21.15.2:6805/1948090427","heartbeat_back_addr":"172.21.15.2:6809/1948090427","heartbeat_front_addr":"172.21.15.2:6807/1948090427","state":["exists"]},{"osd":1,"uuid":"8f524504-91f5-44f3-afbd-69dfc19f03d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6818","nonce":121537835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6822","nonce":121537835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6827","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6829","nonce":121537835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":121537835},{"type":"v1","addr":"172.21.15.53:6826","nonce":121537835}]},"public_addr":"172.21.15.53:6818/121537835","cluster_addr":"172.21.15.53:6822/121537835","heartbeat_back_addr":"172.21.15.53:6829/121537835","heartbeat_front_addr":"172.21.15.53:6826/121537835","state":["exists","up"]},{"osd":2,"uuid":"696f71d0-a98b-46b5-bdc0-0da3d6065469","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6801","nonce":1463424062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6803","nonce":1463424062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6807","nonce":1463424062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1463424062},{"type":"v1","addr":"172.21.15.69:6805","nonce":1463424062}]},"public_addr":"172.21.15.69:6801/1463424062","cluster_addr":"172.21.15.69:6803/1463424062","heartbeat_back_addr":"172.21.15.69:6807/1463424062","heartbeat_front_addr":"172.21.15.69:6805/1463424062","state":["exists","up"]},{"osd":3,"uuid":"fc979d30-659b-47f4-b06b-97a4d41c2e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":43,"up_from":93,"up_thru":154,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6801","nonce":2085535590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6803","nonce":2085535590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6807","nonce":2085535590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2085535590},{"type":"v1","addr":"172.21.15.90:6805","nonce":2085535590}]},"public_addr":"172.21.15.90:6801/2085535590","cluster_addr":"172.21.15.90:6803/2085535590","heartbeat_back_addr":"172.21.15.90:6807/2085535590","heartbeat_front_addr":"172.21.15.90:6805/2085535590","state":["exists","up"]},{"osd":4,"uuid":"2ed87cb9-9ced-46fd-8346-3f47709bb69e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6811","nonce":2916800887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6813","nonce":2916800887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6817","nonce":2916800887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2916800887},{"type":"v1","addr":"172.21.15.2:6815","nonce":2916800887}]},"public_addr":"172.21.15.2:6811/2916800887","cluster_addr":"172.21.15.2:6813/2916800887","heartbeat_back_addr":"172.21.15.2:6817/2916800887","heartbeat_front_addr":"172.21.15.2:6815/2916800887","state":["exists","up"]},{"osd":5,"uuid":"5bb59d8d-9b93-48b2-a5ee-13646b19ce50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6809","nonce":588137330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6811","nonce":588137330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6815","nonce":588137330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":588137330},{"type":"v1","addr":"172.21.15.53:6813","nonce":588137330}]},"public_addr":"172.21.15.53:6809/588137330","cluster_addr":"172.21.15.53:6811/588137330","heartbeat_back_addr":"172.21.15.53:6815/588137330","heartbeat_front_addr":"172.21.15.53:6813/588137330","state":["exists","up"]},{"osd":6,"uuid":"46cd8f39-841d-4b7b-a4ef-6ec6493eea5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6818","nonce":3720624375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6821","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6823","nonce":3720624375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6829","nonce":3720624375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3720624375},{"type":"v1","addr":"172.21.15.69:6826","nonce":3720624375}]},"public_addr":"172.21.15.69:6818/3720624375","cluster_addr":"172.21.15.69:6823/3720624375","heartbeat_back_addr":"172.21.15.69:6829/3720624375","heartbeat_front_addr":"172.21.15.69:6826/3720624375","state":["exists","up"]},{"osd":7,"uuid":"c98d93d9-c1e5-4ac0-8f71-9fc1d2767963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6809","nonce":2028724524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6811","nonce":2028724524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6815","nonce":2028724524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2028724524},{"type":"v1","addr":"172.21.15.90:6813","nonce":2028724524}]},"public_addr":"172.21.15.90:6809/2028724524","cluster_addr":"172.21.15.90:6811/2028724524","heartbeat_back_addr":"172.21.15.90:6815/2028724524","heartbeat_front_addr":"172.21.15.90:6813/2028724524","state":["exists","up"]},{"osd":8,"uuid":"fcb20ed4-d041-4ef8-bf17-724a43758e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6820","nonce":1795116288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6823","nonce":1795116288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6830","nonce":1795116288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1795116288},{"type":"v1","addr":"172.21.15.2:6827","nonce":1795116288}]},"public_addr":"172.21.15.2:6820/1795116288","cluster_addr":"172.21.15.2:6823/1795116288","heartbeat_back_addr":"172.21.15.2:6830/1795116288","heartbeat_front_addr":"172.21.15.2:6827/1795116288","state":["exists","up"]},{"osd":9,"uuid":"f85ae771-21d9-4974-a203-7ff48225c8a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6801","nonce":1366183694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6803","nonce":1366183694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6807","nonce":1366183694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1366183694},{"type":"v1","addr":"172.21.15.53:6805","nonce":1366183694}]},"public_addr":"172.21.15.53:6801/1366183694","cluster_addr":"172.21.15.53:6803/1366183694","heartbeat_back_addr":"172.21.15.53:6807/1366183694","heartbeat_front_addr":"172.21.15.53:6805/1366183694","state":["exists","up"]},{"osd":10,"uuid":"496442a8-9bc1-4b66-a8bc-294488f6f089","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6813","nonce":2763568647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6822","nonce":2763568647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6831","nonce":2763568647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":2763568647},{"type":"v1","addr":"172.21.15.69:6827","nonce":2763568647}]},"public_addr":"172.21.15.69:6813/2763568647","cluster_addr":"172.21.15.69:6822/2763568647","heartbeat_back_addr":"172.21.15.69:6831/2763568647","heartbeat_front_addr":"172.21.15.69:6827/2763568647","state":["exists","up"]},{"osd":11,"uuid":"5680ea76-e74b-43e8-8933-39204c641f00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6817","nonce":2584956675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6819","nonce":2584956675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6823","nonce":2584956675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2584956675},{"type":"v1","addr":"172.21.15.90:6821","nonce":2584956675}]},"public_addr":"172.21.15.90:6817/2584956675","cluster_addr":"172.21.15.90:6819/2584956675","heartbeat_back_addr":"172.21.15.90:6823/2584956675","heartbeat_front_addr":"172.21.15.90:6821/2584956675","state":["exists","up"]},{"osd":12,"uuid":"2bfa4d5a-f32e-4f13-9940-f139dfd7bdaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6821","nonce":205377094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6826","nonce":205377094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6833","nonce":205377094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6829","nonce":205377094},{"type":"v1","addr":"172.21.15.2:6831","nonce":205377094}]},"public_addr":"172.21.15.2:6821/205377094","cluster_addr":"172.21.15.2:6826/205377094","heartbeat_back_addr":"172.21.15.2:6833/205377094","heartbeat_front_addr":"172.21.15.2:6831/205377094","state":["exists","up"]},{"osd":13,"uuid":"356e7e3c-675d-4e78-ba4a-1d993093dc20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6819","nonce":714840017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6823","nonce":714840017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6831","nonce":714840017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6825","nonce":714840017},{"type":"v1","addr":"172.21.15.53:6828","nonce":714840017}]},"public_addr":"172.21.15.53:6819/714840017","cluster_addr":"172.21.15.53:6823/714840017","heartbeat_back_addr":"172.21.15.53:6831/714840017","heartbeat_front_addr":"172.21.15.53:6828/714840017","state":["exists","up"]},{"osd":14,"uuid":"1c49bef4-a6a5-44eb-91fa-08dc1fcf9e1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6809","nonce":567567051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6812","nonce":567567051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6817","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6820","nonce":567567051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":567567051},{"type":"v1","addr":"172.21.15.69:6816","nonce":567567051}]},"public_addr":"172.21.15.69:6809/567567051","cluster_addr":"172.21.15.69:6812/567567051","heartbeat_back_addr":"172.21.15.69:6820/567567051","heartbeat_front_addr":"172.21.15.69:6816/567567051","state":["exists","up"]},{"osd":15,"uuid":"4a37e23e-e1ec-4444-b87b-a9897a448f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6825","nonce":739151808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6827","nonce":739151808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6831","nonce":739151808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":739151808},{"type":"v1","addr":"172.21.15.90:6829","nonce":739151808}]},"public_addr":"172.21.15.90:6825/739151808","cluster_addr":"172.21.15.90:6827/739151808","heartbeat_back_addr":"172.21.15.90:6831/739151808","heartbeat_front_addr":"172.21.15.90:6829/739151808","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:32:25.637689+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:51.825640+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:47.397970+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:53.227768+0000","dead_epoch":45},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:48.218324+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:52.107453+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:21.322237+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:36:35.734825+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:07.524366+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:55.292885+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:38.716590+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:41.020270+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:40.923342+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:13.578948+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:50.500167+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-05T08:37:36.972533+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":2},{"begin":5,"length":4},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":19,"length":1},{"begin":21,"length":1},{"begin":23,"length":2},{"begin":26,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":43,"length":2},{"begin":46,"length":3},{"begin":52,"length":1},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":65,"length":2},{"begin":68,"length":3},{"begin":73,"length":1},{"begin":77,"length":2},{"begin":80,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-05T08:38:00.475 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~305s 2023-11-05T08:38:00.475 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-11-05T08:38:02.493 INFO:tasks.ceph.osd.4:Sent signal 15 2023-11-05T08:38:02.493 INFO:tasks.ceph.osd.8:Sent signal 15 2023-11-05T08:38:02.494 INFO:tasks.ceph.osd.12:Sent signal 15 2023-11-05T08:38:02.494 INFO:tasks.ceph.osd.1:Sent signal 15 2023-11-05T08:38:02.495 INFO:tasks.ceph.osd.5:Sent signal 15 2023-11-05T08:38:02.495 INFO:tasks.ceph.osd.9:Sent signal 15 2023-11-05T08:38:02.495 INFO:tasks.ceph.osd.13:Sent signal 15 2023-11-05T08:38:02.496 INFO:tasks.ceph.osd.2:Sent signal 15 2023-11-05T08:38:02.496 INFO:tasks.ceph.osd.6:Sent signal 15 2023-11-05T08:38:02.496 INFO:tasks.ceph.osd.10:Sent signal 15 2023-11-05T08:38:02.497 INFO:tasks.ceph.osd.14:Sent signal 15 2023-11-05T08:38:02.497 INFO:tasks.ceph.osd.3:Sent signal 15 2023-11-05T08:38:02.497 INFO:tasks.ceph.osd.7:Sent signal 15 2023-11-05T08:38:02.498 INFO:tasks.ceph.osd.11:Sent signal 15 2023-11-05T08:38:02.498 INFO:tasks.ceph.osd.15:Sent signal 15 2023-11-05T08:38:02.498 INFO:tasks.ceph.mon.a:Sent signal 15 2023-11-05T08:38:02.499 INFO:tasks.ceph.mon.b:Sent signal 15 2023-11-05T08:38:02.499 INFO:tasks.ceph.mon.c:Sent signal 15 2023-11-05T08:38:02.500 INFO:tasks.ceph.mgr.y:Sent signal 15 2023-11-05T08:38:02.500 INFO:tasks.ceph.mgr.x:Sent signal 15 2023-11-05T08:38:02.501 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:38:02.498+0000 7f168a3b6640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35683) UID: 0 2023-11-05T08:38:02.502 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:38:02.498+0000 7f168a3b6640 -1 osd.2 192 *** Got signal Terminated *** 2023-11-05T08:38:02.502 INFO:tasks.ceph.osd.2.smithi069.stderr:2023-11-05T08:38:02.498+0000 7f168a3b6640 -1 osd.2 192 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-05T08:38:02.502 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:38:02.497+0000 7f55c564d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35548) UID: 0 2023-11-05T08:38:02.502 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:38:02.497+0000 7f55c564d640 -1 osd.1 192 *** Got signal Terminated *** 2023-11-05T08:38:02.503 INFO:tasks.ceph.osd.1.smithi053.stderr:2023-11-05T08:38:02.497+0000 7f55c564d640 -1 osd.1 192 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-05T08:38:02.503 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:38:02.492+0000 7f8cbc47c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35668) UID: 0 2023-11-05T08:38:02.503 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:38:02.492+0000 7f8cbc47c640 -1 osd.4 192 *** Got signal Terminated *** 2023-11-05T08:38:02.503 INFO:tasks.ceph.osd.4.smithi002.stderr:2023-11-05T08:38:02.492+0000 7f8cbc47c640 -1 osd.4 192 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-05T08:38:02.504 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-11-05T08:38:02.496+0000 7f72b2c5f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 42525) UID: 0 2023-11-05T08:38:02.504 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-11-05T08:38:02.496+0000 7f72b2c5f640 -1 osd.3 192 *** Got signal Terminated *** 2023-11-05T08:38:02.504 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-11-05T08:38:02.496+0000 7f72b2c5f640 -1 osd.3 192 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-05T08:38:02.505 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:38:02.498+0000 7fa106acd640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35686) UID: 0 2023-11-05T08:38:02.505 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:38:02.498+0000 7fa106acd640 -1 osd.6 192 *** Got signal Terminated *** 2023-11-05T08:38:02.505 INFO:tasks.ceph.osd.6.smithi069.stderr:2023-11-05T08:38:02.498+0000 7fa106acd640 -1 osd.6 192 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-05T08:38:02.505 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:38:02.497+0000 7f429c947640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35551) UID: 0 2023-11-05T08:38:02.506 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:38:02.497+0000 7f429c947640 -1 osd.5 192 *** Got signal Terminated *** 2023-11-05T08:38:02.506 INFO:tasks.ceph.osd.5.smithi053.stderr:2023-11-05T08:38:02.497+0000 7f429c947640 -1 osd.5 192 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-05T08:38:02.506 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:38:02.492+0000 7fdc5cd58640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35670) UID: 0 2023-11-05T08:38:02.507 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:38:02.492+0000 7fdc5cd58640 -1 osd.8 192 *** Got signal Terminated *** 2023-11-05T08:38:02.507 INFO:tasks.ceph.osd.8.smithi002.stderr:2023-11-05T08:38:02.492+0000 7fdc5cd58640 -1 osd.8 192 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-05T08:38:02.507 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:38:02.496+0000 7f00940d2640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35979) UID: 0 2023-11-05T08:38:02.507 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:38:02.496+0000 7f00940d2640 -1 osd.7 192 *** Got signal Terminated *** 2023-11-05T08:38:02.507 INFO:tasks.ceph.osd.7.smithi090.stderr:2023-11-05T08:38:02.496+0000 7f00940d2640 -1 osd.7 192 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-05T08:38:02.508 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:38:02.498+0000 7ff0d5267640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35690) UID: 0 2023-11-05T08:38:02.508 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:38:02.498+0000 7ff0d5267640 -1 osd.14 192 *** Got signal Terminated *** 2023-11-05T08:38:02.508 INFO:tasks.ceph.osd.14.smithi069.stderr:2023-11-05T08:38:02.498+0000 7ff0d5267640 -1 osd.14 192 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-05T08:38:02.509 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:38:02.497+0000 7f1c6dde4640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35553) UID: 0 2023-11-05T08:38:02.509 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:38:02.497+0000 7f1c6dde4640 -1 osd.9 192 *** Got signal Terminated *** 2023-11-05T08:38:02.509 INFO:tasks.ceph.osd.9.smithi053.stderr:2023-11-05T08:38:02.497+0000 7f1c6dde4640 -1 osd.9 192 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-05T08:38:02.509 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:38:02.496+0000 7fda6d004640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35672) UID: 0 2023-11-05T08:38:02.510 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:38:02.496+0000 7fda6d004640 -1 osd.12 192 *** Got signal Terminated *** 2023-11-05T08:38:02.510 INFO:tasks.ceph.osd.12.smithi002.stderr:2023-11-05T08:38:02.496+0000 7fda6d004640 -1 osd.12 192 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-05T08:38:02.510 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:38:02.496+0000 7ffbcd342640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35982) UID: 0 2023-11-05T08:38:02.511 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:38:02.496+0000 7ffbcd342640 -1 osd.11 192 *** Got signal Terminated *** 2023-11-05T08:38:02.511 INFO:tasks.ceph.osd.11.smithi090.stderr:2023-11-05T08:38:02.496+0000 7ffbcd342640 -1 osd.11 192 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-05T08:38:02.511 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:38:02.498+0000 7f226237e640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35688) UID: 0 2023-11-05T08:38:02.511 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:38:02.498+0000 7f226237e640 -1 osd.10 192 *** Got signal Terminated *** 2023-11-05T08:38:02.511 INFO:tasks.ceph.osd.10.smithi069.stderr:2023-11-05T08:38:02.498+0000 7f226237e640 -1 osd.10 192 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-05T08:38:02.512 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:38:02.497+0000 7fcd95813640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35555) UID: 0 2023-11-05T08:38:02.512 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:38:02.497+0000 7fcd95813640 -1 osd.13 192 *** Got signal Terminated *** 2023-11-05T08:38:02.512 INFO:tasks.ceph.osd.13.smithi053.stderr:2023-11-05T08:38:02.497+0000 7fcd95813640 -1 osd.13 192 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-05T08:38:02.512 INFO:tasks.ceph.mon.a.smithi002.stderr:2023-11-05T08:38:02.500+0000 7ff18b8cc640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 35553) UID: 0 2023-11-05T08:38:02.513 INFO:tasks.ceph.mon.a.smithi002.stderr:2023-11-05T08:38:02.500+0000 7ff18b8cc640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-11-05T08:38:02.513 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:38:02.496+0000 7fc10ec39640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35984) UID: 0 2023-11-05T08:38:02.513 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:38:02.496+0000 7fc10ec39640 -1 osd.15 192 *** Got signal Terminated *** 2023-11-05T08:38:02.513 INFO:tasks.ceph.osd.15.smithi090.stderr:2023-11-05T08:38:02.496+0000 7fc10ec39640 -1 osd.15 192 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-05T08:38:02.514 INFO:tasks.ceph.mon.c.smithi069.stderr:2023-11-05T08:38:02.502+0000 7fbc57fc4640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster ceph -i c (PID: 35631) UID: 0 2023-11-05T08:38:02.514 INFO:tasks.ceph.mon.c.smithi069.stderr:2023-11-05T08:38:02.502+0000 7fbc57fc4640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2023-11-05T08:38:02.514 INFO:tasks.ceph.mon.b.smithi053.stderr:2023-11-05T08:38:02.501+0000 7f3d98fd8640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster ceph -i b (PID: 35495) UID: 0 2023-11-05T08:38:02.514 INFO:tasks.ceph.mon.b.smithi053.stderr:2023-11-05T08:38:02.501+0000 7f3d98fd8640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2023-11-05T08:38:02.515 INFO:tasks.ceph.mgr.y.smithi002.stderr:2023-11-05T08:38:02.500+0000 7fddc20e5640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mgr -f --cluster ceph -i y (PID: 35558) UID: 0 2023-11-05T08:38:02.515 INFO:tasks.ceph.mgr.y.smithi002.stderr:2023-11-05T08:38:02.500+0000 7fddc20e5640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-11-05T08:38:02.702 INFO:tasks.ceph.mgr.x.smithi090.stderr:daemon-helper: command crashed with signal 15 2023-11-05T08:38:03.290 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-05T08:40:03.309 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-05T08:40:03.311 ERROR:teuthology.run_tasks:Manager failed: thrashosds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/thrashosds.py", line 215, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2792, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2782, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2545, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2537, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi002 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2023-11-05T08:40:03.311 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-11-05T08:40:03.321 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-05T08:40:03.322 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-05T08:42:03.341 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-05T08:42:03.342 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/thrashosds.py", line 215, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2792, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2782, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2545, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2537, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi002 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi002 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-11-05T08:42:03.343 INFO:teuthology.misc:Shutting down mds daemons... 2023-11-05T08:42:03.344 INFO:teuthology.misc:Shutting down osd daemons... 2023-11-05T08:42:03.344 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-11-05T08:42:03.345 INFO:teuthology.orchestra.run:waiting for 300 2023-11-05T08:42:03.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-05T08:42:03.346 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/thrashosds.py", line 215, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2792, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2782, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2545, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2537, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi002 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi002 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi002 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0' 2023-11-05T08:42:03.347 INFO:tasks.ceph.osd.0:Stopped 2023-11-05T08:42:03.347 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-11-05T08:42:03.348 INFO:teuthology.orchestra.run:waiting for 300 2023-11-05T08:42:03.349 INFO:tasks.ceph.osd.4:Stopped 2023-11-05T08:42:03.349 DEBUG:tasks.ceph.osd.8:waiting for process to exit 2023-11-05T08:42:03.350 INFO:teuthology.orchestra.run:waiting for 300 2023-11-05T08:42:03.350 INFO:tasks.ceph.osd.8:Stopped 2023-11-05T08:42:03.351 DEBUG:tasks.ceph.osd.12:waiting for process to exit 2023-11-05T08:42:03.351 INFO:teuthology.orchestra.run:waiting for 300 2023-11-05T08:42:03.352 INFO:tasks.ceph.osd.12:Stopped 2023-11-05T08:42:03.352 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-11-05T08:42:03.353 INFO:teuthology.orchestra.run:waiting for 300 2023-11-05T08:42:03.353 INFO:tasks.ceph.osd.1:Stopped 2023-11-05T08:42:03.354 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-11-05T08:42:03.355 INFO:teuthology.orchestra.run:waiting for 300 2023-11-05T08:42:03.355 INFO:tasks.ceph.osd.5:Stopped 2023-11-05T08:42:03.356 DEBUG:tasks.ceph.osd.9:waiting for process to exit 2023-11-05T08:42:03.356 INFO:teuthology.orchestra.run:waiting for 300 2023-11-05T08:42:03.357 INFO:tasks.ceph.osd.9:Stopped 2023-11-05T08:42:03.357 DEBUG:tasks.ceph.osd.13:waiting for process to exit 2023-11-05T08:42:03.358 INFO:teuthology.orchestra.run:waiting for 300 2023-11-05T08:42:03.358 INFO:tasks.ceph.osd.13:Stopped 2023-11-05T08:42:03.360 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-11-05T08:42:03.360 INFO:teuthology.orchestra.run:waiting for 300 2023-11-05T08:42:03.361 INFO:tasks.ceph.osd.2:Stopped 2023-11-05T08:42:03.362 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-11-05T08:42:03.363 INFO:teuthology.orchestra.run:waiting for 300 2023-11-05T08:42:03.364 INFO:tasks.ceph.osd.6:Stopped 2023-11-05T08:42:03.364 DEBUG:tasks.ceph.osd.10:waiting for process to exit 2023-11-05T08:42:03.365 INFO:teuthology.orchestra.run:waiting for 300 2023-11-05T08:42:03.365 INFO:tasks.ceph.osd.10:Stopped 2023-11-05T08:42:03.366 DEBUG:tasks.ceph.osd.14:waiting for process to exit 2023-11-05T08:42:03.366 INFO:teuthology.orchestra.run:waiting for 300 2023-11-05T08:42:03.367 INFO:tasks.ceph.osd.14:Stopped 2023-11-05T08:42:03.367 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-11-05T08:42:03.368 INFO:teuthology.orchestra.run:waiting for 300 2023-11-05T08:42:03.368 INFO:tasks.ceph.osd.3:Stopped 2023-11-05T08:42:03.369 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-11-05T08:42:03.369 INFO:teuthology.orchestra.run:waiting for 300 2023-11-05T08:42:03.370 INFO:tasks.ceph.osd.7:Stopped 2023-11-05T08:42:03.371 DEBUG:tasks.ceph.osd.11:waiting for process to exit 2023-11-05T08:42:03.371 INFO:teuthology.orchestra.run:waiting for 300 2023-11-05T08:42:03.372 INFO:tasks.ceph.osd.11:Stopped 2023-11-05T08:42:03.372 DEBUG:tasks.ceph.osd.15:waiting for process to exit 2023-11-05T08:42:03.373 INFO:teuthology.orchestra.run:waiting for 300 2023-11-05T08:42:03.373 INFO:tasks.ceph.osd.15:Stopped 2023-11-05T08:42:03.374 INFO:teuthology.misc:Shutting down mgr daemons... 2023-11-05T08:42:03.375 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-11-05T08:42:03.376 INFO:teuthology.orchestra.run:waiting for 300 2023-11-05T08:42:03.377 INFO:tasks.ceph.mgr.y:Stopped 2023-11-05T08:42:03.378 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-11-05T08:42:03.378 INFO:teuthology.orchestra.run:waiting for 300 2023-11-05T08:42:03.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-05T08:42:03.379 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/thrashosds.py", line 215, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2792, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2782, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2545, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2537, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi002 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi002 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x' 2023-11-05T08:42:03.380 INFO:tasks.ceph.mgr.x:Stopped 2023-11-05T08:42:03.381 INFO:teuthology.misc:Shutting down mon daemons... 2023-11-05T08:42:03.381 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-11-05T08:42:03.382 INFO:teuthology.orchestra.run:waiting for 300 2023-11-05T08:42:03.382 INFO:tasks.ceph.mon.a:Stopped 2023-11-05T08:42:03.383 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-11-05T08:42:03.383 INFO:teuthology.orchestra.run:waiting for 300 2023-11-05T08:42:03.384 INFO:tasks.ceph.mon.b:Stopped 2023-11-05T08:42:03.384 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-11-05T08:42:03.385 INFO:teuthology.orchestra.run:waiting for 300 2023-11-05T08:42:03.386 INFO:tasks.ceph.mon.c:Stopped 2023-11-05T08:42:03.386 INFO:tasks.ceph:Checking cluster log for badness... 2023-11-05T08:42:03.387 DEBUG:teuthology.orchestra.run.smithi002:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'slow request' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | head -n 1 2023-11-05T08:42:03.404 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi002.front.sepia.ceph.com 2023-11-05T08:42:03.405 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-11-05T08:42:03.632 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi002.front.sepia.ceph.com 2023-11-05T08:42:03.633 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-11-05T08:42:03.826 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-8 on ubuntu@smithi002.front.sepia.ceph.com 2023-11-05T08:42:03.826 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-8 2023-11-05T08:42:04.008 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-12 on ubuntu@smithi002.front.sepia.ceph.com 2023-11-05T08:42:04.009 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-12 2023-11-05T08:42:04.176 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi053.front.sepia.ceph.com 2023-11-05T08:42:04.177 DEBUG:teuthology.orchestra.run.smithi053:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-11-05T08:42:04.258 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi053.front.sepia.ceph.com 2023-11-05T08:42:04.259 DEBUG:teuthology.orchestra.run.smithi053:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-11-05T08:42:04.442 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-9 on ubuntu@smithi053.front.sepia.ceph.com 2023-11-05T08:42:04.443 DEBUG:teuthology.orchestra.run.smithi053:> sync && sudo umount -f /var/lib/ceph/osd/ceph-9 2023-11-05T08:42:04.648 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-13 on ubuntu@smithi053.front.sepia.ceph.com 2023-11-05T08:42:04.649 DEBUG:teuthology.orchestra.run.smithi053:> sync && sudo umount -f /var/lib/ceph/osd/ceph-13 2023-11-05T08:42:04.817 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi069.front.sepia.ceph.com 2023-11-05T08:42:04.817 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-11-05T08:42:04.963 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi069.front.sepia.ceph.com 2023-11-05T08:42:04.964 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-11-05T08:42:05.121 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-10 on ubuntu@smithi069.front.sepia.ceph.com 2023-11-05T08:42:05.121 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/ceph-10 2023-11-05T08:42:05.293 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-14 on ubuntu@smithi069.front.sepia.ceph.com 2023-11-05T08:42:05.293 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/ceph-14 2023-11-05T08:42:05.478 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi090.front.sepia.ceph.com 2023-11-05T08:42:05.479 DEBUG:teuthology.orchestra.run.smithi090:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-11-05T08:42:05.588 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi090.front.sepia.ceph.com 2023-11-05T08:42:05.588 DEBUG:teuthology.orchestra.run.smithi090:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-11-05T08:42:05.748 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-11 on ubuntu@smithi090.front.sepia.ceph.com 2023-11-05T08:42:05.749 DEBUG:teuthology.orchestra.run.smithi090:> sync && sudo umount -f /var/lib/ceph/osd/ceph-11 2023-11-05T08:42:05.956 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-15 on ubuntu@smithi090.front.sepia.ceph.com 2023-11-05T08:42:05.957 DEBUG:teuthology.orchestra.run.smithi090:> sync && sudo umount -f /var/lib/ceph/osd/ceph-15 2023-11-05T08:42:06.150 INFO:tasks.ceph:Archiving mon data... 2023-11-05T08:42:06.156 DEBUG:teuthology.misc:Transferring archived files from smithi002:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/benhanokh-2023-10-31_16:46:33-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442398/data/mon.a.tgz 2023-11-05T08:42:06.157 DEBUG:teuthology.orchestra.run.smithi002:> mktemp 2023-11-05T08:42:06.171 INFO:teuthology.orchestra.run.smithi002.stdout:/tmp/tmp.f3tUduzJCo 2023-11-05T08:42:06.172 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.f3tUduzJCo 2023-11-05T08:42:06.352 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0666 /tmp/tmp.f3tUduzJCo 2023-11-05T08:42:06.420 DEBUG:teuthology.orchestra.remote:smithi002:/tmp/tmp.f3tUduzJCo is 548KB 2023-11-05T08:42:06.500 DEBUG:teuthology.orchestra.run.smithi002:> rm -fr /tmp/tmp.f3tUduzJCo 2023-11-05T08:42:06.508 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/benhanokh-2023-10-31_16:46:33-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442398/data/mon.b.tgz 2023-11-05T08:42:06.508 DEBUG:teuthology.orchestra.run.smithi053:> mktemp 2023-11-05T08:42:06.515 INFO:teuthology.orchestra.run.smithi053.stdout:/tmp/tmp.WINjvRzxM6 2023-11-05T08:42:06.515 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.WINjvRzxM6 2023-11-05T08:42:06.743 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0666 /tmp/tmp.WINjvRzxM6 2023-11-05T08:42:06.818 DEBUG:teuthology.orchestra.remote:smithi053:/tmp/tmp.WINjvRzxM6 is 770KB 2023-11-05T08:42:06.908 DEBUG:teuthology.orchestra.run.smithi053:> rm -fr /tmp/tmp.WINjvRzxM6 2023-11-05T08:42:06.921 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/benhanokh-2023-10-31_16:46:33-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442398/data/mon.c.tgz 2023-11-05T08:42:06.922 DEBUG:teuthology.orchestra.run.smithi069:> mktemp 2023-11-05T08:42:06.933 INFO:teuthology.orchestra.run.smithi069.stdout:/tmp/tmp.XrQ52grLjg 2023-11-05T08:42:06.933 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.XrQ52grLjg 2023-11-05T08:42:07.160 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0666 /tmp/tmp.XrQ52grLjg 2023-11-05T08:42:07.228 DEBUG:teuthology.orchestra.remote:smithi069:/tmp/tmp.XrQ52grLjg is 770KB 2023-11-05T08:42:07.300 DEBUG:teuthology.orchestra.run.smithi069:> rm -fr /tmp/tmp.XrQ52grLjg 2023-11-05T08:42:07.306 INFO:tasks.ceph:Cleaning ceph cluster... 2023-11-05T08:42:07.306 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-11-05T08:42:07.311 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-11-05T08:42:07.314 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-11-05T08:42:07.350 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-11-05T08:42:07.366 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-11-05T08:42:07.367 DEBUG:teuthology.orchestra.run.smithi002:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-05T08:42:07.371 DEBUG:teuthology.orchestra.run.smithi053:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-05T08:42:07.375 DEBUG:teuthology.orchestra.run.smithi069:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-05T08:42:07.390 INFO:teuthology.orchestra.run.smithi002.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-11-05T08:42:07.396 INFO:teuthology.orchestra.run.smithi053.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-11-05T08:42:07.406 DEBUG:teuthology.orchestra.run.smithi090:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-05T08:42:07.425 INFO:teuthology.orchestra.run.smithi069.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-11-05T08:42:07.451 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-11-05T08:42:07.456 INFO:tasks.ceph:Archiving crash dumps... 2023-11-05T08:42:07.459 DEBUG:teuthology.misc:Transferring archived files from smithi002:/var/lib/ceph/crash to /home/teuthworker/archive/benhanokh-2023-10-31_16:46:33-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442398/remote/smithi002/crash 2023-11-05T08:42:07.465 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-05T08:42:07.522 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/lib/ceph/crash to /home/teuthworker/archive/benhanokh-2023-10-31_16:46:33-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442398/remote/smithi053/crash 2023-11-05T08:42:07.523 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-05T08:42:07.538 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/lib/ceph/crash to /home/teuthworker/archive/benhanokh-2023-10-31_16:46:33-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442398/remote/smithi069/crash 2023-11-05T08:42:07.540 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-05T08:42:07.554 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/lib/ceph/crash to /home/teuthworker/archive/benhanokh-2023-10-31_16:46:33-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442398/remote/smithi090/crash 2023-11-05T08:42:07.556 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-05T08:42:07.570 INFO:tasks.ceph:Compressing logs... 2023-11-05T08:42:07.570 DEBUG:teuthology.orchestra.run.smithi002:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-05T08:42:07.573 DEBUG:teuthology.orchestra.run.smithi053:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-05T08:42:07.587 DEBUG:teuthology.orchestra.run.smithi069:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-05T08:42:07.587 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40573.log 2023-11-05T08:42:07.590 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45618.log 2023-11-05T08:42:07.590 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39706.log 2023-11-05T08:42:07.590 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40573.log.gz 2023-11-05T08:42:07.591 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45618.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45618.log.gz 2023-11-05T08:42:07.591 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46846.log 2023-11-05T08:42:07.591 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47002.log 2023-11-05T08:42:07.591 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39706.log.gz 2023-11-05T08:42:07.591 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46846.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.46846.log.gz --verbose 2023-11-05T08:42:07.592 INFO:teuthology.orchestra.run.smithi002.stderr: -- /var/log/ceph/ceph-client.admin.44268.log 2023-11-05T08:42:07.592 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45904.log 2023-11-05T08:42:07.592 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47002.log.gz 2023-11-05T08:42:07.592 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46294.log 2023-11-05T08:42:07.592 INFO:teuthology.orchestra.run.smithi002.stderr: 23.8% -- replaced with /var/log/ceph/ceph-client.admin.44268.log.gz 2023-11-05T08:42:07.592 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42496.log 2023-11-05T08:42:07.593 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45904.log.gz 2023-11-05T08:42:07.600 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-11-05T08:42:07.601 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-11-05T08:42:07.601 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-11-05T08:42:07.603 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46294.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-11-05T08:42:07.604 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46294.log.gz 2023-11-05T08:42:07.604 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42496.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42496.log.gzgzip 2023-11-05T08:42:07.604 INFO:teuthology.orchestra.run.smithi002.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40229.log 2023-11-05T08:42:07.604 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47760.log 2023-11-05T08:42:07.604 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47474.log 2023-11-05T08:42:07.605 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40229.log.gz 2023-11-05T08:42:07.605 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47760.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47760.log.gzgzip 2023-11-05T08:42:07.605 INFO:teuthology.orchestra.run.smithi002.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.45644.log 2023-11-05T08:42:07.605 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46898.log 2023-11-05T08:42:07.605 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47474.log.gz 2023-11-05T08:42:07.606 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43510.log 2023-11-05T08:42:07.606 INFO:teuthology.orchestra.run.smithi002.stderr: 25.5% -- replaced with /var/log/ceph/ceph-client.admin.45644.log.gz 2023-11-05T08:42:07.606 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40677.log 2023-11-05T08:42:07.606 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46898.log.gz 2023-11-05T08:42:07.606 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5/var/log/ceph/ceph-client.admin.43510.log: --verbose -- /var/log/ceph/ceph-client.admin.40221.log 2023-11-05T08:42:07.607 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43510.log.gz 2023-11-05T08:42:07.607 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41360.log 2023-11-05T08:42:07.607 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40677.log.gz 2023-11-05T08:42:07.607 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44940.log 2023-11-05T08:42:07.607 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40221.log.gz 2023-11-05T08:42:07.608 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45431.log 2023-11-05T08:42:07.608 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41360.log.gz 2023-11-05T08:42:07.608 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39663.log 2023-11-05T08:42:07.608 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44940.log.gz 2023-11-05T08:42:07.608 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40521.log 2023-11-05T08:42:07.608 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45431.log.gz 2023-11-05T08:42:07.609 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45670.log 2023-11-05T08:42:07.609 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39663.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39663.log.gz 2023-11-05T08:42:07.609 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43884.log 2023-11-05T08:42:07.609 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40521.log.gz 2023-11-05T08:42:07.609 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47500.log 2023-11-05T08:42:07.610 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45670.log.gz 2023-11-05T08:42:07.610 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41471.log 2023-11-05T08:42:07.610 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43884.log.gz 2023-11-05T08:42:07.610 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47500.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-11-05T08:42:07.610 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47500.log.gz 2023-11-05T08:42:07.611 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41279.log 2023-11-05T08:42:07.611 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41471.log.gz 2023-11-05T08:42:07.611 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41121.log 2023-11-05T08:42:07.611 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45058.log 2023-11-05T08:42:07.611 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41279.log.gz 2023-11-05T08:42:07.612 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40547.log 2023-11-05T08:42:07.612 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41121.log.gz 2023-11-05T08:42:07.612 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40246.log 2023-11-05T08:42:07.612 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45058.log.gz 2023-11-05T08:42:07.612 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45956.log 2023-11-05T08:42:07.612 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40547.log.gz 2023-11-05T08:42:07.613 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40927.log 2023-11-05T08:42:07.613 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40246.log.gz 2023-11-05T08:42:07.613 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45956.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45956.log.gz 2023-11-05T08:42:07.613 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42277.log 2023-11-05T08:42:07.613 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41592.log 2023-11-05T08:42:07.614 INFO:teuthology.orchestra.run.smithi002.stderr: 23.8% -- replaced with /var/log/ceph/ceph-client.admin.40927.log.gz 2023-11-05T08:42:07.614 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46060.log 2023-11-05T08:42:07.614 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42277.log.gz 2023-11-05T08:42:07.614 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46690.log 2023-11-05T08:42:07.614 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41592.log.gz 2023-11-05T08:42:07.615 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46060.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46060.log.gz 2023-11-05T08:42:07.615 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42140.log 2023-11-05T08:42:07.615 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46690.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46690.log.gz 2023-11-05T08:42:07.615 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47734.log 2023-11-05T08:42:07.615 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41627.log 2023-11-05T08:42:07.618 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-11-05T08:42:07.619 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.9.log 2023-11-05T08:42:07.619 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.13.log 2023-11-05T08:42:07.619 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-11-05T08:42:07.619 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-osd.9.log: /var/log/ceph/ceph-osd.13.log: /var/log/ceph/ceph-mon.b.log: 89.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-11-05T08:42:07.619 INFO:teuthology.orchestra.run.smithi053.stderr: 86.7% -- replaced with /var/log/ceph/ceph.log.gz 2023-11-05T08:42:07.622 DEBUG:teuthology.orchestra.run.smithi090:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-05T08:42:07.642 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-11-05T08:42:07.643 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-11-05T08:42:07.643 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.14.log 2023-11-05T08:42:07.643 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-11-05T08:42:07.643 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-11-05T08:42:07.643 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-osd.14.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-11-05T08:42:07.644 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.10.log 2023-11-05T08:42:07.644 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-osd.10.log: 86.7% -- replaced with /var/log/ceph/ceph.log.gz 2023-11-05T08:42:07.646 INFO:teuthology.orchestra.run.smithi069.stderr: 89.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-11-05T08:42:07.655 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35631.log 2023-11-05T08:42:07.655 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.40850.log 2023-11-05T08:42:07.655 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35830.log 2023-11-05T08:42:07.655 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35631.log: /var/log/ceph/objectstore_tool.40850.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.39368.log 2023-11-05T08:42:07.656 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35631.log.gz 2023-11-05T08:42:07.656 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.39252.log 2023-11-05T08:42:07.656 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35830.log.gz 2023-11-05T08:42:07.656 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/objectstore_tool.39368.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-11-05T08:42:07.656 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.0.39252.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.41592.log 2023-11-05T08:42:07.656 INFO:teuthology.orchestra.run.smithi090.stderr: 54.6% -- replaced with /var/log/ceph/ceph-client.0.39252.log.gz 2023-11-05T08:42:07.657 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-11-05T08:42:07.657 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/objectstore_tool.41592.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.11.log 2023-11-05T08:42:07.657 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose/var/log/ceph/ceph-osd.3.log: -- /var/log/ceph/ceph-client.admin.35878.log 2023-11-05T08:42:07.657 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35655.log 2023-11-05T08:42:07.657 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35878.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35878.log.gz 2023-11-05T08:42:07.658 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.15.log 2023-11-05T08:42:07.658 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-11-05T08:42:07.658 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35655.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35655.log.gz 2023-11-05T08:42:07.658 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-osd.15.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35607.log 2023-11-05T08:42:07.670 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-mgr.x.log: 93.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-11-05T08:42:07.671 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35727.log 2023-11-05T08:42:07.671 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35607.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35607.log.gz 2023-11-05T08:42:07.671 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35782.log 2023-11-05T08:42:07.671 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35727.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35727.log.gz 2023-11-05T08:42:07.672 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35679.log 2023-11-05T08:42:07.672 INFO:teuthology.orchestra.run.smithi090.stderr: 93.4% -- replaced with /var/log/ceph/objectstore_tool.40850.log.gz 2023-11-05T08:42:07.672 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35782.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35782.log.gz 2023-11-05T08:42:07.672 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.40109.log 2023-11-05T08:42:07.672 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35583.log 2023-11-05T08:42:07.673 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35679.log.gz 2023-11-05T08:42:07.673 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/objectstore_tool.40109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35902.log 2023-11-05T08:42:07.673 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35583.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35583.log.gz 2023-11-05T08:42:07.673 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35703.log 2023-11-05T08:42:07.673 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35902.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35902.log.gz 2023-11-05T08:42:07.673 INFO:teuthology.orchestra.run.smithi090.stderr: 93.3% -- replaced with /var/log/ceph/objectstore_tool.41592.log.gz 2023-11-05T08:42:07.674 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35751.log 2023-11-05T08:42:07.674 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35703.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35854.log 2023-11-05T08:42:07.674 INFO:teuthology.orchestra.run.smithi090.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35703.log.gz 2023-11-05T08:42:07.674 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35951.log 2023-11-05T08:42:07.674 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35751.log.gz 2023-11-05T08:42:07.675 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35854.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35854.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35927.log 2023-11-05T08:42:07.675 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-05T08:42:07.675 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35806.log 2023-11-05T08:42:07.675 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35951.log.gz 2023-11-05T08:42:07.675 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35927.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35927.log.gz 2023-11-05T08:42:07.676 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35806.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35806.log.gz 2023-11-05T08:42:07.676 INFO:teuthology.orchestra.run.smithi090.stderr: 93.8% -- replaced with /var/log/ceph/objectstore_tool.39368.log.gz 2023-11-05T08:42:07.677 INFO:teuthology.orchestra.run.smithi090.stderr: 93.4% -- replaced with /var/log/ceph/objectstore_tool.40109.log.gz 2023-11-05T08:42:07.713 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39102.log 2023-11-05T08:42:07.713 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47734.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42140.log.gz 2023-11-05T08:42:07.713 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47734.log.gz 2023-11-05T08:42:07.714 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41627.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41627.log.gz 2023-11-05T08:42:07.714 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39285.log 2023-11-05T08:42:07.714 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44743.log 2023-11-05T08:42:07.714 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39102.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39102.log.gz 2023-11-05T08:42:07.714 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39285.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39285.log.gz 2023-11-05T08:42:07.715 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39699.log 2023-11-05T08:42:07.715 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46950.log 2023-11-05T08:42:07.715 INFO:teuthology.orchestra.run.smithi002.stderr: 23.8% -- replaced with /var/log/ceph/ceph-client.admin.44743.log.gz 2023-11-05T08:42:07.715 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39699.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39699.log.gz 2023-11-05T08:42:07.715 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45514.log 2023-11-05T08:42:07.716 INFO:teuthology.orchestra.run.smithi002.stderr:gzip/var/log/ceph/ceph-client.admin.46950.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39283.log 2023-11-05T08:42:07.716 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46950.log.gz 2023-11-05T08:42:07.716 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45514.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45514.log.gz 2023-11-05T08:42:07.716 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41859.log 2023-11-05T08:42:07.716 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46480.log 2023-11-05T08:42:07.717 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39283.log.gz 2023-11-05T08:42:07.717 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41859.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41859.log.gz 2023-11-05T08:42:07.717 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42358.log 2023-11-05T08:42:07.717 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39169.log 2023-11-05T08:42:07.717 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46480.log.gz 2023-11-05T08:42:07.718 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42358.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42358.log.gz 2023-11-05T08:42:07.718 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43248.log 2023-11-05T08:42:07.718 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43940.log 2023-11-05T08:42:07.718 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39169.log.gz 2023-11-05T08:42:07.718 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43248.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43248.log.gz -5 2023-11-05T08:42:07.719 INFO:teuthology.orchestra.run.smithi002.stderr: --verbose -- /var/log/ceph/ceph-client.admin.43718.log 2023-11-05T08:42:07.719 INFO:teuthology.orchestra.run.smithi002.stderr:gzip/var/log/ceph/ceph-client.admin.43940.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40888.log 2023-11-05T08:42:07.719 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43940.log.gz 2023-11-05T08:42:07.719 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43718.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.43718.log.gz /var/log/ceph/ceph-client.admin.45852.log 2023-11-05T08:42:07.719 INFO:teuthology.orchestra.run.smithi002.stderr: 2023-11-05T08:42:07.720 INFO:teuthology.orchestra.run.smithi002.stderr:gzip/var/log/ceph/ceph-client.admin.40888.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.43064.log 2023-11-05T08:42:07.720 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40888.log.gz 2023-11-05T08:42:07.720 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45852.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45852.log.gz 2023-11-05T08:42:07.720 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42535.log 2023-11-05T08:42:07.720 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35560.log 2023-11-05T08:42:07.721 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43064.log.gz 2023-11-05T08:42:07.721 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42535.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42535.log.gz 2023-11-05T08:42:07.721 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42188.log 2023-11-05T08:42:07.721 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45696.log 2023-11-05T08:42:07.721 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.35560.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35560.log.gz 2023-11-05T08:42:07.721 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42188.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42188.log.gz 2023-11-05T08:42:07.722 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.8.log 2023-11-05T08:42:07.722 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45722.log/var/log/ceph/ceph-client.admin.45696.log: 2023-11-05T08:42:07.722 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45696.log.gz 2023-11-05T08:42:07.722 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46453.log 2023-11-05T08:42:07.722 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42047.log 2023-11-05T08:42:07.723 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45722.log.gz 2023-11-05T08:42:07.723 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47396.log 2023-11-05T08:42:07.723 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46453.log.gz 2023-11-05T08:42:07.723 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44217.log 2023-11-05T08:42:07.723 INFO:teuthology.orchestra.run.smithi002.stderr: 24.6% -- replaced with /var/log/ceph/ceph-client.admin.42047.log.gz 2023-11-05T08:42:07.724 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45930.log 2023-11-05T08:42:07.724 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47396.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47396.log.gz 2023-11-05T08:42:07.724 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42872.log 2023-11-05T08:42:07.724 INFO:teuthology.orchestra.run.smithi002.stderr: 25.7% -- replaced with /var/log/ceph/ceph-client.admin.44217.log.gz 2023-11-05T08:42:07.724 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45592.log 2023-11-05T08:42:07.725 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45930.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45930.log.gz 2023-11-05T08:42:07.725 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44802.log 2023-11-05T08:42:07.725 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42872.log.gz 2023-11-05T08:42:07.725 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40241.log 2023-11-05T08:42:07.725 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45592.log.gz 2023-11-05T08:42:07.726 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47880.log 2023-11-05T08:42:07.726 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44802.log.gz 2023-11-05T08:42:07.726 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40247.log 2023-11-05T08:42:07.726 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40241.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40241.log.gz 2023-11-05T08:42:07.726 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40222.log 2023-11-05T08:42:07.726 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47880.log.gz 2023-11-05T08:42:07.727 INFO:teuthology.orchestra.run.smithi002.stderr:gzip/var/log/ceph/ceph-client.admin.40247.log: -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-11-05T08:42:07.727 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40247.log.gz 2023-11-05T08:42:07.727 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44062.log 2023-11-05T08:42:07.727 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40222.log.gz 2023-11-05T08:42:07.727 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46716.log 2023-11-05T08:42:07.728 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42332.log 2023-11-05T08:42:07.728 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44062.log.gz 2023-11-05T08:42:07.728 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40651.log 2023-11-05T08:42:07.728 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46716.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46716.log.gz 2023-11-05T08:42:07.728 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42332.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.42332.log.gz 2023-11-05T08:42:07.729 INFO:teuthology.orchestra.run.smithi002.stderr: --verbose -- /var/log/ceph/ceph-client.admin.42581.log 2023-11-05T08:42:07.729 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40651.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40651.log.gz 2023-11-05T08:42:07.729 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40599.log 2023-11-05T08:42:07.729 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47448.log 2023-11-05T08:42:07.729 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42581.log.gz 2023-11-05T08:42:07.729 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42048.log 2023-11-05T08:42:07.730 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40599.log.gz 2023-11-05T08:42:07.730 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47448.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.47448.log.gz 2023-11-05T08:42:07.730 INFO:teuthology.orchestra.run.smithi002.stderr: --verbose -- /var/log/ceph/ceph-client.admin.46612.log 2023-11-05T08:42:07.730 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44851.log 2023-11-05T08:42:07.730 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42048.log.gz 2023-11-05T08:42:07.731 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46612.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46612.log.gz 2023-11-05T08:42:07.731 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42760.log 2023-11-05T08:42:07.731 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47344.log 2023-11-05T08:42:07.731 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44851.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44851.log.gz 2023-11-05T08:42:07.731 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47028.log 2023-11-05T08:42:07.732 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42760.log.gz 2023-11-05T08:42:07.732 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41411.log 2023-11-05T08:42:07.732 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47344.log.gz 2023-11-05T08:42:07.732 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47578.log 2023-11-05T08:42:07.732 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47028.log.gz 2023-11-05T08:42:07.732 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45800.log 2023-11-05T08:42:07.733 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41411.log.gz 2023-11-05T08:42:07.733 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47106.log 2023-11-05T08:42:07.733 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47578.log.gz 2023-11-05T08:42:07.733 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43274.log 2023-11-05T08:42:07.733 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45800.log.gz 2023-11-05T08:42:07.734 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45349.log 2023-11-05T08:42:07.734 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47106.log.gz 2023-11-05T08:42:07.734 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47158.log 2023-11-05T08:42:07.734 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43274.log.gz 2023-11-05T08:42:07.734 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47240.log 2023-11-05T08:42:07.735 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45349.log.gz 2023-11-05T08:42:07.735 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47158.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47158.log.gz 2023-11-05T08:42:07.735 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41963.log 2023-11-05T08:42:07.735 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46190.log 2023-11-05T08:42:07.735 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47240.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47240.log.gz 2023-11-05T08:42:07.735 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39307.log 2023-11-05T08:42:07.736 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41963.log.gz 2023-11-05T08:42:07.736 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47292.log 2023-11-05T08:42:07.736 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46190.log.gz 2023-11-05T08:42:07.736 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39304.log 2023-11-05T08:42:07.736 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39307.log.gz 2023-11-05T08:42:07.737 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39306.log 2023-11-05T08:42:07.737 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47292.log.gz 2023-11-05T08:42:07.737 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39746.log 2023-11-05T08:42:07.737 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39304.log.gz 2023-11-05T08:42:07.737 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40749.log 2023-11-05T08:42:07.738 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39306.log.gz 2023-11-05T08:42:07.738 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44016.log 2023-11-05T08:42:07.738 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39746.log.gz 2023-11-05T08:42:07.738 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44515.log 2023-11-05T08:42:07.738 INFO:teuthology.orchestra.run.smithi002.stderr: 24.2% -- replaced with /var/log/ceph/ceph-client.admin.40749.log.gz 2023-11-05T08:42:07.738 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41038.log 2023-11-05T08:42:07.739 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44016.log.gz 2023-11-05T08:42:07.739 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46398.log 2023-11-05T08:42:07.739 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44515.log.gz 2023-11-05T08:42:07.739 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43403.log 2023-11-05T08:42:07.739 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41038.log.gz 2023-11-05T08:42:07.740 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47708.log 2023-11-05T08:42:07.740 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46398.log.gz 2023-11-05T08:42:07.740 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46164.log 2023-11-05T08:42:07.740 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43403.log.gz 2023-11-05T08:42:07.740 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47682.log 2023-11-05T08:42:07.741 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47708.log.gz 2023-11-05T08:42:07.741 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46138.log 2023-11-05T08:42:07.741 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46164.log.gz 2023-11-05T08:42:07.741 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41467.log 2023-11-05T08:42:07.741 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47682.log.gz 2023-11-05T08:42:07.742 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46820.log 2023-11-05T08:42:07.742 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46138.log.gz 2023-11-05T08:42:07.742 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44801.log 2023-11-05T08:42:07.742 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41467.log.gz 2023-11-05T08:42:07.742 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46820.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46820.log.gz 2023-11-05T08:42:07.743 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39221.log 2023-11-05T08:42:07.743 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.30787.log 2023-11-05T08:42:07.743 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44801.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44801.log.gz 2023-11-05T08:42:07.743 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43429.log 2023-11-05T08:42:07.743 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39221.log.gz 2023-11-05T08:42:07.743 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph.tmp-client.admin.30787.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39300.log 2023-11-05T08:42:07.744 INFO:teuthology.orchestra.run.smithi002.stderr: -- replaced with /var/log/ceph/ceph.tmp-client.admin.30787.log.gz 2023-11-05T08:42:07.744 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39684.log 2023-11-05T08:42:07.744 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43429.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43429.log.gz 2023-11-05T08:42:07.744 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39656.log 2023-11-05T08:42:07.744 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39300.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39300.log.gz 2023-11-05T08:42:07.745 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43560.log 2023-11-05T08:42:07.745 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39684.log.gz 2023-11-05T08:42:07.745 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39286.log 2023-11-05T08:42:07.745 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39656.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39656.log.gz 2023-11-05T08:42:07.745 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39303.log 2023-11-05T08:42:07.746 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43560.log.gz 2023-11-05T08:42:07.746 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39286.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39286.log.gz 2023-11-05T08:42:07.746 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40625.log 2023-11-05T08:42:07.746 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43456.log 2023-11-05T08:42:07.746 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39303.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39303.log.gz 2023-11-05T08:42:07.746 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44192.log 2023-11-05T08:42:07.747 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40625.log.gz 2023-11-05T08:42:07.747 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46586.log 2023-11-05T08:42:07.747 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43456.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43456.log.gz 2023-11-05T08:42:07.747 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42412.log 2023-11-05T08:42:07.747 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44192.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44192.log.gz 2023-11-05T08:42:07.748 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44351.log 2023-11-05T08:42:07.748 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46586.log.gz 2023-11-05T08:42:07.748 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44559.log 2023-11-05T08:42:07.748 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42412.log.gz 2023-11-05T08:42:07.748 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40110.log 2023-11-05T08:42:07.749 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44351.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44351.log.gz 2023-11-05T08:42:07.749 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46112.log 2023-11-05T08:42:07.749 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44559.log.gz 2023-11-05T08:42:07.749 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40110.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46424.log 2023-11-05T08:42:07.749 INFO:teuthology.orchestra.run.smithi002.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40110.log.gz 2023-11-05T08:42:07.750 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41730.log 2023-11-05T08:42:07.750 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46112.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46112.log.gz 2023-11-05T08:42:07.750 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39074.log 2023-11-05T08:42:07.750 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46424.log.gz 2023-11-05T08:42:07.750 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39294.log 2023-11-05T08:42:07.750 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41730.log.gz 2023-11-05T08:42:07.751 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47054.log 2023-11-05T08:42:07.751 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39074.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39074.log.gz 2023-11-05T08:42:07.751 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45826.log 2023-11-05T08:42:07.751 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39294.log.gz 2023-11-05T08:42:07.751 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35691.log 2023-11-05T08:42:07.752 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47054.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47054.log.gz 2023-11-05T08:42:07.752 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44692.log 2023-11-05T08:42:07.752 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45826.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45826.log.gz 2023-11-05T08:42:07.752 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.35691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43195.log 2023-11-05T08:42:07.752 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35691.log.gz 2023-11-05T08:42:07.753 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44692.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.44692.log.gz 2023-11-05T08:42:07.753 INFO:teuthology.orchestra.run.smithi002.stderr: /var/log/ceph/ceph-client.admin.45982.log 2023-11-05T08:42:07.753 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45566.log 2023-11-05T08:42:07.753 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43195.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43195.log.gz 2023-11-05T08:42:07.753 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44903.log 2023-11-05T08:42:07.754 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45982.log.gz 2023-11-05T08:42:07.754 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45566.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45566.log.gz 2023-11-05T08:42:07.754 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39298.log 2023-11-05T08:42:07.754 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45015.log 2023-11-05T08:42:07.754 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44903.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44903.log.gz 2023-11-05T08:42:07.755 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43090.log 2023-11-05T08:42:07.755 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39298.log.gz 2023-11-05T08:42:07.755 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45015.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45015.log.gz 2023-11-05T08:42:07.755 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40228.log 2023-11-05T08:42:07.755 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46320.log 2023-11-05T08:42:07.755 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43090.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43090.log.gz 2023-11-05T08:42:07.756 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43151.log 2023-11-05T08:42:07.756 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40228.log.gz 2023-11-05T08:42:07.756 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46320.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46320.log.gz 2023-11-05T08:42:07.756 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47266.log 2023-11-05T08:42:07.756 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44877.log 2023-11-05T08:42:07.757 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43151.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43151.log.gz 2023-11-05T08:42:07.757 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44592.log 2023-11-05T08:42:07.757 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47266.log.gz 2023-11-05T08:42:07.757 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44877.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44877.log.gz -5 2023-11-05T08:42:07.757 INFO:teuthology.orchestra.run.smithi002.stderr: --verbose -- /var/log/ceph/ceph-client.admin.42438.log 2023-11-05T08:42:07.757 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44939.log 2023-11-05T08:42:07.758 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44592.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44592.log.gz 2023-11-05T08:42:07.758 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39734.log 2023-11-05T08:42:07.758 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42438.log.gz 2023-11-05T08:42:07.758 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42618.log 2023-11-05T08:42:07.758 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44939.log.gz 2023-11-05T08:42:07.759 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.39734.log: /var/log/ceph/ceph-client.admin.44063.log 2023-11-05T08:42:07.759 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39734.log.gz 2023-11-05T08:42:07.759 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40220.log 2023-11-05T08:42:07.759 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42618.log.gz 2023-11-05T08:42:07.759 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44063.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39271.log 2023-11-05T08:42:07.760 INFO:teuthology.orchestra.run.smithi002.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44063.log.gz 2023-11-05T08:42:07.760 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45110.log 2023-11-05T08:42:07.760 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40220.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40220.log.gz 2023-11-05T08:42:07.760 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46506.log 2023-11-05T08:42:07.760 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39271.log.gz 2023-11-05T08:42:07.761 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43014.log 2023-11-05T08:42:07.761 INFO:teuthology.orchestra.run.smithi002.stderr: 23.4% -- replaced with /var/log/ceph/ceph-client.admin.45110.log.gz 2023-11-05T08:42:07.761 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44385.log 2023-11-05T08:42:07.761 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46506.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46506.log.gz 2023-11-05T08:42:07.761 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46532.log 2023-11-05T08:42:07.761 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43014.log.gz 2023-11-05T08:42:07.762 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43377.log 0.0% 2023-11-05T08:42:07.762 INFO:teuthology.orchestra.run.smithi002.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44385.log.gz 2023-11-05T08:42:07.762 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46268.log 2023-11-05T08:42:07.762 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46532.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46532.log.gz 2023-11-05T08:42:07.762 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47080.log 2023-11-05T08:42:07.763 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43377.log.gz 2023-11-05T08:42:07.763 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45300.log 2023-11-05T08:42:07.763 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46268.log.gz 2023-11-05T08:42:07.763 INFO:teuthology.orchestra.run.smithi002.stderr:gzip/var/log/ceph/ceph-client.admin.47080.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.43094.log 2023-11-05T08:42:07.763 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47080.log.gz 2023-11-05T08:42:07.764 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39305.log 2023-11-05T08:42:07.764 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45300.log.gz 2023-11-05T08:42:07.764 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43094.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-11-05T08:42:07.764 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43094.log.gz 2023-11-05T08:42:07.764 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40495.log 2023-11-05T08:42:07.765 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39305.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39305.log.gz 2023-11-05T08:42:07.765 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39136.log 2023-11-05T08:42:07.765 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39683.log 2023-11-05T08:42:07.765 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40495.log.gz 2023-11-05T08:42:07.765 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40746.log 2023-11-05T08:42:07.766 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39136.log.gz 2023-11-05T08:42:07.766 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40990.log 2023-11-05T08:42:07.766 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39683.log: 0.0%/var/log/ceph/ceph-client.admin.40746.log: -- replaced with /var/log/ceph/ceph-client.admin.39683.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41810.log 2023-11-05T08:42:07.766 INFO:teuthology.orchestra.run.smithi002.stderr: 2023-11-05T08:42:07.766 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40746.log.gz 2023-11-05T08:42:07.766 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40990.log: 86.7% -- replaced with /var/log/ceph/ceph.log.gz 2023-11-05T08:42:07.767 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40990.log.gz 2023-11-05T08:42:07.767 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44111.log 2023-11-05T08:42:07.767 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46008.log 2023-11-05T08:42:07.767 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41810.log.gz 2023-11-05T08:42:07.768 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45250.log 2023-11-05T08:42:07.768 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44111.log.gz 2023-11-05T08:42:07.768 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46372.log 2023-11-05T08:42:07.768 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46008.log.gz 2023-11-05T08:42:07.768 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42756.log 2023-11-05T08:42:07.768 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45250.log.gz 2023-11-05T08:42:07.769 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45878.log 2023-11-05T08:42:07.769 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46372.log.gz 2023-11-05T08:42:07.769 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43304.log 2023-11-05T08:42:07.769 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42756.log.gz 2023-11-05T08:42:07.769 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41042.log 2023-11-05T08:42:07.770 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45878.log.gz 2023-11-05T08:42:07.770 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43040.log 2023-11-05T08:42:07.770 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43304.log.gz 2023-11-05T08:42:07.770 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44718.log 2023-11-05T08:42:07.770 INFO:teuthology.orchestra.run.smithi002.stderr: 23.5% -- replaced with /var/log/ceph/ceph-client.admin.41042.log.gz 2023-11-05T08:42:07.771 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46216.log 2023-11-05T08:42:07.771 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43040.log.gz 2023-11-05T08:42:07.771 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39284.log 2023-11-05T08:42:07.771 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44718.log.gz 2023-11-05T08:42:07.771 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39708.log 2023-11-05T08:42:07.771 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46216.log.gz 2023-11-05T08:42:07.772 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45301.log 2023-11-05T08:42:07.772 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39284.log.gz 2023-11-05T08:42:07.772 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43662.log 2023-11-05T08:42:07.772 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39708.log.gz 2023-11-05T08:42:07.772 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45748.log 2023-11-05T08:42:07.773 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45301.log.gz 2023-11-05T08:42:07.773 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40079.log 2023-11-05T08:42:07.773 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43662.log.gz 2023-11-05T08:42:07.773 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44447.log 2023-11-05T08:42:07.773 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45748.log.gz 2023-11-05T08:42:07.773 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47786.log 2023-11-05T08:42:07.774 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40079.log.gz 2023-11-05T08:42:07.774 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46346.log 2023-11-05T08:42:07.774 INFO:teuthology.orchestra.run.smithi002.stderr: 23.8% -- replaced with /var/log/ceph/ceph-client.admin.44447.log.gz 2023-11-05T08:42:07.774 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43798.log 2023-11-05T08:42:07.774 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47786.log.gz 2023-11-05T08:42:07.775 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43222.log 2023-11-05T08:42:07.775 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46346.log.gz 2023-11-05T08:42:07.775 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45246.log 2023-11-05T08:42:07.775 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43798.log.gz 2023-11-05T08:42:07.775 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46742.log 2023-11-05T08:42:07.776 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43222.log.gz 2023-11-05T08:42:07.776 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42988.log 2023-11-05T08:42:07.776 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45246.log.gz 2023-11-05T08:42:07.776 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44135.log 2023-11-05T08:42:07.776 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46742.log.gz 2023-11-05T08:42:07.777 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39733.log 2023-11-05T08:42:07.777 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42988.log.gz 2023-11-05T08:42:07.777 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44312.log 2023-11-05T08:42:07.777 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44135.log.gz 2023-11-05T08:42:07.777 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47630.log 2023-11-05T08:42:07.777 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39733.log.gz 2023-11-05T08:42:07.778 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47422.log 2023-11-05T08:42:07.778 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44312.log.gz 2023-11-05T08:42:07.778 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-11-05T08:42:07.778 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47630.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47630.log.gz 2023-11-05T08:42:07.778 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43638.log 2023-11-05T08:42:07.779 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47422.log.gz 2023-11-05T08:42:07.779 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41386.log 2023-11-05T08:42:07.779 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40985.log 2023-11-05T08:42:07.779 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43638.log.gz 2023-11-05T08:42:07.779 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42698.log 2023-11-05T08:42:07.780 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41386.log.gz 2023-11-05T08:42:07.780 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47604.log 2023-11-05T08:42:07.780 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40985.log: /var/log/ceph/ceph-client.admin.42698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43278.log 2023-11-05T08:42:07.780 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40985.log.gz 2023-11-05T08:42:07.780 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42698.log.gz 2023-11-05T08:42:07.780 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47604.log: 89.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-11-05T08:42:07.781 INFO:teuthology.orchestra.run.smithi002.stderr:gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.47604.log.gz --verbose 2023-11-05T08:42:07.781 INFO:teuthology.orchestra.run.smithi002.stderr: -- /var/log/ceph/ceph-client.admin.47552.log 2023-11-05T08:42:07.781 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40707.log 2023-11-05T08:42:07.781 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43278.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43278.log.gz 2023-11-05T08:42:07.781 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40755.log 2023-11-05T08:42:07.782 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47552.log.gz 2023-11-05T08:42:07.782 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40707.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40707.log.gz 2023-11-05T08:42:07.782 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45353.log 2023-11-05T08:42:07.782 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40755.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40755.log.gz 2023-11-05T08:42:07.782 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42018.log 2023-11-05T08:42:07.782 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47132.log 2023-11-05T08:42:07.783 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45353.log.gz 2023-11-05T08:42:07.783 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45191.log 2023-11-05T08:42:07.783 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42018.log.gz 2023-11-05T08:42:07.783 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39696.log 2023-11-05T08:42:07.783 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47132.log.gz 2023-11-05T08:42:07.784 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47214.log 2023-11-05T08:42:07.784 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45191.log.gz 2023-11-05T08:42:07.784 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46976.log 2023-11-05T08:42:07.784 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39696.log.gz 2023-11-05T08:42:07.784 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39681.log 2023-11-05T08:42:07.784 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47214.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47214.log.gz 2023-11-05T08:42:07.785 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39291.log 2023-11-05T08:42:07.785 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46976.log.gz 2023-11-05T08:42:07.785 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39681.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.12.log 2023-11-05T08:42:07.785 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39681.log.gz 2023-11-05T08:42:07.785 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45540.log 2023-11-05T08:42:07.786 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39291.log.gz 2023-11-05T08:42:07.786 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-osd.12.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43744.log 2023-11-05T08:42:07.786 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39710.log 2023-11-05T08:42:07.786 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45540.log.gz 2023-11-05T08:42:07.786 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46924.log 2023-11-05T08:42:07.787 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43744.log.gz 2023-11-05T08:42:07.787 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39710.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39710.log.gz 2023-11-05T08:42:07.787 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40185.log 2023-11-05T08:42:07.787 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41334.log 2023-11-05T08:42:07.787 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46924.log.gz 2023-11-05T08:42:07.787 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43977.log 2023-11-05T08:42:07.788 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40185.log.gz 2023-11-05T08:42:07.788 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44381.log 2023-11-05T08:42:07.788 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41334.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41334.log.gz 2023-11-05T08:42:07.788 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41676.log 2023-11-05T08:42:07.788 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43977.log.gz 2023-11-05T08:42:07.789 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44381.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44381.log.gz 2023-11-05T08:42:07.789 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39577.log 2023-11-05T08:42:07.789 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43855.log 2023-11-05T08:42:07.789 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41676.log.gz 2023-11-05T08:42:07.789 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40117.log 2023-11-05T08:42:07.789 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39577.log.gz 2023-11-05T08:42:07.790 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46242.log 2023-11-05T08:42:07.790 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43855.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43855.log.gz 2023-11-05T08:42:07.790 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39195.log 2023-11-05T08:42:07.790 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40117.log.gz 2023-11-05T08:42:07.790 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46242.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46242.log.gz 2023-11-05T08:42:07.791 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41938.log 2023-11-05T08:42:07.791 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41756.log 2023-11-05T08:42:07.791 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39195.log.gz 2023-11-05T08:42:07.791 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47838.log 2023-11-05T08:42:07.791 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41938.log.gz 2023-11-05T08:42:07.792 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41756.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41756.log.gz 2023-11-05T08:42:07.792 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45483.log 2023-11-05T08:42:07.792 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46664.log 2023-11-05T08:42:07.792 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47838.log.gz 2023-11-05T08:42:07.792 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45483.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45483.log.gz 2023-11-05T08:42:07.792 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39711.log 2023-11-05T08:42:07.793 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41241.log 2023-11-05T08:42:07.793 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46664.log.gz 2023-11-05T08:42:07.793 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40248.log 2023-11-05T08:42:07.793 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39711.log.gz 2023-11-05T08:42:07.793 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46794.log 2023-11-05T08:42:07.794 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41241.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41241.log.gz 2023-11-05T08:42:07.794 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45165.log 2023-11-05T08:42:07.794 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40248.log.gz 2023-11-05T08:42:07.794 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46794.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46794.log.gz 2023-11-05T08:42:07.794 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-11-05T08:42:07.795 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35871.log 2023-11-05T08:42:07.795 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45165.log.gz 2023-11-05T08:42:07.795 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42252.log 2023-11-05T08:42:07.795 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41152.log 2023-11-05T08:42:07.795 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.35871.log: /var/log/ceph/ceph-client.admin.42252.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35871.log.gz 2023-11-05T08:42:07.795 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42252.log.gz 2023-11-05T08:42:07.796 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42934.log 2023-11-05T08:42:07.796 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40092.log 2023-11-05T08:42:07.796 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41152.log: /var/log/ceph/ceph-client.admin.42934.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41152.log.gz 2023-11-05T08:42:07.796 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42934.log.gz 2023-11-05T08:42:07.796 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47184.log 2023-11-05T08:42:07.797 INFO:teuthology.orchestra.run.smithi002.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-11-05T08:42:07.797 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47370.log 2023-11-05T08:42:07.797 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40092.log: /var/log/ceph/ceph-client.admin.47184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47656.log 2023-11-05T08:42:07.797 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40092.log.gz 2023-11-05T08:42:07.797 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47184.log.gz/var/log/ceph/ceph-client.admin.47370.log: 2023-11-05T08:42:07.798 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43350.log 2023-11-05T08:42:07.798 INFO:teuthology.orchestra.run.smithi002.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47370.log.gz 2023-11-05T08:42:07.798 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39705.log 2023-11-05T08:42:07.798 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47656.log: /var/log/ceph/ceph-client.admin.43350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39275.log 2023-11-05T08:42:07.798 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47656.log.gz 2023-11-05T08:42:07.798 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43350.log.gz 2023-11-05T08:42:07.799 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39705.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39705.log.gz 2023-11-05T08:42:07.799 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46872.log 2023-11-05T08:42:07.799 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46034.log 2023-11-05T08:42:07.799 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39275.log: /var/log/ceph/ceph-client.admin.46872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40179.log 2023-11-05T08:42:07.799 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39275.log.gz 2023-11-05T08:42:07.800 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46872.log.gz 2023-11-05T08:42:07.800 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46086.log 2023-11-05T08:42:07.800 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46034.log.gz 2023-11-05T08:42:07.800 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46638.log 2023-11-05T08:42:07.800 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40179.log: /var/log/ceph/ceph-client.admin.46086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47526.log 2023-11-05T08:42:07.800 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40179.log.gz 2023-11-05T08:42:07.801 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46086.log.gz 2023-11-05T08:42:07.801 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46638.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.46638.log.gz --verbose 2023-11-05T08:42:07.801 INFO:teuthology.orchestra.run.smithi002.stderr: -- /var/log/ceph/ceph-client.admin.47318.log 2023-11-05T08:42:07.801 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46768.log 2023-11-05T08:42:07.801 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47526.log: /var/log/ceph/ceph-client.admin.47318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39274.log 2023-11-05T08:42:07.801 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47318.log.gz 2023-11-05T08:42:07.802 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46768.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47526.log.gz 2023-11-05T08:42:07.802 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46768.log.gz 2023-11-05T08:42:07.802 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44585.log 2023-11-05T08:42:07.802 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44222.log 2023-11-05T08:42:07.802 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39274.log: /var/log/ceph/ceph-client.admin.44585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47864.log 2023-11-05T08:42:07.803 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39274.log.gz 2023-11-05T08:42:07.803 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44585.log.gz 2023-11-05T08:42:07.803 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42964.log 2023-11-05T08:42:07.803 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44222.log.gz 2023-11-05T08:42:07.803 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40111.log 2023-11-05T08:42:07.804 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47864.log: /var/log/ceph/ceph-client.admin.42964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47812.log 2023-11-05T08:42:07.804 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47864.log.gz 2023-11-05T08:42:07.804 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42964.log.gz 2023-11-05T08:42:07.804 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45774.log 2023-11-05T08:42:07.804 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40111.log.gz 2023-11-05T08:42:07.805 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44593.log 2023-11-05T08:42:07.805 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47812.log: /var/log/ceph/ceph-client.admin.45774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46560.log 2023-11-05T08:42:07.805 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47812.log.gz 2023-11-05T08:42:07.805 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45774.log.gz 2023-11-05T08:42:07.805 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45457.log 2023-11-05T08:42:07.806 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44593.log.gz 2023-11-05T08:42:07.806 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42673.log 2023-11-05T08:42:07.806 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46560.log: /var/log/ceph/ceph-client.admin.45457.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46560.log.gz 2023-11-05T08:42:07.806 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45457.log.gz 2023-11-05T08:42:07.806 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42673.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42673.log.gz 2023-11-05T08:42:07.807 INFO:teuthology.orchestra.run.smithi002.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-11-05T08:42:07.807 INFO:teuthology.orchestra.run.smithi053.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-11-05T08:42:07.844 INFO:teuthology.orchestra.run.smithi069.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-11-05T08:42:07.992 INFO:teuthology.orchestra.run.smithi002.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-11-05T08:42:08.032 INFO:teuthology.orchestra.run.smithi002.stderr: 90.9% -- replaced with /var/log/ceph/ceph-osd.12.log.gz 2023-11-05T08:42:08.072 INFO:teuthology.orchestra.run.smithi090.stderr: 91.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-11-05T08:42:08.170 INFO:teuthology.orchestra.run.smithi053.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-11-05T08:42:08.179 INFO:teuthology.orchestra.run.smithi069.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.10.log.gz 2023-11-05T08:42:08.185 INFO:teuthology.orchestra.run.smithi090.stderr: 91.8% -- replaced with /var/log/ceph/ceph-osd.15.log.gz 2023-11-05T08:42:08.207 INFO:teuthology.orchestra.run.smithi069.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.14.log.gz 2023-11-05T08:42:08.219 INFO:teuthology.orchestra.run.smithi053.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.9.log.gz 2023-11-05T08:42:08.236 INFO:teuthology.orchestra.run.smithi069.stderr: 91.8% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-11-05T08:42:08.244 INFO:teuthology.orchestra.run.smithi053.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-11-05T08:42:08.265 INFO:teuthology.orchestra.run.smithi090.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-11-05T08:42:08.287 INFO:teuthology.orchestra.run.smithi069.stderr: 91.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-11-05T08:42:08.289 INFO:teuthology.orchestra.run.smithi069.stderr: 2023-11-05T08:42:08.289 INFO:teuthology.orchestra.run.smithi069.stderr:real 0m0.669s 2023-11-05T08:42:08.289 INFO:teuthology.orchestra.run.smithi069.stderr:user 0m2.503s 2023-11-05T08:42:08.289 INFO:teuthology.orchestra.run.smithi069.stderr:sys 0m0.081s 2023-11-05T08:42:08.330 INFO:teuthology.orchestra.run.smithi002.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.8.log.gz 2023-11-05T08:42:08.399 INFO:teuthology.orchestra.run.smithi002.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-11-05T08:42:08.401 INFO:teuthology.orchestra.run.smithi002.stderr: 2023-11-05T08:42:08.401 INFO:teuthology.orchestra.run.smithi002.stderr:real 0m0.825s 2023-11-05T08:42:08.401 INFO:teuthology.orchestra.run.smithi002.stderr:user 0m2.567s 2023-11-05T08:42:08.402 INFO:teuthology.orchestra.run.smithi002.stderr:sys 0m0.153s 2023-11-05T08:42:08.405 INFO:teuthology.orchestra.run.smithi090.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.11.log.gz 2023-11-05T08:42:08.406 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-11-05T08:42:08.406 INFO:teuthology.orchestra.run.smithi090.stderr:real 0m0.776s 2023-11-05T08:42:08.407 INFO:teuthology.orchestra.run.smithi090.stderr:user 0m2.379s 2023-11-05T08:42:08.407 INFO:teuthology.orchestra.run.smithi090.stderr:sys 0m0.095s 2023-11-05T08:42:08.598 INFO:teuthology.orchestra.run.smithi053.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.13.log.gz 2023-11-05T08:42:08.599 INFO:teuthology.orchestra.run.smithi053.stderr: 2023-11-05T08:42:08.599 INFO:teuthology.orchestra.run.smithi053.stderr:real 0m1.010s 2023-11-05T08:42:08.599 INFO:teuthology.orchestra.run.smithi053.stderr:user 0m2.944s 2023-11-05T08:42:08.599 INFO:teuthology.orchestra.run.smithi053.stderr:sys 0m0.088s 2023-11-05T08:42:08.600 INFO:tasks.ceph:Archiving logs... 2023-11-05T08:42:08.601 DEBUG:teuthology.misc:Transferring archived files from smithi002:/var/log/ceph to /home/teuthworker/archive/benhanokh-2023-10-31_16:46:33-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442398/remote/smithi002/log 2023-11-05T08:42:08.602 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-05T08:42:09.126 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/log/ceph to /home/teuthworker/archive/benhanokh-2023-10-31_16:46:33-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442398/remote/smithi053/log 2023-11-05T08:42:09.127 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-05T08:42:09.492 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/log/ceph to /home/teuthworker/archive/benhanokh-2023-10-31_16:46:33-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442398/remote/smithi069/log 2023-11-05T08:42:09.494 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-05T08:42:09.818 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/log/ceph to /home/teuthworker/archive/benhanokh-2023-10-31_16:46:33-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442398/remote/smithi090/log 2023-11-05T08:42:09.819 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-05T08:42:10.177 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/thrashosds.py", line 215, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2792, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2782, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2545, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2537, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi002 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi002 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-11-05T08:42:10.178 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-05T08:42:10.190 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/thrashosds.py", line 215, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2792, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2782, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2545, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2537, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi002 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2023-11-05T08:42:10.190 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-05T08:42:10.191 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-05T08:42:10.194 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-05T08:42:10.197 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-05T08:42:10.199 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-05T08:42:10.353 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-05T08:42:10.368 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-05T08:42:10.369 DEBUG:teuthology.orchestra.run.smithi002:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-05T08:42:10.372 DEBUG:teuthology.orchestra.run.smithi053:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-05T08:42:10.375 DEBUG:teuthology.orchestra.run.smithi069:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-05T08:42:10.377 DEBUG:teuthology.orchestra.run.smithi090:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-05T08:42:10.383 INFO:teuthology.orchestra.run.smithi002.stdout: remote refid st t when poll reach delay offset jitter 2023-11-05T08:42:10.384 INFO:teuthology.orchestra.run.smithi002.stdout:============================================================================== 2023-11-05T08:42:10.384 INFO:teuthology.orchestra.run.smithi002.stdout:*hv01.front.sepi 44.4.53.2 2 u 1 64 377 0.074 -4.319 1.703 2023-11-05T08:42:10.384 INFO:teuthology.orchestra.run.smithi002.stdout:+hv02.front.sepi 63.231.80.2 3 u 65 64 377 0.087 -0.124 1.486 2023-11-05T08:42:10.384 INFO:teuthology.orchestra.run.smithi002.stdout:+hv03.front.sepi 74.6.168.72 3 u 10 64 377 0.088 -0.389 1.857 2023-11-05T08:42:10.385 INFO:teuthology.orchestra.run.smithi002.stdout: hv04.front.sepi .INIT. 16 u - 256 0 0.000 +0.000 0.000 2023-11-05T08:42:10.387 INFO:teuthology.orchestra.run.smithi069.stdout: remote refid st t when poll reach delay offset jitter 2023-11-05T08:42:10.387 INFO:teuthology.orchestra.run.smithi069.stdout:============================================================================== 2023-11-05T08:42:10.388 INFO:teuthology.orchestra.run.smithi069.stdout:*hv01.front.sepi 44.4.53.2 2 u 3 64 377 0.094 -0.703 4.420 2023-11-05T08:42:10.388 INFO:teuthology.orchestra.run.smithi069.stdout:+hv02.front.sepi 63.231.80.2 3 u 8 64 377 0.096 -3.650 2.846 2023-11-05T08:42:10.388 INFO:teuthology.orchestra.run.smithi069.stdout:+hv03.front.sepi 74.6.168.72 3 u 59 64 377 0.089 -1.075 2.303 2023-11-05T08:42:10.388 INFO:teuthology.orchestra.run.smithi069.stdout: hv04.front.sepi .INIT. 16 u - 256 0 0.000 +0.000 0.000 2023-11-05T08:42:10.388 INFO:teuthology.orchestra.run.smithi053.stdout: remote refid st t when poll reach delay offset jitter 2023-11-05T08:42:10.389 INFO:teuthology.orchestra.run.smithi053.stdout:============================================================================== 2023-11-05T08:42:10.389 INFO:teuthology.orchestra.run.smithi053.stdout:*hv01.front.sepi 44.4.53.2 2 u 11 64 377 0.100 -6.318 3.326 2023-11-05T08:42:10.389 INFO:teuthology.orchestra.run.smithi053.stdout:+hv02.front.sepi 63.231.80.2 3 u 11 64 377 0.115 +0.039 2.670 2023-11-05T08:42:10.389 INFO:teuthology.orchestra.run.smithi053.stdout:+hv03.front.sepi 74.6.168.72 3 u 6 64 377 0.073 -1.648 1.853 2023-11-05T08:42:10.390 INFO:teuthology.orchestra.run.smithi053.stdout: hv04.front.sepi .INIT. 16 u - 256 0 0.000 +0.000 0.000 2023-11-05T08:42:10.392 INFO:teuthology.orchestra.run.smithi090.stdout: remote refid st t when poll reach delay offset jitter 2023-11-05T08:42:10.392 INFO:teuthology.orchestra.run.smithi090.stdout:============================================================================== 2023-11-05T08:42:10.392 INFO:teuthology.orchestra.run.smithi090.stdout:*hv01.front.sepi 44.4.53.2 2 u 1 64 377 0.107 -2.638 1.123 2023-11-05T08:42:10.392 INFO:teuthology.orchestra.run.smithi090.stdout:+hv02.front.sepi 63.231.80.2 3 u 7 64 377 0.119 -0.896 1.163 2023-11-05T08:42:10.393 INFO:teuthology.orchestra.run.smithi090.stdout:+hv03.front.sepi 74.6.168.72 3 u 6 64 377 0.091 -0.873 1.132 2023-11-05T08:42:10.393 INFO:teuthology.orchestra.run.smithi090.stdout: hv04.front.sepi .INIT. 16 u - 256 0 0.000 +0.000 0.000 2023-11-05T08:42:10.393 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-05T08:42:10.411 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-05T08:42:10.411 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-05T08:42:10.426 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-05T08:42:10.442 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-05T08:42:10.460 INFO:teuthology.task.internal:Duration was 1275.387260 seconds 2023-11-05T08:42:10.461 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-05T08:42:10.477 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-05T08:42:10.478 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-05T08:42:10.481 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-05T08:42:10.483 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-05T08:42:10.486 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-05T08:42:10.564 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-05T08:42:10.565 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi002.front.sepia.ceph.com 2023-11-05T08:42:10.565 DEBUG:teuthology.orchestra.run.smithi002:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-05T08:42:10.575 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi053.front.sepia.ceph.com 2023-11-05T08:42:10.575 DEBUG:teuthology.orchestra.run.smithi053:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-05T08:42:10.586 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi069.front.sepia.ceph.com 2023-11-05T08:42:10.586 DEBUG:teuthology.orchestra.run.smithi069:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-05T08:42:10.596 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi090.front.sepia.ceph.com 2023-11-05T08:42:10.597 DEBUG:teuthology.orchestra.run.smithi090:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-05T08:42:10.607 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-05T08:42:10.607 DEBUG:teuthology.orchestra.run.smithi002:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-05T08:42:10.617 DEBUG:teuthology.orchestra.run.smithi053:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-05T08:42:10.630 DEBUG:teuthology.orchestra.run.smithi069:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-05T08:42:10.641 DEBUG:teuthology.orchestra.run.smithi090:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-05T08:42:10.670 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-05T08:42:10.671 DEBUG:teuthology.orchestra.run.smithi002:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-05T08:42:10.681 DEBUG:teuthology.orchestra.run.smithi053:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-05T08:42:10.694 DEBUG:teuthology.orchestra.run.smithi069:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-05T08:42:10.706 DEBUG:teuthology.orchestra.run.smithi090:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-05T08:42:10.768 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-05T08:42:10.778 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-05T08:42:10.779 DEBUG:teuthology.orchestra.run.smithi002:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-05T08:42:10.790 DEBUG:teuthology.orchestra.run.smithi053:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-05T08:42:10.803 DEBUG:teuthology.orchestra.run.smithi069:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-05T08:42:10.817 DEBUG:teuthology.orchestra.run.smithi090:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-05T08:42:10.827 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-05T08:42:10.840 DEBUG:teuthology.orchestra.run.smithi002:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-05T08:42:10.843 DEBUG:teuthology.orchestra.run.smithi053:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-05T08:42:10.846 DEBUG:teuthology.orchestra.run.smithi069:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-05T08:42:10.853 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern = core 2023-11-05T08:42:10.854 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern = core 2023-11-05T08:42:10.862 DEBUG:teuthology.orchestra.run.smithi090:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-05T08:42:10.872 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern = core 2023-11-05T08:42:10.878 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern = core 2023-11-05T08:42:10.907 DEBUG:teuthology.orchestra.run.smithi002:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-05T08:42:10.920 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi002.front.sepia.ceph.com, flagging run as failed 2023-11-05T08:42:10.920 DEBUG:teuthology.orchestra.run.smithi053:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-05T08:42:10.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-05T08:42:10.926 DEBUG:teuthology.orchestra.run.smithi069:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-05T08:42:10.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-05T08:42:10.945 DEBUG:teuthology.orchestra.run.smithi090:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-05T08:42:10.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-05T08:42:10.952 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-05T08:42:10.963 INFO:teuthology.task.internal:Transferring archived files... 2023-11-05T08:42:10.964 DEBUG:teuthology.misc:Transferring archived files from smithi002:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/benhanokh-2023-10-31_16:46:33-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442398/remote/smithi002 2023-11-05T08:42:10.964 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-05T08:42:30.821 INFO:teuthology.task.internal:Transferring binaries for coredumps... 2023-11-05T08:42:30.868 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/benhanokh-2023-10-31_16:46:33-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442398/remote/smithi002/coredump/1699173169.35674.core.gz: gzip compressed data, was "1699173169.35674.core", last modified: Sun Nov 5 08:42:11 2023, max compression, original size modulo 2^32 759234560 2023-11-05T08:42:30.869 INFO:teuthology.task.internal:core is compressed, try accessing gzip file ... 2023-11-05T08:42:34.025 INFO:teuthology.task.internal: core looks like: /tmp/tmpiae9tt52: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'ceph-osd -f --cluster ceph -i 0', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0, execfn: '/usr/bin/ceph-osd', platform: 'x86_64' 2023-11-05T08:42:34.113 INFO:teuthology.task.internal: dump_program: ceph-osd -f --cluster ceph -i 0 2023-11-05T08:42:34.114 DEBUG:teuthology.orchestra.run.smithi002:> which 'ceph-osd -f --cluster ceph -i 0' 2023-11-05T08:42:34.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-05T08:42:34.126 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/thrashosds.py", line 215, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2792, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2782, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2545, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2537, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi002 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/task/internal/__init__.py", line 420, in archive fetch_binaries_for_coredumps(path, rem) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/task/internal/__init__.py", line 345, in fetch_binaries_for_coredumps remote_path = remote.sh(['which', dump_program]).rstrip() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi002 with status 1: "which 'ceph-osd -f --cluster ceph -i 0'" 2023-11-05T08:42:34.127 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-05T08:42:34.140 INFO:teuthology.task.internal:Not uploading archives. 2023-11-05T08:42:34.141 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-05T08:42:34.156 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-05T08:42:34.157 DEBUG:teuthology.orchestra.run.smithi002:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-05T08:42:34.175 DEBUG:teuthology.orchestra.run.smithi053:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-05T08:42:34.179 DEBUG:teuthology.orchestra.run.smithi069:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-05T08:42:34.181 INFO:teuthology.orchestra.run.smithi002.stdout: 655472 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Nov 5 08:42 /home/ubuntu/cephtest 2023-11-05T08:42:34.181 INFO:teuthology.orchestra.run.smithi002.stdout: 655473 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Nov 5 08:20 /home/ubuntu/cephtest/archive 2023-11-05T08:42:34.181 INFO:teuthology.orchestra.run.smithi002.stdout: 658634 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 5 08:42 /home/ubuntu/cephtest/archive/syslog 2023-11-05T08:42:34.181 INFO:teuthology.orchestra.run.smithi002.stdout: 658636 4 -rw-rw-rw- 1 ubuntu ubuntu 795 Nov 5 08:42 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2023-11-05T08:42:34.181 INFO:teuthology.orchestra.run.smithi002.stdout: 658635 100 -rw-rw-r-- 1 ubuntu ubuntu 102275 Nov 5 08:42 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-05T08:42:34.182 INFO:teuthology.orchestra.run.smithi002.stdout: 663689 92 -rw-rw-rw- 1 ubuntu ubuntu 90667 Nov 5 08:42 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2023-11-05T08:42:34.182 INFO:teuthology.orchestra.run.smithi002.stdout: 658633 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 5 08:32 /home/ubuntu/cephtest/archive/coredump 2023-11-05T08:42:34.182 INFO:teuthology.orchestra.run.smithi002.stdout: 673783 61748 -rw------- 1 root root 759234560 Nov 5 08:32 /home/ubuntu/cephtest/archive/coredump/1699173169.35674.core 2023-11-05T08:42:34.183 INFO:teuthology.orchestra.run.smithi002.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-11-05T08:42:34.184 INFO:teuthology.orchestra.run.smithi053.stdout: 655472 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Nov 5 08:42 /home/ubuntu/cephtest 2023-11-05T08:42:34.184 INFO:teuthology.orchestra.run.smithi053.stdout: 658466 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Nov 5 08:42 /home/ubuntu/cephtest/archive 2023-11-05T08:42:34.185 INFO:teuthology.orchestra.run.smithi053.stdout: 658468 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 5 08:42 /home/ubuntu/cephtest/archive/syslog 2023-11-05T08:42:34.185 INFO:teuthology.orchestra.run.smithi053.stdout: 658472 4 -rw-rw-rw- 1 ubuntu ubuntu 790 Nov 5 08:42 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2023-11-05T08:42:34.185 INFO:teuthology.orchestra.run.smithi053.stdout: 658469 96 -rw-rw-r-- 1 ubuntu ubuntu 96921 Nov 5 08:42 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-05T08:42:34.185 INFO:teuthology.orchestra.run.smithi053.stdout: 663523 84 -rw-rw-rw- 1 ubuntu ubuntu 82555 Nov 5 08:42 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2023-11-05T08:42:34.186 INFO:teuthology.orchestra.run.smithi053.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-11-05T08:42:34.189 DEBUG:teuthology.orchestra.run.smithi090:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-05T08:42:34.193 INFO:teuthology.orchestra.run.smithi069.stdout: 655472 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Nov 5 08:42 /home/ubuntu/cephtest 2023-11-05T08:42:34.193 INFO:teuthology.orchestra.run.smithi069.stdout: 658631 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Nov 5 08:42 /home/ubuntu/cephtest/archive 2023-11-05T08:42:34.193 INFO:teuthology.orchestra.run.smithi069.stdout: 658633 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 5 08:42 /home/ubuntu/cephtest/archive/syslog 2023-11-05T08:42:34.193 INFO:teuthology.orchestra.run.smithi069.stdout: 658637 4 -rw-rw-rw- 1 ubuntu ubuntu 825 Nov 5 08:42 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2023-11-05T08:42:34.194 INFO:teuthology.orchestra.run.smithi069.stdout: 658634 96 -rw-rw-r-- 1 ubuntu ubuntu 96750 Nov 5 08:42 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-05T08:42:34.194 INFO:teuthology.orchestra.run.smithi069.stdout: 663688 84 -rw-rw-rw- 1 ubuntu ubuntu 82704 Nov 5 08:42 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2023-11-05T08:42:34.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-05T08:42:34.195 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/thrashosds.py", line 215, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2792, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2782, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2545, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2537, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi002 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi002 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-11-05T08:42:34.197 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-05T08:42:34.213 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-05T08:42:34.252 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi002 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2023-11-05T08:42:34.253 INFO:teuthology.nuke:Checking targets against current locks 2023-11-05T08:42:34.256 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-11-05T08:42:34.260 INFO:teuthology.orchestra.run.smithi090.stdout: 655469 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Nov 5 08:42 /home/ubuntu/cephtest 2023-11-05T08:42:34.260 INFO:teuthology.orchestra.run.smithi090.stdout: 655470 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Nov 5 08:42 /home/ubuntu/cephtest/archive 2023-11-05T08:42:34.261 INFO:teuthology.orchestra.run.smithi090.stdout: 658630 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 5 08:42 /home/ubuntu/cephtest/archive/syslog 2023-11-05T08:42:34.261 INFO:teuthology.orchestra.run.smithi090.stdout: 658632 4 -rw-rw-rw- 1 ubuntu ubuntu 829 Nov 5 08:42 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2023-11-05T08:42:34.261 INFO:teuthology.orchestra.run.smithi090.stdout: 658631 96 -rw-rw-r-- 1 ubuntu ubuntu 97100 Nov 5 08:42 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-05T08:42:34.261 INFO:teuthology.orchestra.run.smithi090.stdout: 663688 84 -rw-rw-rw- 1 ubuntu ubuntu 82696 Nov 5 08:42 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2023-11-05T08:42:34.263 INFO:teuthology.orchestra.run.smithi090.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-11-05T08:42:34.289 DEBUG:teuthology.nuke:shortname: smithi002 2023-11-05T08:42:34.290 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-05T08:42:34.316 DEBUG:teuthology.nuke:shortname: smithi053 2023-11-05T08:42:34.317 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-05T08:42:34.322 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi002.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/benhanokh-2023-10-31_16:46:33-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442398', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-05 08:11:03.357295', 'locked_by': 'scheduled_benhanokh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFYnh2HerGKUWiNK+JZ1+sOjYIB226bYu1t+bYcFbrDbwRbToSpMKjOKohouju5rpo4JGem8VCkcd8WCrEfDO90='} 2023-11-05T08:42:34.343 DEBUG:teuthology.nuke:shortname: smithi069 2023-11-05T08:42:34.344 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-05T08:42:34.353 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi053.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/benhanokh-2023-10-31_16:46:33-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442398', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-05 08:11:03.353634', 'locked_by': 'scheduled_benhanokh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJhqMGwSC/AaSpescnzcOcLSXMVxyFf8TWoJBg6tZz9J7np1OvfHqj0Q6WgZ/zHx4+f+RgNMtOA5jvGv7ZeAjsU='} 2023-11-05T08:42:34.373 DEBUG:teuthology.nuke:shortname: smithi090 2023-11-05T08:42:34.374 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-05T08:42:34.379 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/benhanokh-2023-10-31_16:46:33-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442398', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-05 08:11:03.358959', 'locked_by': 'scheduled_benhanokh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDUJGpZU1aZPH20MVSpQexFWypB4vrahbDUJdquA7RrNdQ0snsbTNu0jtgDOtsAX775dxnSa84npB5kPsKt0MRM='} 2023-11-05T08:42:34.391 INFO:teuthology.orchestra.console.smithi002:Power off 2023-11-05T08:42:34.393 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-05T08:42:34.411 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi090.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/benhanokh-2023-10-31_16:46:33-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442398', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-05 08:11:03.355607', 'locked_by': 'scheduled_benhanokh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNtyBiM3I8tIXFthmH5r8xkuDBDlJ50iU5mm6jCAXzGFO6LLWNfz+LKJeC7G8VVF6lJYYzPO5xDEdjKdlVey0UI='} 2023-11-05T08:42:34.427 DEBUG:teuthology.orchestra.console.smithi002:power off output: Chassis Power Control: Down/Off 2023-11-05T08:42:34.428 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-05T08:42:34.452 INFO:teuthology.orchestra.console.smithi053:Power off 2023-11-05T08:42:34.453 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-05T08:42:34.472 DEBUG:teuthology.orchestra.console.smithi002:check power output: Chassis Power is on 2023-11-05T08:42:34.489 DEBUG:teuthology.orchestra.console.smithi053:power off output: Chassis Power Control: Down/Off 2023-11-05T08:42:34.489 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-05T08:42:34.508 INFO:teuthology.orchestra.console.smithi090:Power off 2023-11-05T08:42:34.509 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-05T08:42:34.525 INFO:teuthology.orchestra.console.smithi069:Power off 2023-11-05T08:42:34.526 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-05T08:42:34.535 DEBUG:teuthology.orchestra.console.smithi090:power off output: Chassis Power Control: Down/Off 2023-11-05T08:42:34.535 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-05T08:42:34.546 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is on 2023-11-05T08:42:34.553 DEBUG:teuthology.orchestra.console.smithi069:power off output: Chassis Power Control: Down/Off 2023-11-05T08:42:34.553 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-05T08:42:34.566 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on 2023-11-05T08:42:34.581 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2023-11-05T08:42:38.473 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-05T08:42:38.548 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-05T08:42:38.567 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-05T08:42:38.582 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-05T08:42:38.593 DEBUG:teuthology.orchestra.console.smithi002:check power output: Chassis Power is on 2023-11-05T08:42:38.658 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is on 2023-11-05T08:42:38.678 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on 2023-11-05T08:42:38.693 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2023-11-05T08:42:42.598 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-05T08:42:42.659 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-05T08:42:42.679 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-05T08:42:42.694 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-05T08:42:42.708 DEBUG:teuthology.orchestra.console.smithi002:check power output: Chassis Power is off 2023-11-05T08:42:42.770 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off 2023-11-05T08:42:42.788 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on 2023-11-05T08:42:42.803 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2023-11-05T08:42:42.810 INFO:teuthology.orchestra.console.smithi002:Power off completed 2023-11-05T08:42:42.871 INFO:teuthology.orchestra.console.smithi053:Power off completed 2023-11-05T08:42:46.793 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-05T08:42:46.804 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-05T08:42:46.905 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is off 2023-11-05T08:42:46.913 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is off 2023-11-05T08:42:47.006 INFO:teuthology.orchestra.console.smithi090:Power off completed 2023-11-05T08:42:47.015 INFO:teuthology.orchestra.console.smithi069:Power off completed 2023-11-05T08:42:47.119 INFO:teuthology.run:Summary data: description: rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} duration: 1275.3872601985931 failure_reason: 'Command failed on smithi002 with status 124: ''sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json''' flavor: default owner: scheduled_benhanokh@teuthology status: fail success: false 2023-11-05T08:42:47.120 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-05T08:42:47.200 INFO:teuthology.run:FAIL